Skip to content
This repository has been archived by the owner on Apr 27, 2018. It is now read-only.
/ warcbase Public archive

Warcbase is an open-source platform for managing analyzing web archives

Notifications You must be signed in to change notification settings

lintool/warcbase

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Warcbase

Warcbase is an open-source platform for managing web archives built on Hadoop and HBase. The platform provides a flexible data model for storing and managing raw content as well as metadata and extracted knowledge. Tight integration with Hadoop provides powerful tools for analytics and data processing via Spark.

Bad news: Warcbase is defunct and no longer under active development!

Good news: In June 2017, the University of Waterloo and York University were awarded a grant from the Andrew W. Mellon Foundation to build the next generation of tools that will make historical internet content accessible to scholars. Warcbase serves as the foundation for the ArchivesUnleashed Toolkit!

If you're interested in reading about the development of Warcbase, check out this article:

Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.

License

Licensed under the Apache License, Version 2.0.

Acknowledgments

This work has been supported in part by the U.S. National Science Foundation, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, the Ontario Ministry of Research and Innovation's Early Researcher Award program, and the Mellon Foundation (via Columbia University). Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.

About

Warcbase is an open-source platform for managing analyzing web archives

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published