The Archives Unleashed Toolkit
The toolkit grew out of a previous project called Warcbase. The following article provides a nice overview, much of which is still relevant:
- Jimmy Lin, Ian Milligan, Jeremy Wiebe, and Alice Zhou. Warcbase: Scalable Analytics Infrastructure for Exploring Web Archives. ACM Journal on Computing and Cultural Heritage, 10(4), Article 22, 2017.
If you have Apache Spark ready to go, it's as easy as:
$ spark-shell --packages "io.archivesunleashed:aut:0.16.0"
A little less easy
You can download the latest release here and include it like so:
$ spark-shell --jars /path/to/aut-0.16.0-fatjar.jar"
Even less easy
Build it yourself as per the instructions below:
Clone the repo:
$ git clone http://github.com/archivesunleashed/aut.git
You can then build The Archives Unleashed Toolkit.
$ mvn clean install
For the impatient, to skip tests:
$ mvn clean install -DskipTests
I want to use Docker!
Ok! Take a quick spin with
aut with Docker.
Documentation! Or, how do I use this?
Once built or downloaded, you can follow the basic set of recipes and tutorials here.
Licensed under the Apache License, Version 2.0.
This work is primarily supported by the Andrew W. Mellon Foundation. Additional funding for the Toolkit has come from the U.S. National Science Foundation, Columbia University Library's Mellon-funded Web Archiving Incentive Award, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, and the Ontario Ministry of Research and Innovation's Early Researcher Award program. Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.