Resources for Archives Unleashed Toolkit
Clone or download
Type Name Latest commit message Commit time
Failed to load latest commit information.
NER initial commit Nov 26, 2015
Sample-Data Updating Sample Data to use AUT name Sep 14, 2017 Create Nov 3, 2017 Updated documentation links in aut-resources repo Nov 16, 2017

Archives Unleashed Toolkit (AUT) Resources

This is the resource repository for the Archives Unleashed Toolkit (AUT), which you can find here. AUT is an open-source platform for analyzing web archives. Tight integration with Hadoop provides powerful tools for analytics and data processing via Apache Spark. Our full documentation can be found at

The Archives Unleashed Toolkit is part of the broader Archives Unleashed project.

Getting Started

Follow the instructions on the main AUT repository, and consult the detailed documentation available here.

This repository contains supporting resources that you need for some of the commands, such as:

  • Named Entity Recognition;
  • Sample WARC and ARC files for protoyping


  • /ner/english.all.3class.distsim.crf.ser.gz: This classifier is trained on English-language locations, organizations, and people. It is a central part of the Spark: Named Entity Recognition recipie.
  • Sample Data: In the /sample-data/ directory, we have one ARC and one WARC file from the University of Toronto's Canadian Political Parties collection. They're indicative of the kind of data you'll find within the Internet Archive or Archive-It.


This work is primarily supported by the Andrew W. Mellon Foundation. Additional funding for the Toolkit has come from the U.S. National Science Foundation, Columbia University Library's Mellon-funded Web Archiving Incentive Award, the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, and the Ontario Ministry of Research and Innovation's Early Researcher Award program. Any opinions, findings, and conclusions or recommendations expressed are those of the researchers and do not necessarily reflect the views of the sponsors.