HAdoop-based Web Archive Record Processing
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
arc2warc-migration-cli
cdx-creator
droid-identify
hawarp-core
tika-identify
tomar-prepare-inputdata
unpack2temp-identify
workflows-scriptsandconfig
.gitignore
.opf.yml
.travis.yml
ABOUT.md
LICENSE
README.md
pom.xml

README.md

Hawarp

Hadoop-based Web Archive Record Processing

Hawarp Logo

These tools are designed to served this usecase: Quality assured ARC to WARC migration

Usage Examples

Examples how these tools can be applied in context are available in form of Taverna Workflows published on myExperiment:

Simple usage examples of the standaline Java executables

Use arc2warc-migration-cli to migrate an ARC container file to a file in the new WARC format:

java -jar hawarp/arc2warc-migration-cli/target/arc2warc-migration-cli-1.0-jar-with-dependencies.jar 
-i /local/input/directory/ -o /local/output/directory/

Run droid-identification using Apache Hadoop:

hadoop jar target/droid-identify-1.0.jar-with-dependencies.jar 
-d /hdfs/path/to/textfiles/with/absolutefilepaths/ -n job_name

The input for this Hadoop job is a text file listing file paths (either local file system paths – accessible from each worker node – or hadoop distributed file system paths).

More usage example on other tools can be found in the documentation of the individual modules.

Hawarp Modules

The following set of command line interface applications is included in this project as modules:

arc2warc-migration-cli

Command line application to convert ARC container files to the new ISO standard format WARC.

Download (v1.0)

Documentation

CDX creator

A tool to create CDX index files which are required if the wayback software is used to display archived resources contained in ARC or WARC container files.

Download (v1.0)

Documentation

droid-identify

Hadoop Job for Identifying files using DROID (Digital Record Object Identification), Version 6.1, http://digital-preservation.github.io/droid/.

Documentation

unpack2temp-identify

unpack2temp-identify is a tool to identify and/or characterise files packaged in container files using a standalone java application or a Hadoop job.

Documentation

tika-identify

Hadoop Job for Identifying files using Apache Tika Version 1.0.

Documentation

tomar-prepare-inputdata

tomar-prepare-inputdata is a tool to prepare web archive container files in the ARC format which are stored in a Hadoop Distributed File System (HDFS) in order to allow processing of the individual files by means of the SCAPE Platform tool Tomar.

Documentation

Requirements