Skip to content
Alex Osborne edited this page Jul 4, 2018 · 11 revisions

This is the public wiki for the Heritrix archival crawler project.

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Heritrix (sometimes spelled heretrix, or misspelled or mis-said as heratrix/heritix/ heretix/heratix) is an archaic word for heiress (woman who inherits). Since our crawler seeks to collect and preserve the digital artifacts of our culture for the benefit of future researchers and generations, this name seemed apt.

All topical contributions to this wiki (corrections, proposals for new features, new FAQ items, etc.) are welcome! Register using the link near the top-right corner of this page.


Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.

If you notice our crawler behaving poorly – The Internet Archive uses archive.org_bot as User Agent when crawling – please send us email at

(If you see a different User-Agent in your logs that still says 'heritrix', it may be someone else using this open-source software. In such a case, even if we can't directly change how your site is crawled, we are happy to help you interpret your logs and identify, contact, or block the source of any troublesome crawling.)


Heritrix 3.x latest stable development snapshot:$heritrix/

Heritrix 3.3.0-LBS-2016-02:;thirdparty~browsestorage

Heritrix 3.2.0:

Heritrix 1.14.4:


Heritrix is free software; you can redistribute it and/or modify it under the terms of the Apache License, Version 2.0: Some individual source code files are subject to or offered under other licenses.

For more details see

Latest Releases

'Master' builds

The latest stable builds of Heritrix can be downloaded from

This represents the most up-to-date version of Heritrix and may include fixes for significant issues.

Heritrix 3.3.0-LBS-2016-02 (May 2016)

A 3rd party build of Heritrix. This release uses Heritrix 3.3.0 master from May 2016. It was tested by several institutions and found to be very stable. It addresses many serious issues in 3.2.0. Blog post about this release.

Binary (-dist) and source (-src) distributions are available at:;thirdparty~browsestorage

Source fork can be found here:

Heritrix 3.2.0 (Jan 2014)

Heritrix 3.2.0 final release is now available. See the following Release Notes page on the project wiki for full details: Release Notes - Heritrix 3.2.0

Binary (-dist) and source (-src) distributions are available at:

As always, problem reports, ideas, fix/feature contributions, and other kinds of feedback are all welcome here on the list and on the project wiki and JIRA issue tracker:

Heritrix Wiki:

Heritrix JIRA:

Heritrix 1.14.4 (May 2010)

Heritrix release 1.14.4 is now available at Sourceforge:

This is a 'micro' release with small bug fixes and improvements. Online release notes for 1.14.4 with a list of issues addressed are available at: Release Notes - 1.14.4

Project bug/issue tracking For Heritrix is in a JIRA-based system:

If you register to report/comment on JIRA issues, that same login will work for wiki edits and comments, and we welcome people to 'be bold' in adding/updating/improving project wiki pages.


For starting with Heritrix3, the Heritrix 3.0 and 3.1 User Guide provides getting-started and reference information.

Javadoc API documentation, which also serves as the canonical documentation for crawl operators on configuration of built-in modules:

The 1.x-based Heritrix User Manual covers getting started with Heritrix and many advanced topics. This User Manual is generally focused on Heritrix 1.X versions, not fully updated for 1.12/1.14 or the larger changes in 2.0/3.0, but provides a reasonable basis for getting started with Heritrix, especially 1.14.4.

There is a FAQ from the 1.x era:

The Knowledge Base contains topical articles documenting parts of the crawler and common problems or usage scenarios.

For developers, the 1.x-based Heritrix Developer Manual provides a guide to extending and customizing Heritrix code for your own purposes, though of course the source code itself, which is fairly well-commented, is the best guide.

For future documentation improvements, we have a [Documentation Wishlist](Documentation Wishlist).

An Introduction to Heritrix provides more detailed information on the structure and design of Heritrix.

Some very-old info can still be gleaned from the old wiki (*/

Mailing lists

Heritrix discussion:

Heritrix source code commits:


As of October 2011, immediately following the 3.1.0 release, the Heritrix 3 source repository is hosted at github:

Older source code can be found in sourceforge svn: Heritrix JIRA bug/feature/issue tracking: [Note to the public: this site does not allow creation of an account with @gmail address :{]


Wiki index


User Guide

API Guide

Knowledge Base

Known Issues

Background Reading

Users of Heritrix

How To Crawl


Clone this wiki locally
You can’t perform that action at this time.