Skip to content
This repository has been archived by the owner on Feb 27, 2021. It is now read-only.


Switch branches/tags

Name already in use

A tag already exists with the provided branch name. Many Git commands accept both tag and branch names, so creating this branch may cause unexpected behavior. Are you sure you want to create this branch?

Latest commit


Git stats


Failed to load latest commit information.
Latest commit message
Commit time

IA Mine

Internet Archive data mining tools.

What Is IA Mine?

IA Mine is a command line tool and Python 3 library for data mining the Internet Archive.

How Do I Get Started?

Command Line Interface

The IA Mine command line tool should work on any Unix-like operating system that has Python 3 installed on it. To start using ia-mine, simply download one of the latest binaries from

# Download ia-mine and make it executable.
$ curl -LO
$ chmod +x ia-mine
$ ./ia-mine --help


$ ia-mine --help
Concurrently retrieve metadata from items.

usage: ia-mine (<itemlist> | -) [--debug] [--workers WORKERS] [--cache]
               [--retries RETRIES] [--secure] [--hosts HOSTS]
       ia-mine [--all | --search QUERY] [[--info | --info --field FIELD...]
               |--num-found | --mine-ids | --field FIELD... | --itemlist]
               [--debug] [--rows ROWS] [--workers WORKERS] [--cache]
               [--retries RETRIES] [--secure] [--hosts HOSTS]
       ia-mine [-h | --version | --configure]

positional arguments:
  itemlist              A file containing identifiers, one per
                        line, for which to retrieve metadata from. If no
                        itemlist is provided, identifiers will be read from

optional arguments:
  -h, --help            Show this help message and exit.
  -v, --version         Show program's version number and exit.
  --configure           Configure ia-mine to use your credentials.
  -d, --debug           Turn on verbose logging [default: False]
  -a, --all             Mine all indexed items.
  -s, --search QUERY    Mine search results. For help formatting your query,
  -m, --mine-ids        Mine items returned from search results.
                        [default: False]
  -i, --info            Print search result response header to stdout and exit.
  -f, --field FIELD     Fields to include in search results.
  -i, --itemlist        Print identifiers only to stdout. [default: False]
  -n, --num-found       Print the number of items found for the given search
  --rows ROWS           The number of rows to return for each request made to
                        the Advancedsearch API. On slower networks,
                        it may be useful to use a lower value, and on faster
                        networks, a higher value. [default: 50]
  -w, --workers WORKERS
                        The maximum number of tasks to run at once.
                        [default: 100]
  -c, --cache           Cache item metadata on Items are not
                        cached are not cached by default.
  -r, --retries RETRIES
                        The maximum number of retries for each item.
                        [default: 10]
  --secure              Use HTTPS. HTTP is used by default.
  -H, --hosts HOSTS     A file containing a list of hosts to shuffle through.

Python Library

The IA Mine Python library can be installed with pip:

# Create a Python 3 virtualenv, and install iamine.
$ virtualenv --python=python3 venv
$ . venv/bin/activate
$ pip install iamine

This will also install the ia-mine comand line tool into your virtualenv:

$ which ia-mine

Data Mining with IA Mine and jq

ia-mine simply retrieves metadata and search results concurrently from, and dumps the JSON returned to stdout and any error messages to stderr. Mining the JSON dumped to stdout can be done using a tool like jq, for example. jq binaries can be downloaded at

ia-mine can mine search results, the items returned from search results, or items provide via an itemlist or stdin.


Please report any bugs or issues on github:


Internet Archive Data Mining Tools







No packages published