iafetch

iafetch

iafetch downloads the archive.org items of a list, in the list's order, to DIR/ITEM/FILE. A SQLite database records every file it got and the state of every item. The list is normally archive.org/import-order.csv; any CSV with an identifier column works.

Reference: cmd/iafetch/README.md and iafetch -h. This page describes the main branch; release v0.3.2 has an older iafetch (see Installation).

Recipes

iafetch -n archive.org/import-order.csv                          # dry run: the items, with sizes from the list
iafetch -o /data/ia -db /data/iafetch.sqlite archive.org/import-order.csv
iafetch -status -db /data/iafetch.sqlite                         # what the database records

Choose the items:

iafetch -tier A archive.org/import-order.csv                     # one tier
iafetch -tier B1 -limit 500 archive.org/import-order.csv         # the next 500 items of B1 not yet done
iafetch -from 1144 -to 2000 archive.org/import-order.csv         # an order range
iafetch -list textusenet.txt archive.org/import-order.csv        # one list (the "list" column)
iafetch -match '^usenet-0\.akita-inu$' archive.org/import-order.csv   # one item
iafetch -tier X archive.org/import-order.csv                     # tier X, which is left out by default

Choose the files of each item, by regexp on the file name:

iafetch -exclude '\.csv\.gz$' archive.org/import-order.csv       # leave out the .csv.gz index files

Careful: done is recorded per item, not per filter. An item downloaded with -exclude or -include is done once the chosen files are in, and a later run without the filter skips it. An item where the filter chose no file is recorded as empty. To get the other files later, delete the item's rows (see The database).

Speed:

iafetch -j 512 -meta 32 -o /data/ia archive.org/import-order.csv # fill a 1 Gbit/s line
iafetch -item-conns 4 archive.org/import-order.csv               # big files in 4 pieces side by side

Stop with Ctrl-C. Files being downloaded stay as FILE.part, and the next run resumes them.

Being gentle with archive.org

flag	default
-j N	64	items at a time; the files of one item come one after another
-item-conns N	1	connections per item; above 1, files larger than -piece come in pieces
-piece MB	32	piece size; 0 for never
-host-conns N	4	connections to one archive.org data node; 0 for no limit
-conns N	-j × -item-conns	connections in all
-meta N	8	metadata requests at once, for up to 4×N items ahead
-nodes	on	download from the item's data nodes; -nodes=false always goes through archive.org/download/
-retries N	8	retries of a failed request, waiting 5 s, 10 s, 20 s ... up to 5m20s
-stall D	2m	give up on a request that gets no data for this long, and retry it

Each item lives on two data nodes. A request goes to the less busy one, and waits while both have -host-conns requests out. archive.org gives one connection well under 1 MB/s, so a fast line needs many items at once. The README has the measurements.

The list order holds only roughly: an item starts as soon as its metadata is in.

Item states

state	meaning	next run
done	every file downloaded and verified	skipped, whether or not its files are still there
incomplete	a file failed	gets what is missing
dark	taken down on archive.org	skipped until -retry-unavailable
missing	archive.org does not know the identifier	skipped until -retry-unavailable
empty	no files to download	skipped until -retry-unavailable

Only the database decides what is skipped. Keep it when you move or delete downloaded files, and keep it outside the download directory.

Some items in the list are gone from archive.org, such as textusenet-backup_a, _c, _e, _g, _h, _n and _u. textusenet-backup_0 and _f are listed but have no files, and are recorded as empty.

Checking

A file is written to FILE.part and renamed to FILE only when its size and md5 match the archive.org metadata. A file already there under its name is checked by md5 and kept if right. A different file there is an error; it is never replaced.

At the end iafetch logs a finished in ... line with the counts. If files failed, it says N files failed; run again to retry them.

The database

-db PATH, default ./iafetch.sqlite. Three tables:

table	columns
items	identifier, ord, tier, list, status, files, bytes, note, updated
files	identifier, name, size, md5, path, done_at
pieces	identifier, name, piece_size, piece (files being fetched in pieces)

files.path is the path under -o as given, so use an absolute -o when other programs read the database. Examples with the sqlite3 program:

# items that are not done, in list order
sqlite3 /data/iafetch.sqlite "SELECT ord, identifier, status, note FROM items WHERE status != 'done' ORDER BY ord"

# fetch item X again
sqlite3 /data/iafetch.sqlite "DELETE FROM items WHERE identifier='X'; DELETE FROM files WHERE identifier='X'"

Importing the collections uses the files table to feed the downloads in import order.