Importing the collections

Importing the archive.org collections

This page goes from an empty disk to a full server:

1. download the items with iafetch, in import order;
2. dry-run one file of each collection you have not imported before;
3. feed the mbox items with mbox2nntp;
4. turn the NewsCache spools into mboxes with ncexport, and feed those;
5. feed the UAT archives last;
6. run again until everything is .done, then free the disk.

The paths below (/data/ia, /data/iafetch.sqlite, localhost:119) are examples.

The import order

archive.org/import-order.csv lists every archive.org item found for Usenet, one row per item. Its columns:

column	meaning
order	the import order; iafetch takes items by it
tier	the group of items, below
list	the search list the item came from (usenet-.txt, gna.txt, ...)
identifier	the archive.org item
content_from, content_to, basis	the dates the content covers, and how they were found
addeddate, item_size, title	from archive.org
format	mbox, newscache, uat, spool, none or check
how	what to run on the item

The tiers, in order. Item counts and sizes are from the item_size column, each identifier counted once:

tier	items	size	what	how
A	32	66.3 GB	the early material: utzoo, news CD-ROMs, Google Groups exports	by hand, per item; see Tier A
B0	1,024	694.9 GB	usenet-* items, giganews hierarchy dump, Aug 2013	mbox2nntp: .mbox.zip
B1	25,322	4,037.1 GB	usenet-* items, giganews captures 2014-06 to 2015-02	mbox2nntp: .mbox.gz (skip the .csv.gz index)
B2	3,416	161.6 GB	gna-* items, up to a Dec 2016 dump	mbox2nntp: .mbox.gz (skip the .csv.gz index)
B3	52,800	3,746.7 GB	FULL-USENET-BACKUP-2020-Oct-*, a snapshot of Oct 2020	mbox2nntp: .mbox.7z
B4	30	976.5 GB	textusenet-backup_*, NewsCache spools	ncexport the .tar.gz, then mbox2nntp
C	333	480.2 GB	the rest: UAT archives, gmane backups (NewsCache), other mboxes, items not checked yet	per item, see how
X	105	159.1 GB	not Usenet, or no articles (PDFs, listings, web crawls)	nothing; iafetch leaves them out

In all, 83,062 items and 10.32 TB. item_size is archive.org's size of the whole item. It includes files iafetch does not download, such as the old copies under history/, so the download is smaller. iafetch -n adds up the same item_size numbers: without -tier it takes every tier but X, 82,957 items and 10,163 GB.

Why the order matters. A server keeps the first copy of an article it gets, and answers 435 to every later copy with the same Message-ID. So whichever archive you feed first decides which copy the server keeps.

UAT archives go last. The Usenet Archive Toolkit rewrote every message: headers decoded to UTF-8, only the text part of the body kept, Content-Type dropped. The articles keep their original Message-IDs. Fed first, the rewritten copy would win. In the CSV the UAT items (order 82712 to 82725) sit inside tier C, before the gmane backups and other C items, so hold them back until everything else is in.

1. Download with iafetch

See what would come, and how big:

iafetch -n archive.org/import-order.csv

Download a tier at a time. Give absolute paths, and keep the database outside the download directory:

iafetch -tier A -o /data/ia -db /data/iafetch.sqlite archive.org/import-order.csv
iafetch -tier B0 -o /data/ia -db /data/iafetch.sqlite archive.org/import-order.csv

Files land in /data/ia/ITEM/FILE. A finished item is recorded as done in the database and is never fetched again, even when its files are gone. So you can feed a tier, delete its .done files, and download the next one. -from, -to and -limit cut a tier into smaller parts:

iafetch -tier B1 -limit 2000 -o /data/ia -db /data/iafetch.sqlite archive.org/import-order.csv
iafetch -status -db /data/iafetch.sqlite

More recipes, and how fast and gentle it is: iafetch.

2. Dry-run a new collection

Before you feed a collection for the first time, dry-run one of its files and read the Done read line:

mbox2nntp -dry-run /data/ia/usenet-0.akita-inu/0.akita-inu.20140617.mbox.gz

stopped-early must be false, and the near-misses samples must look like prose. See Quick start and Reading the log.

3. Feed the mbox items

mbox2nntp takes one input per run. A batch script loops over the files. A file that is done is renamed NAME.done, so it no longer matches the loop's pattern, and the next run of the script skips it.

The simplest loop, over one tier's files (here B1, .mbox.gz):

#!/usr/bin/env bash
mkdir -p /var/tmp/m2n
for f in /data/ia/usenet-*/*.mbox.gz; do
    [ -f "$f" ] || continue
    mbox2nntp -server localhost:119 -workers 4 -tmpdir /var/tmp/m2n "$f" </dev/null >>import.log 2>&1
    [ -f "$f.done" ] || echo "$f" >>not-done.txt
done

The pattern *.mbox.gz leaves out the .mbox.csv.gz index files beside the mboxes. Use /data/ia/usenet-*/*.mbox.zip for tier B0, /data/ia/gna-*/*.mbox.gz for tier B2, and /data/ia/FULL-USENET-BACKUP-*/*.mbox.7z for tier B3. A .7z input is unpacked into -tmpdir first, so that needs room for the largest unpacked mbox.

The shell sorts the files by name, not by import order. To keep the import order, take the file list from the iafetch database, which knows each item's order:

#!/usr/bin/env bash
# feed-tier.sh TIER: feed the downloaded mbox files of one tier, in import order.
set -u
tier=${1:?usage: feed-tier.sh TIER}
db=/data/iafetch.sqlite
mkdir -p /var/tmp/m2n
sqlite3 "$db" "SELECT f.path FROM files f JOIN items i ON i.identifier = f.identifier
    WHERE i.tier = '$tier'
      AND (f.name LIKE '%.mbox.gz' OR f.name LIKE '%.mbox.zip' OR f.name LIKE '%.mbox.7z')
    ORDER BY i.ord, f.name" |
while IFS= read -r f; do
    [ -f "$f" ] || continue    # already .done, or moved away
    mbox2nntp -server localhost:119 -workers 4 -tmpdir /var/tmp/m2n "$f" </dev/null >>"import-$tier.log" 2>&1
    [ -f "$f.done" ] || echo "$f" >>"not-done-$tier.txt"
done

files.path is the path as iafetch wrote it, so give iafetch an absolute -o. The </dev/null keeps the programs mbox2nntp starts from reading the file list.

Check not-done-*.txt and the log, fix what Troubleshooting says, and run the script again. Files already done are skipped; articles the server already has come back as 435.

Do not trust the exit code: mbox2nntp exits 0 when it finishes a run, whether or not the input was marked done. It exits 1 on a fatal error and 2 on a command-line error.

4. The NewsCache spools (tier B4, and the gmane backups in C)

These items hold .tar.gz files, one per hierarchy, each the spool of a NewsCache server. ncexport writes each spool out as one mbox per group, then mbox2nntp feeds those:

#!/usr/bin/env bash
mkdir -p /data/nc
for t in /data/ia/textusenet-backup_*/*.tar.gz /data/ia/gmane.*.BACKUP.2021-03/*.tar.gz; do
    [ -f "$t" ] || continue
    item=$(basename "$(dirname "$t")")
    out=/data/nc/$item/$(basename "$t" .tar.gz)
    [ -e "$out" ] && continue    # exported before; ncexport never replaces an output
    mkdir -p "/data/nc/$item"
    ncexport -gzip -max-db 0 -o "$out" "$t" >>ncexport.log 2>&1
done
for f in /data/nc/*/*/*.mbox.gz; do
    [ -f "$f" ] || continue
    mbox2nntp -server localhost:119 -workers 4 "$f" </dev/null >>import-nc.log 2>&1
    [ -f "$f.done" ] || echo "$f" >>not-done-nc.txt
done

ncexport builds its output directory under a temporary name and renames it only when complete, so [ -e "$out" ] is true only for finished exports. -max-db 0 lifts the 4 GiB limit on a group's .db file: the textusenet backup came from a 64-bit NewsCache, which can write bigger ones. A .db over 1 MiB is copied to -tmpdir while it is read (default: the output's directory), so that needs room for the largest .db. Each group's .mbox.gz is renamed .done on its own. The gmane backups of tier C (gmane.*.BACKUP.2021-03) are NewsCache spools too, which is why the loop takes them as well; leave them out if you want to keep tier order strictly. Details: ncexport.

5. The UAT archives, last

Give the archives to mbox2nntp as they are. It needs zstd, and 7-Zip for .usenet.7z or xz (or 7-Zip) for .usenet.xz:

#!/usr/bin/env bash
for f in /data/ia/*/*.usenet /data/ia/*/*.usenet.xz /data/ia/*/*.usenet.7z; do
    [ -f "$f" ] || continue
    mbox2nntp -server localhost:119 -workers 4 "$f" </dev/null >>import-uat.log 2>&1
done

You do not need uatexport to feed a server. The how column of the CSV says "uatexport, then mbox2nntp", but the README says to give mbox2nntp the archive itself.

The Polish groups are also published at usenet.nereid.pl. contrib/uat-fetch-export.sh --download-only fetches them; see uatexport.

Tier A, by hand

The tier A items are CD-ROM images, a squashfs image and Google Groups exports. The how column says what each needs. List them with:

iafetch -n -tier A archive.org/import-order.csv

The kinds of how in tier A:

how	what to do
unsquashfs utzoo.sqsh (2.1M article files), then mbox2nntp directory mode	unpack the image, then mbox2nntp -server localhost:119 DIR/
7z x the CD image, then mbox2nntp directory mode	unpack with 7-Zip, then feed the directory
7z x the .7z, then the .bin/.cue CD image, then mbox2nntp directory mode	two steps of 7-Zip, then the directory
unzip, then mbox2nntp directory mode	unzip, then the directory
mbox2nntp NAME.txt (v0.2.1+; older: rename to .mbox): an mbox from Google Groups; skip the PDF	mbox2nntp -server localhost:119 NAME.txt
floppy images (FAT); contents not checked	not checked yet

In directory mode, articles without a Message-ID are skipped, and index files, archives, sources and binaries are skipped by name. A directory is never renamed .done, so keep your own note of which ones you fed.

Tier C has items whose how is not checked yet. Dry-run them before you feed them.

6. Clean up

When an item's files are all .done, you can delete them. iafetch will not fetch a done item again. To see what the database records:

iafetch -status -db /data/iafetch.sqlite