uatexport

uatexport

uatexport writes a Usenet Archive Toolkit (UAT) archive out as an mbox file, or as one file per article.

You do not need it to feed a server. Give the .usenet, .usenet.xz or .usenet.7z file to mbox2nntp as it is. uatexport is for using the articles elsewhere.

Reference: cmd/uatexport/README.md and uatexport -h. What the toolkit changed in the articles, and what mbox2nntp and uatexport restore, is in the main README under UAT archives.

Recipes

uatexport alt.pl.fan.gophi.usenet.xz                     # writes alt.pl.fan.gophi.mbox
uatexport -gzip alt.pl.fan.gophi.usenet.xz               # writes alt.pl.fan.gophi.mbox.gz
uatexport -gzip=9 -o /srv/mbox/gophi.mbox.gz alt.pl.fan.gophi.usenet.xz
uatexport -format dir -o spool/ rec.games.rogue.usenet.7z    # spool/1, spool/2, ...
uatexport -raw alt.pl.fan.gophi.usenet.xz                # as the toolkit stores them

It prints one line per archive:

alt.pl.fan.gophi: 527 articles to alt.pl.fan.gophi.mbox.gz, 83 header fields re-encoded, 0 left with non-ASCII

It needs zstd, plus xz or 7-Zip for .usenet.xz and 7-Zip for .usenet.7z. See Installation.

Flags

flag	default
-format	mbox	mbox, or dir for one file per article
-gzip	off	write GROUP.mbox.gz; -gzip=N sets the level, 1 to 9, default 6
-o PATH	GROUP.mbox, GROUP.mbox.gz, or the directory GROUP	the output
-raw	off	raw UTF-8 headers, no Content-Type, as the archive stores them
-tmpdir DIR	the system's	where .xz and .7z inputs are unpacked
-zstd, -xz, -7z	found in PATH	the outside programs

An existing output is never replaced. The output is written under a hidden temporary name, .NAME.*.tmp, and renamed when complete. Ctrl-C removes the temporary files.

The output

The mbox is mboxrd. Each article starts with From nobody DATE and ends its header with X-Mbox2nntp-Format: mboxrd. Lines starting From or #! rnews (after any number of >) get one more >. mbox2nntp reads such an mbox back exactly: the field tells it to take that > off again, and the field is not sent. An exported directory works as a directory input.

Size

Measured on 7 groups, per byte of .usenet.xz or .7z archive: the .mbox takes 2.4, the .mbox.gz 0.56. The README has the full table and the timing.

Downloading and exporting whole collections

contrib/uat-fetch-export.sh downloads the two published UAT collections and runs uatexport on each group:

• nereid: usenet.nereid.pl, Polish groups as .usenet.xz;
• archive: the archive.org item usenet-uat-pc-games, as .usenet.7z.

It downloads one file at a time, pauses between files (-p, default 2 s), can cap the bandwidth (-r), retries, resumes, and checks each download (archive.org by md5, nereid by size and xz -t). Finished downloads and exports are skipped, so it can be stopped and run again. It needs bash, curl, uatexport, zstd, xz and 7-Zip, and ia or jq for archive.org.

contrib/uat-fetch-export.sh -n                           # what it would do, with sizes
contrib/uat-fetch-export.sh -s archive --sort size -l 3  # the 3 smallest archive.org groups
contrib/uat-fetch-export.sh -s nereid -m '^pl\.comp\.' -d /srv/uat -o /srv/mbox -r 5M
contrib/uat-fetch-export.sh --download-only              # only the archives, to feed with mbox2nntp
contrib/uat-fetch-export.sh -h                           # all options

For both collections, expect about 25 GB of .mbox.gz, and room to unpack the largest archive, pl.soc.polityka (about 3.5 GB unpacked). --delete-archive removes each archive once its group is exported. Keep the archives if you want to feed them to a server: mbox2nntp reads them directly.

The uat package

Other Go programs can read UAT archives with github.com/go-while/mbox2nntp/uat (standard library, plus the zstd program).