ncexport

ncexport

ncexport turns a NewsCache spool into one mbox per group, for mbox2nntp to feed.

The archive.org textusenet-backup_* items hold such spools as .tar.gz files, one per hierarchy (97comp.tar.gz, rainbow.tar.gz, ...). The gmane backups (gmane.*.BACKUP.2021-03) are NewsCache spools too. mbox2nntp does not read these tarballs itself.

Reference: cmd/ncexport/README.md and ncexport -h.

Recipes

ncexport -gzip rainbow.tar.gz                           # rainbow/rainbow.test.mbox.gz, ...
ncexport -gzip -o /data/nc/rainbow rainbow.tar.gz       # name the output directory
ncexport -o comp /var/spool/newscache/comp              # an unpacked spool directory
ncexport -gzip -max-db 64G -tmpdir /big/tmp r.tar.gz    # .db files up to 64 GiB
ncexport -gzip -missing rainbow.missing.tsv rainbow.tar.gz   # list the articles that are really missing

Straight from the download, without saving the tarball (-o is needed with -):

curl -sL https://archive.org/download/textusenet-backup_9/97comp.tar.gz |
    ncexport -gzip -o 97comp -

Then feed each group, one run per file:

for f in rainbow/*.mbox.gz; do mbox2nntp -server localhost:119 "$f"; done

Each .mbox.gz is renamed .done on its own. Expect mbox2nntp to skip articles with an empty body (24 in 97comp), which some spam has, and yEnc binaries unless you give -keep-binary.

Flags

flag	default
-gzip	off	write GROUP.mbox.gz; -gzip=N sets the level, 1 to 9, default 6
-o DIR	NAME (INPUT without its extension)	the output directory; needed for -
-max-db SIZE	4G	skip a group's .db over SIZE; 0 for no limit
-tmpdir DIR	the output's directory	temporary files: a .db over 1 MiB from a tarball, and the list for -missing
-missing FILE	none	write the articles a .db lists that are in neither the spool nor the export
-dedup	on	write an article once per group mbox, by Message-ID; -dedup=false writes every copy

What to know

• The output appears only when complete. It is written under a temporary name, .NAME.*.tmp, and renamed at the end. An existing output is never replaced, so a batch script can skip inputs whose output directory exists.
• Disk. Nothing is unpacked, except a .db over 1 MiB from a tarball, which is copied to -tmpdir while it is read. -tmpdir needs room for the largest .db. The default is the output's directory, not the system's temporary directory, which is often small.
• -max-db. 4G is the most a 32-bit NewsCache can write. The textusenet backup came from a 64-bit NewsCache, which can write bigger ones. A skipped .db is counted in the summary, and its articles are missing from the export. Raise -max-db, or use -max-db 0.
• Crossposts. An article crossposted to several groups goes to each group's mbox. Within one group's mbox it is written once.
• What changes. Only the line ends (CRLF to LF) and the doubled leading dots. Articles over 1 MiB are skipped with a warning and counted.
• Missing articles. A group's .db lists articles that have no .art file. Most of them are crossposts stored under another group, so they are in the export after all. -missing FILE looks each one up by Message-ID and writes only the others to FILE, one tab-separated line each: group, number, date, subject, from, Message-ID. Use it to look for them in other archives. Articles crossposted to a hierarchy in another tarball count as missing.
• Memory. The dedup is per group, so its memory is bounded by the largest group. With -missing, every exported Message-ID is kept in memory (about 100 bytes each), and the listed articles wait in a temporary file in -tmpdir (about 150 bytes each).

The summary

At the end ncexport counts the articles, the group files, the groups, the .art files and the articles found inside .db files. It also counts duplicates skipped within a group, the articles a .db lists without an .art file (split into found elsewhere and really missing when -missing is given), articles over 1 MiB, and skipped .db files. For example:

$ ncexport -gzip -missing 97comp.missing.tsv 97comp.tar.gz
97comp.tar.gz: 757 articles to 97comp/ (1 group files), from 1 groups: 757 .art files, 0 inside .db files; 1 listed in .db without an .art file: 0 of them exported from another group, 1 lost, listed in 97comp.missing.tsv

The -missing file starts with a header line: group, number, date, subject, from, message-id.

If the output directory exists already, ncexport stops with ERROR: 97comp exists and exit code 1.

The newscache package

Other Go programs can read NewsCache spools with github.com/go-while/mbox2nntp/newscache (standard library only).