FAQ

FAQ

Does mbox2nntp change my files? No. The input is only read. The one change is the rename to INPUT.done when it is finished. Archives are unpacked into a new .mbox2nntp-* directory in -tmpdir, which is removed at the end of the run, whatever the outcome, and on Ctrl-C. Files already in -tmpdir are never touched. The exception is -dos2unix: if you give it, a plain .mbox is rewritten in place.

Do I need dos2unix? No. Older versions ran it on every mbox. The parser handles CRLF line ends itself; across the archive.org corpus (gna-clari, usenet-clari, akita-inu) every article comes out the same with and without it.

Is it safe to feed a file twice? Yes. The server answers 435 for articles it already has, and those cost little. That is the normal way to finish a file that was not marked done.

How do I feed many files? One run per file, in a loop. A finished file is renamed .done and drops out of the loop's pattern. See Importing the collections.

Can I run several files at once? Each run unpacks into its own .mbox2nntp-* directory, so runs sharing a -tmpdir do not get in each other's way (since v0.2.3). Within one run, -workers N opens N connections.

Why was an article not sent? It had no Message-ID, no body (in practice, cancel control messages), a yEnc body (unless -keep-binary), no Date: or no Subject:, more than 256 header lines, or a body over 1 MiB. See Reading the log.

What does mbox2nntp change in an article? A few header fields: it prepends mbox2nntp-INPUTNAME! to Path:, drops Xref:, Expires: and stray From lines, keeps one Message-ID, repairs Key:value to Key: value, and turns NUL, backspace and tab in header lines into spaces. Body lines are sent as stored. The full list is in the README under What is sent.

Does it un-escape >From lines? No, because none of the archives measured is mbox-escaped: a >From line there is part of the article as posted. The exception is an article with X-Mbox2nntp-Format: mboxrd in its header, as uatexport and ncexport write it. Its quoted lines lose one >, and the field is dropped.

Which mbox files can it read? The separator shapes of the archive.org collections, and the RFC 4155 line most mbox writers use (From user@example.com Thu Dec 8 12:55:04 2016) when it is the whole line. So mboxes from mutt, Thunderbird, Python's mailbox and usenet_archiver are read. Before a new collection, dry-run one file and check the near-misses (README).

Can it read a .txt file? Only when its first line begins with From , as in the Google Groups exports in the comp_sys_atari items. Any other .txt is refused.

What about the .csv.gz files next to the mboxes? They are index files, not inputs. Leave them out of your loops (*.mbox.gz does), or leave them out of the download with iafetch -exclude '\.csv\.gz$'.

Why feed the UAT archives last? The toolkit rewrote every message (headers decoded to UTF-8, only the text part kept), but the articles keep their original Message-IDs. A server keeps the first copy it gets. Feed the UAT archives after everything else, so the originals win where another archive has them.

Do I need uatexport to feed a UAT archive? No. Give the .usenet, .usenet.xz or .usenet.7z file to mbox2nntp. uatexport is for using the articles elsewhere.

Can I move or delete files iafetch downloaded? Yes. iafetch skips an item recorded as done without asking archive.org, whether or not its files are still there. Keep the database.

Why is iafetch -n bigger than what arrives? It adds up the item_size column of the list, archive.org's size of the whole item. iafetch leaves out the files archive.org makes itself and the old copies under history/.

Which server does it work with? It speaks IHAVE, and was written for INN2. -prehash (on by default) sends IHAVE <msgid> <sha256-of-msgid>; -prehash=false sends plain IHAVE <msgid>.

Old scripts use positional arguments. Do they still work? Yes, with a deprecation note in the log: mbox2nntp INPUT HOST:PORT [WORKERS [TMPDIR [DRYRUN [- [NODROPBINARY_FILE]]]]]. See The old positional form.

Where is the list of bugs fixed? In the CHANGELOG and the code review. If you imported with an older version, see Troubleshooting.