Troubleshooting

Troubleshooting

First read the summary line and the Done read line; Reading the log explains them. Then find your case below.

mbox2nntp

Near-misses that look like separators

near-misses counts body lines starting From or #! rnews that were not taken as the start of a message. Up to five are logged with line numbers.

• Prose, such as From Table 1 ...: normal. Nothing to do.
• Lines like From user@host Thu Dec 8 12:55 2016: the collection uses a separator shape the parser does not know, and articles were merged. Do not import it. The shapes are listed in sepPatterns in mbox.go of go-lib-mbox2nntp; its README explains the shapes and how to add one.

The parser also takes the RFC 4155 line that most mbox writers use, From user@example.com Thu Dec 8 12:55:04 2016, when it is the whole line in exactly that form.

stopped-early=true

The parser lost its place before the end of the file, or a read error ended it. The input is not marked done. Look at last-err and the ERROR lines before it.

A line over 1 MiB also stops the file there. Running it again gives the same result, so such a file needs a closer look; rerunning does not help.

ERROR: all N messages dropped, none offered

Every message lacked a Message-ID or a body, or could not be parsed. That more likely means a separator shape the parser does not know. Dry-run the file and read the near-misses.

ERROR: N articles were never sent

No worker could connect. Check -server HOST:PORT, and that the server takes connections and IHAVE from your machine. Then run again.

err above 0

The server answered something other than 235, 435, 436 or 437, or an article ran out of -max-tries after 436 replies or lost connections. Look for ERROR Worker N) code=... giving up lines and the server's own log. Run again later; the articles already accepted come back as 435.

The default -prehash sends IHAVE <msgid> <sha256-of-msgid>. -prehash=false sends the plain IHAVE <msgid> of RFC 3977. If your server answers every IHAVE with an error, try -prehash=false.

Many e437

A 437 is not an error, and the input can still be marked done. It comes either from the server, or from mbox2nntp itself, which skips articles with no Date:, no Subject:, more than 256 header lines, or a body over 1 MiB (also counted in too-big).

Lost connections, timeouts, 436

These are retried, on a new connection for a lost one, up to -max-tries (default 10) per article. -timeout (default 2m) is the limit on each read or write. If the server keeps answering 436, the article ends as an error after -max-tries; run the file again later.

ERROR head=N body=0 msgid=...

One line per message without a body, mostly cancel control messages. They are the empty count. Normal; not a failure.

Exit code 1 at the start

log	fix
ERROR: input not found '...'	check the path
no 7z, 7zz or 7za in PATH: install 7-Zip, or give its path with -7z	install 7-Zip, or give -7z PATH
no zstd in PATH: ...	install zstd, or give -zstd PATH
no xz or 7-Zip in PATH: ...	install one, or give -xz or -7z
gunzip failed ... stat DIR: no such file or directory	-tmpdir must exist: mkdir -p DIR
file NAME.txt is not an mbox: a .txt input must begin with a "From " line	a .txt is read only when it is an mbox
file NAME is not an mboxfile?	the name does not end in a known suffix; see the input table in the README
ERROR: -dos2unix: ...	the -dos2unix program was not found; it is not needed, leave the flag out

When unpacking fails to write, for example on a full disk, the error says so. Free space in -tmpdir, or point it elsewhere.

A directory was not renamed .done

Directory inputs are never renamed. Keep your own note of which ones you fed.

Articles imported with an older version

The code review found bugs that lost or changed articles in earlier versions, and says which collections were affected:

• B6: one malformed message made the importer drop the rest of its file. In usenet-clari, a default run (with dos2unix) lost 1,421 articles in clari.world.mideast.iraq and clari.world.mideast.palestine; with .nodos2unix, 112,828 articles in 191 files. Those files were renamed .done anyway. Feed them again with the current version.
• E1: a folded Xref: tail was appended to the header before it, in 82% of gna-clari articles. The server either stored them that way, or rejected them with 437. Feeding again sends the rejected ones; it does not repair stored ones, because the server answers 435 for Message-IDs it already has.

Also, v0.2.2 with the default -tmpdir . could delete a NAME.mbox lying next to the NAME.mbox.gz being imported. v0.2.3 and later unpack into a directory of their own.

iafetch

symptom	what to do
N files failed; run again to retry them	run again; the item is incomplete and gets what is missing
interrupted; run again to go on	run again; .part files are resumed
an item is dark, missing or empty	it is skipped until -retry-unavailable. Some items are gone from archive.org for good (see iafetch)
a different file is already there under the same name	iafetch reports an error and never replaces it. Move the file away
a request hangs	after -stall (default 2m) without data it is given up and retried
an item you need again is done	delete its rows from items and files (see iafetch)
files you excluded with -exclude/-include are wanted now	same: done is per item, not per filter

ncexport

symptom	what to do
the summary counts skipped .db files	their articles are missing: raise -max-db, or -max-db 0
out of disk while reading a tarball	a .db over 1 MiB is copied to -tmpdir (default: the output's directory); give it room for the largest .db
ERROR: NAME exists	the output directory is there already; ncexport never replaces it
many articles "listed in .db without an .art file"	most are crossposts stored under another group; -missing FILE lists the ones really missing