Reading the log

Reading the log

mbox2nntp logs to standard error. To keep it, redirect it: mbox2nntp ... 2>>import.log.

The README sections Reading the log, Server replies and What happens to an input are the reference. This page puts them together.

The summary line

The last count in the log:

Done sending. 1465/1467 dup=0 e437=2 err=0 filter=0 dropped=61 binary=0

field	meaning
1465/1467	articles accepted (235) out of articles offered. In a dry run, the first number counts what would have been offered
dup	435: the server has it already
e437	437: rejected by the server, or skipped by mbox2nntp (see below). Not an error
err	any other reply, or out of tries. The input is not marked done
filter	a leftover, always 0
dropped	messages read but never offered: no Message-ID, no body, or not parseable. Not in the total
binary	yEnc articles left out (see -keep-binary). Not in the total

Neither dropped nor binary keeps an input from being marked done. The exception: when every message was dropped and none offered, which more likely means a separator shape the parser does not know.

The per-file line (mbox inputs)

[iterateMbox] Done read mbox file='...' msgNumber=1528 skipped=0 no-msgid=4 empty=57 binary=0 too-big=0 near-misses=0 stopped-early=false last-err=<nil>

field	meaning
msgNumber	messages read from the file
skipped	malformed messages the parser stepped over; the rest of the file is still read
no-msgid, empty	messages without a Message-ID, or without a body. With skipped they make up dropped
binary	yEnc articles left out
too-big	articles with a body over 1 MiB; not sent, counted as e437
near-misses	body lines starting From or #! rnews that were not taken as a separator. Up to five are logged with line numbers
stopped-early	true: the parser lost its place before the end, or a read error ended it. The input is not marked done
last-err	the error that ended reading, if any

Each message without a body also gets its own line, ERROR head=N body=0 msgid='...'. These are the empty count; they are normal (mostly cancel control messages) and do not stop the input from being marked done.

The per-file line (UAT archives)

[procUAT] Done read '.../alt.pl.fan.gophi.usenet' messages=527 no-msgid=0 binary=0 empty=0 too-big=0 re-encoded-fields=83 added-mime-version=149 stopped-early=false

re-encoded-fields counts the header fields put back into RFC 2047 encoded-words. The line also names any field left with non-ASCII in it. See UAT archives.

Directory inputs

A directory has no Done read line. Instead:

mbox2nntp done is_dir reading totalfiles=2 badfiles=1

A bad file is one that could not be read as an article; each gets an ERROR line with its path. A file over 1 MiB is not read, and counts as a 437. Files are also skipped by name: index and overview files, archives, sources and binaries, patch*, and names containing ).

Server replies

reply	meaning	counted as
235	accepted	sent
435	already there	dup
437	rejected	e437; not an error, the input can still be marked done
436	try later	retried with backoff, up to -max-tries (default 10)
lost connection or timeout		retried on a new connection, up to -max-tries
anything else		err; the input is not marked done

mbox2nntp also counts these as 437 without sending them:

• no Date: or no Subject: (in directory mode it adds Subject: -- and From: anonymous where missing);
• more than 256 header lines;
• a body over 1 MiB (also counted in too-big).

An article that runs out of -max-tries is logged as ERROR Worker N) code=... giving up msgid='...' after N tries and counts as err.

Marked done, or not

The input is renamed INPUT.done when every article has been answered and none failed. For a plain NAME.mbox with an archive beside it (NAME.mbox.gz, .zip or .7z), the archive is renamed instead. A directory input is never renamed.

It is not marked done when:

cause	what the log says
a dry run	dry run: '...' not marked done
an article ended in an error, or ran out of -max-tries	err= above 0 in the summary
no worker could connect, so articles were never sent	ERROR: N articles were never sent, not marking done
the parser stopped before the end of the file	stopped-early=true, and [iterateMbox] ERROR: mbox parser stopped early ... not marking done
a body could not be read to the end	[iterateMbox] ERROR: ... reading the body: ..., not marking done
every message was dropped, none offered	ERROR: all N messages dropped, none offered, not marking done

Rerun it after fixing the cause. Articles the server already has come back as 435 and cost little. An article is never sent cut short.

Exit codes

code	when
0	a run finished, whether or not the input was marked done
1	a fatal error: an input that cannot be found, opened or unpacked, a -tmpdir that does not exist, a -dos2unix that cannot be found, no 7-Zip for a .7z input, no zstd for a UAT archive
2	a command-line error

Scripts should look for the .done file, not the exit code.