Skip to content

v0.2.3

Choose a tag to compare

@github-actions github-actions released this 22 Oct 06:36

Each archive holds mbox2nntp, uatexport, ncexport and iafetch for one system, with README and LICENSE; unpack and run, nothing to install. Check downloads against the _checksums.txt file (sha256sum -c).

.gz, .zip, mbox and directory inputs, ncexport and iafetch need no other program. .7z inputs need 7-Zip, UAT archives (.usenet) need zstd, and .usenet.xz needs xz or 7-Zip: see the README.

Changes since v0.2.2

Fixes from a code review. If you run mbox2nntp with the default tmpdir (.), update: v0.2.2 can delete a NAME.mbox that sits next to the NAME.mbox.gz you import.

mbox2nntp

  • Archives are unpacked into a new directory of the run's own, tmpdir/.mbox2nntp-*, and only that directory is removed. It is removed whatever the outcome, also on exit 1 and on SIGINT or SIGTERM. Before, an unpacked NAME.mbox replaced and then deleted a file of that name next to the archive, and two runs sharing a tmpdir overwrote each other's copy.
  • A worker adds its counts to the totals before it reports done. Before, a run could report 0/0, or miss its last error and rename the input to .done.
  • Header lines are dot-stuffed too, as RFC 3977 3.1.1 asks. Before, a header line starting with . lost the dot at the server.
  • The Message-ID comes from the first Message-ID field only. Two such fields used to give <a@b> <a@b>. Over the 5.28 million messages of the test corpus, 3 IDs change.
  • A dropped header field (Message-ID copy, Expires, a line without a colon) takes its continuation lines with it, so they no longer join the field before. A continuation line of only spaces no longer drops the article.
  • A message with headers and no body no longer merges into the next one.
  • An article is never sent cut short. A read error stops the file, and the input is not marked done.
  • Size limits:
    • An article over 1 MiB is not sent. It is counted as a 437 and as too-big, and the input can still be marked done.
    • A line over 1 MiB stops the file, and the input is not marked done.
    • In directory mode, files over 1 MiB are not read.
    • In UAT archives, a message over 1 MiB is skipped and counted. Before, a 99 KB crafted .usenet could make mbox2nntp allocate 3 GiB.

uatexport and ncexport

  • Body lines starting #! rnews are quoted with >, like From lines. Before, the importer split such articles there.
  • uatexport uses the archive's group name for the default output file only if it is a safe file name. Before, a name like ../../x could write anywhere.
  • ncexport checks the gzip trailer of a .tar.gz, so a damaged tarball no longer passes unnoticed.
  • ncexport skips and counts .artN files over 1 MiB and .db files over 256 MiB. uatexport skips and counts messages over 1 MiB.

iafetch

  • A download that receives no data for -stall (default 2m) is given up and retried, resuming the .part. Before, a stalled server could hang the whole run.
  • No panic when two workers hit different kinds of database error.
  • A file X is not downloaded through X.part when the item also has a file named X.part. Before, the two overwrote each other.
  • A .part longer than the file restarts with a fresh MD5 state. Before, it cost a second download.

Full Changelog: v0.2.2...v0.2.3