NewsCache 1.3.22

Upgrading

• The upstream news server's password was written to the log (see Bug
  fixes). Check old logs, and change the password if others could read
  them.
• The active database becomes .active64 at the first start; the old
  .active is kept as .active.v1. Older versions can still open the
  spool, but drop each group's cached articles the first time they use
  it; to go back to 1.3.21, rename .active.v1 to .active again.
• "ListenTo DEFAULT" now listens on IPv6 as well as IPv4. Check your
  firewall, or use "ListenTo 0.0.0.0" for IPv4 only.
• ARTICLE, HEAD, BODY and STAT by message-ID, LISTGROUP and XPAT now
  check the Read list of the client's entry, and clients that a login
  can't help get 502 instead of 480.
• LISTGROUP answers "211 count first last group", as RFC 3977 says.
• In the prefetch file of updatenews, a line starting with ! now
  excludes groups; it used to be ignored.
• newscache -p no longer prints the upstream servers' Password; put it
  back into a config made from -p output.
• The spool databases take the disk space of their full size instead of
  being sparse files (with History on, about 8 MB more for the history),
  so that a full disk is an error instead of a crash.
• Packagers: configure.in is now configure.ac (autoconf 2.69 or later)
  and the manual doc/NewsCache.texi; the end-to-end tests need Go 1.26
  and are skipped with an older Go.

New functions

• XPAT (RFC 2980 2.9): "XPAT header range| pattern" lists
  the articles whose header matches the wildmat pattern, in the format of
  XHDR. Like XHDR, it sees the overview fields of every article and all
  headers of the articles in the spool; for other headers of articles
  that are only in the overview, nothing matches. XPAT is in HELP and in
  LIST EXTENSIONS ("pat").
• The prefetch file of updatenews takes wildmat patterns: a line like
  "de.*" or "comp.lang.[cg]o" prefetches every matching group of the
  active list, and a line "!pattern" (or "!group") leaves the matching
  groups out, wherever they come from in the file. Plain group names work
  as before, even for groups the active list doesn't have. Empty lines and
  lines starting with # are skipped, and each group is prefetched once.
  A line starting with ! used to be ignored; it now excludes.
• HDR and LIST HEADERS (RFC 3977 8.5, 8.6). By article number HDR reads
  the overview: Subject, From, Date, Message-ID, References, Xref and the
  metadata :bytes and :lines; other headers get 503, as the RFC asks. By
  message-ID it fetches the article, so every header is there (and
  :bytes). An article without the header gets a line with an empty value,
  and a range without articles 423.
• Articles by message-ID from the spool, with the new option
  "History on" (off by default: then NewsCache behaves as before). It
  keeps a history of the articles in its spool (the directory .history in
  the spool: 16 files of 65536 buckets, keyed by the MD5 hash of the
  Message-ID, like go-pugleaf's and INN's history), and ARTICLE, HEAD,
  BODY and STAT , and the message-ID forms of XHDR, XPAT and
  HDR, serve an article from there. An article the spool knows only from
  its overview record is fetched by number and stored, so a crossposted
  article is fetched once (but not into a group the client may not read);
  in offline mode, or with the news server down, articles in the spool
  can be read by message-ID too. Every message-ID request used to go to
  the news server. A spool of an older version gets its entries as
  clients read its articles. newscacheclean drops the entries of groups
  whose database it removed, counts the history toward SpoolSize and
  removes it when it alone takes more than half of SpoolSize (or History
  is off); a damaged history file is renamed to .bad and started anew.
  A history written by a development build since #56 (256 buckets per
  file, "history 1") is moved aside and started anew.

Bug fixes

• Articles bigger than about 15 KB, which NewsCache stores in their own
  .art file, were never deleted from the spool after they expired on the
  upstream server: the database entry went, the file stayed. Two bugs in
  CNewsgroup: the file name was built as .artd instead of
  .art, and the overview update skipped every article after a range
  it had just fetched, so it never noticed the expired ones there.
• One group name longer than 511 characters in the upstream server's
  active list made NewsCache give up reading the list: clients saw no
  groups at all (LIST empty, GROUP 411 for every group), the database
  stayed locked and the rest of the list was left unread on the
  connection. Such lines are now logged and skipped.
• XOVER and XHDR accepted a range that doesn't start with a number
  ("XOVER -5", "XHDR subject -3") and used an uninitialized first article
  number for it; such ranges now get 501. For an open range ("XOVER 3-")
  the XHDR status line showed the last number as -1, and it showed no
  header name; it now reads
  "221 subject header follows [3-18446744073709551615]".
• The login examples added in 1.3.21 (FEDORA.md and the commented entry in
  the container config) did not require a login: with "allow read post"
  and "Authentication file:", clients that don't log in still read and
  post, and a login only adds rights. The examples now use
  "allow authentication" with "List/Read/PostTo !*" and give the rights in
  the user file, whose format (user:hash:list:read:postto:allow) FEDORA.md
  showed wrongly as user:password. The shipped configs were not affected:
  they let nobody in.
• Before logging in, a client of an entry with Authentication got
  "500 What?" for commands that only a login allows, e.g. every command
  with "allow authentication", or POST with "allow read". Newsreaders wait
  for 480 before they send AUTHINFO, so they couldn't log in. Such commands
  now get "480 Authentication required". Unknown commands, and entries
  without Authentication, still get 500.
• newscache -p and the xdebug command printed "allow authentication" as a
  bare "allow" (and "allow read authentication" as "allow read"). Read
  back as a config, a bare "allow" is allow none, so a login-only entry
  copied from the -p output let nobody in.
• Started as root (by hand, from inetd or from cron), newscache,
  newscacheclean and updatenews switched to Username and Groupname but
  kept root's supplementary groups, group 0 among them; with only a
  Username, also root's primary group. They now set the supplementary
  groups to the user's (initgroups) and, without a Groupname, use the
  user's primary group. A service that systemd starts as root has no
  supplementary groups, so there only the Username-only case was affected.
• In the allow field of an auth-file line (and the last field of
  "Authentication unix:" and "pam:"), "!read", "!post" and "!debug" set
  every other right instead of removing that one: "read,!post" gave a
  logged-in user debug (XDEBUG, which shows the configuration), and "!post"
  didn't take away the post right of the Client entry. They now remove only
  the named right.
• Article numbers above 2^31 were cut down to 32 bits. RFC 3977 stops at
  2^31-1, but the big binary providers passed that long ago (alt.binaries
  groups are above 6*10^10). The numbers went wrong in the upstream
  server's GROUP reply, in OVER and ARTICLE requests to it, in the
  overview records of the spool, in ARTICLE/HEAD/BODY/STAT/XOVER/XHDR and
  in newscacheclean, so such groups showed wrong numbers or an empty
  XOVER. Article numbers are 64-bit now.
  The active database is now the file .active64. An existing .active is
  converted once at the first start (each group keeps its creation time,
  which NEWGROUPS reports) and renamed to .active.v1, which can then be
  deleted.
  A number with trailing characters ("ARTICLE 12x") or one that doesn't
  fit in 64 bits now gets 501; atoi read the leading digits or overflowed.
  On 32-bit systems the spool can't index numbers above 4294967295; such
  articles are passed through from the upstream server without caching.
• STAT with the number of an article that doesn't exist (a gap in the
  group, or past its last article) answered "223 status",
  without a Message-ID, instead of 423. And a successful STAT
  didn't make that article the current one, so NEXT, LAST and a following
  HEAD or BODY without a number went on from the previous article
  (RFC 3977 6.2.4).
• XHDR copied the header name into a 512-byte buffer on the stack, but a
  command line can be 1024 characters long: a client with read access
  could overflow it with a long header name and crash its session (or
  worse, where the stack isn't protected). The name is now a string.
• An auth-file line with only user and password ("alice:$6$...") was
  refused, so that user could never log in; at least the List field had
  to follow. Such lines are valid now: fields missing at the end add
  nothing, like empty ones, so the user keeps the rights of the Client
  entry. The sample newscache.conf-dist also showed the file format
  without its List and allow fields in one place.
• "Authentication unix" never read /etc/shadow: configure's default
  (--with-shadow=auto) didn't enable shadow password support even where
  shadow.h exists, so the password was compared against the "x" in
  /etc/passwd and every login failed, silently. Shadow support is now on
  by default where shadow.h exists. Note that NewsCache checks passwords
  after it has switched to its Username (news), and only root may read
  /etc/shadow, so unix authentication still can't check local accounts;
  the same is true of PAM with pam_unix. NewsCache now logs why ("cannot
  read the shadow password", and PAM's error with a hint about pam_unix),
  and the manual, newscache.conf(5), newscache.conf-dist and FEDORA.md
  describe the limitation and what works instead: Authentication file:,
  or PAM modules that work without root (pam_sss, pam_ldap, pam_pwdfile).
• newscache -p and "xdebug dump configuration" printed the Password of
  each upstream server in plain text. -p output often ends up in bug
  reports, and xdebug sends it to the client over the network. Both now
  print "# Password set, not shown" instead. Read back as a config, that
  line is a comment, so the upstream login fails (the server answers
  AUTHINFO PASS with an error) until the password is put back.
• newscache, newscacheclean and updatenews copied the -c argument (the
  config file name) with strcpy into a buffer of MAXPATHLEN (4096)
  characters, so a longer argument wrote past its end. Such an argument is
  now refused with "configuration file name is too long" and exit status 1.
  Only whoever starts the programs controls the argument.
• A SpoolDirectory longer than the limit (MAXPATHLEN minus room for the
  longest group name, 3568 characters on Linux) passed the config check,
  and then every session ended with "terminate called after throwing an
  instance of 'AssertionError'", without a reply. It is now a config error
  ("SpoolDirectory too long"), reported at startup by newscache,
  newscacheclean and updatenews (exit status 2), and the check no longer
  depends on --enable-assertions.
• "ListenTo DEFAULT" listened on IPv4 only: the daemon took the first
  wildcard address getaddrinfo returned, which is 0.0.0.0 on Linux, so
  IPv6 clients could not connect. DEFAULT (and "DEFAULT:port") now listens
  on one IPv6 socket that accepts IPv4 clients too, and falls back to IPv4
  where the host has no IPv6. IPv4 clients are still logged, matched
  against Client entries and named in NNTP-Posting-Host by their plain
  IPv4 address. To listen on IPv4 only, use "ListenTo 0.0.0.0".
• The pid file (PidFile) was never removed: after the daemon stopped, it
  still held the old pid. newscache now removes it when it stops on
  SIGTERM or SIGINT, if it still holds its own pid. The file is written as
  root, before the switch to Username, so removing it needs a directory
  that Username can write to (e.g. /run/newscache owned by news); in /run
  itself the file is emptied instead. The configs for systemd (Debian
  package, FEDORA.md) and the container no longer set a PidFile, since the
  service manager knows the pid; without PidFile no file is written and
  nothing is logged (it used to warn "cannot open pid file").
• Overview records from the upstream server were converted with the tab
  positions in a fixed array of 256, so a record with more fields than
  that overflowed the stack. Servers replace tabs in header values with
  spaces, so only a broken or hostile upstream sends such a record. The
  conversion only runs when the upstream's overview format differs from
  NewsCache's.
• The size fields that RFC 3977 servers list as ":bytes" and ":lines" in
  LIST OVERVIEW.FMT matched none of NewsCache's fields ("Bytes:",
  "Lines:"), so with such a server the Bytes and Lines fields of XOVER
  were empty. And a field the upstream sends with its name (e.g.
  "Subject:full") lost the name but kept the colon (": text").
• LISTGROUP with a group the client may not read (Read in its Client
  entry) answered 480, but only after it had made that group the current
  one; XOVER, XHDR, ARTICLE and the other commands for the
  current group then read it, so "LISTGROUP secret.group" followed by
  "XOVER" showed the group. LISTGROUP now checks first.
• ARTICLE, HEAD, BODY and STAT with a message-ID didn't check the Read
  list at all: a client that may read only some groups got any article
  whose message-ID it knew. The article must now be in at least one group
  the client may read (its Newsgroups header).
• A group or article the client may not read got 480 even when no login
  could help (no Authentication in its entry, or already logged in). It
  now gets 502, as RFC 4643 says; 480 stays for clients that can log in.
• The PostTo check split the Newsgroups header wrongly: with three or more
  groups a name was checked with the following ones appended, which
  matched only if PostTo listed the same groups in the same order, and
  the space after a comma was kept, so such posts were refused. A refused
  group now gets 441 once the client is logged in (or can't log in), not
  480.
• The first XOVER (or XHDR, LISTGROUP, STAT, NEXT, LAST) on a group made
  NewsCache synchronize the whole group, whatever range the client asked
  for: a LISTGROUP of the whole group from the upstream server, the
  overview records of every article it didn't have, a 1-byte-per-number
  table in memory and a spool index of 8 bytes for every number from the
  group's first to its last, written out in full. For a binary group with
  numbers from 60000000000 to 67486767815 that is a 57 GB index and 7.5 GB
  of memory, so such groups failed ("412 operation failed") or filled the
  disk. Groups without caching (Options not-cached) fetched the overview
  of the whole group at every GROUP.
  NewsCache now asks for the numbers a client asks for, in steps of 10000
  ("LISTGROUP group a-b", RFC 3977, then OVER for the missing records), and
  compares each step with the upstream server again only after its
  GroupTimeout. The spool's index covers what clients read, at most the
  newest 10000000 numbers of a group; articles further back are passed
  through without caching. LISTGROUP from a client passes the upstream
  server's list on as it arrives, takes a range ("LISTGROUP group 100-200")
  and answers "211 count first last group" as RFC 3977 says (it was
  "211 list of article numbers follow"). updatenews -g and -o prefetch the
  newest 10000000 numbers of each group. Spools of older versions keep
  working. Older versions open the new spools, but drop a group's cached
  articles the first time they use it (its index now starts above the
  server's first number, which they take for a renumbered group); the
  .art files of big articles stay until newscacheclean needs the space.
  When the server's last number goes down (its newest articles cancelled
  or expired, or a server farm whose frontends lag behind), only the
  cached articles above it are dropped; numbers far above, up to 2^64,
  can't make a session loop.
  An OVER/XOVER range without articles (423, or 420 from older servers)
  made NewsCache assume the upstream server has no OVER, then no XOVER,
  and give up on the overview; it now means no records.
• CAPABILITIES listed READER, POST and, for entries with Authentication,
  AUTHINFO USER to every client: also to clients without the right to
  read or post, and AUTHINFO also after the login (RFC 4643 says not
  then). It now lists only what the client can do at that moment, and
  adds HDR, OVER and LIST HEADERS for readers.
• XPAT with a message-ID ("XPAT subject *") returned the header of
  any article whose message-ID the client knew, also in groups its Read
  list doesn't allow (XPAT came in this release, the Read check of
  ARTICLE after it). It now checks like ARTICLE, as XHDR and HDR do.
• XHDR takes a message-ID ("XHDR subject ", RFC 2980 2.6); it
  answered 501. The line starts with the message-ID, and no group needs
  to be selected.
• A damaged database in the spool (a group's .db, or .active64) could
  crash NewsCache. Damage behind the header crashed every session that
  touched the group (SIGSEGV), a short or damaged .active64 aborted every
  session before the greeting, a short .db gave "412 operation failed"
  for good, and newscacheclean ended with SIGABRT at a damaged .db, before
  it had removed anything. The offsets and sizes in these files are now
  checked before they are used; a damaged file is moved to name.bad (one
  error line in the log), a new one is started and filled from the
  upstream server again, and the command that ran into it is retried.
  newscacheclean goes on with the other groups and removes the .bad
  files. A command that fails with any other unexpected error now gets a
  503 instead of ending the session (e.g. a stored article that doesn't
  parse; in inetd mode that aborted with SIGABRT).
  Processes that run into the same damaged file at once move it aside
  only once (under a lock on its directory) and all go on with the same
  new file; one could move another's new file aside, and the updates
  made there were lost. Opening a database checks it under a lock, so a
  sound file that another process is writing to is no longer taken for
  damaged. If no new file can be started (no file descriptor left, no
  space), the database answers errors until it can, instead of using
  the old mapping (a crash).
• CachePort never worked: the check of its first character was the wrong
  way round since the IPv6 support of 2008, so "CachePort nntp" listened
  on the service "ntp" read as a number and "CachePort #12000" passed the
  "#" on, and the daemon couldn't start. It also took precedence over
  ListenTo, which the documentation says it doesn't. It now takes a
  service name, a number or #number, and is ignored (with a log line)
  when ListenTo is set; newscache -p no longer adds a ListenTo line that
  would hide it.
• DATE with an argument answered "501 Syntax: group Newsgroup"; it now
  gives DATE's own syntax.
• An upstream server that returns at most so many overview records per
  OVER, whatever the range asked for, cut groups short: NewsCache asks
  for a run of up to 10000 numbers, took everything after the last record
  it got for articles without a record, and clients saw only the first
  part of the range (until the next sync, which went the same way). It
  now asks again after the last record received until the range is done.
  A server that returns whole ranges costs at most one more request
  (answered 423), when a range ends in numbers without articles.
• When the upstream server asked for the login again in the middle of a
  session (480, as servers do when a login times out), NewsCache took the
  480 to an OVER for "no OVER", then the one to XOVER for "no XOVER", and
  gave up on the overview: clients got an empty XOVER without an error
  until the session ended; other commands failed with the 480. NewsCache
  logged in only when it connected. It now logs in again (with the
  server's User and Password) and sends the command once more.
• The password of the upstream news server was written to the log:
  every command to the server is logged at level info ("RServer::issue:
  issue ..."), "authinfo pass " too. With syslog that is the
  news facility, which Debian's default rsyslog also writes to
  /var/log/syslog (readable by group adm), and the systemd journal; with
  --without-syslog, newscache.log. AUTHINFO to the server is now logged
  without its arguments, as the client's AUTHINFO already was. Check old
  logs, and change the password if they were readable by others.
• Once a client had read an article (up to 15 kB, kept in the group
  database), its overview record was made from the article: :lines was
  empty when the article has no Lines header (most articles; RFC 5536
  made it obsolete), XHDR Lines gave "(none)" and HDR :lines nothing, and
  :bytes counted the dots added for dot-stuffing (RFC 3977 counts the
  article without them). :lines now counts the body lines, and :bytes
  leaves those dots out, also for HDR :bytes .
• With several news servers, the Xref of an article lost the groups that
  another server serves (their numbers are the other server's), but the
  Xref of its overview record kept them: a newsreader that marks
  crossposts read by the overview's Xref marked articles of the other
  group that have those numbers on the other server. Overview records
  from a server now get the same Xref as its articles.
• A full spool crashed NewsCache with SIGBUS: the databases (group
  databases, the active list, the history) grow with ftruncate, which
  leaves a hole, and are written through mmap, where a hole the file
  system cannot fill is a signal, not an error. With History on it
  happened at the first history file. The space is now taken when a
  database grows (posix_fallocate, or zeros written where there is
  none; the file size limit counts too), so a full disk is an error,
  and clients hardly notice it: articles and overview records that
  can't be kept are passed on from the news server (a 423 or a cut
  XOVER before), the active list is read to its end so the connection
  to the news server stays in step, and every database lock is given
  back when the error comes (a lock left held would have kept every
  other client of that database waiting until the session ended).
  Nothing is left half written: a database record is replaced only
  after the new one is allocated, and an article whose own file (over
  15 kB) could not be written completely was kept cut off and served
  so from then on; such a file is now removed. A file that is empty
  or can't be opened (no file descriptor left) is no longer served as
  an article ("503 internal error"): the article is fetched again.
  The databases now take the disk space of their size (the history's
  16 files about 8 MB) instead of being sparse.

Other changes

• Spool, config and log file paths are built with bounded snprintf instead
  of sprintf, group2dir() checks the group name's length, and a group whose
  spool path wouldn't leave room for its article files is refused. These
  overflows needed an extremely long SpoolDirectory or LogDirectory, so
  they were not reachable from outside. The buffers for generated
  Message-IDs are sized from the host name limit. Together this removes
  the -Wformat-overflow and -Wformat-truncation warnings.
• Debian packaging brought up to date for Debian 13 and Ubuntu 24.04:
  "dpkg-buildpackage -us -uc -b" builds a newscache package with GnuTLS
  and PAM that runs as user news under systemd, cleans the spool daily
  with a timer and ships an /etc/newscache.conf that lets nobody in. See
  debian/README.Debian.
• The info manual has a dir entry, and newscache.conf(5) a whatis line.
• The end-to-end tests use nntp-testkit 0.5.0 and need Go 1.26 or later.
  With an older Go, make check now skips them with a message, instead of
  having Go try to download a newer toolchain (which fails offline). Set
  GOTOOLCHAIN to let Go fetch one anyway. The fake upstream server can
  now behave like real servers that don't follow RFC 3977 to the letter,
  and tests/e2e/upstream_test.go runs NewsCache against them: LISTGROUP
  that rejects or ignores a range, only OVER or only XOVER, 420 or an
  empty 224 for an empty range, a last number that goes down (cancels,
  lagging frontends of a server farm), groups whose articles all expired
  ("211 0 0 0"), a connection limit, unusual message-IDs, and clients
  that pipeline their commands. tests/e2e/upstreams_test.go adds a group
  of 10^7 articles (generated on demand), an overview record whose
  Message-ID is not the article's, and two servers that number the same
  group differently; tests/e2e/spoolfull_test.go runs NewsCache with few
  file descriptors, with writes that fail (ulimit -f) and on a spool that
  fills up (a small tmpfs, where user namespaces are allowed).
• doc/websites/ holds an offline copy, from the Wayback Machine, of the
  two original NewsCache websites (Thomas Gschwind, TU Wien; Herbert
  Straub, linuxhacker.at), the 1.2rc6 tarball, patches and man pages they
  link to, and an index page in every folder so it can be served as a
  website.
• The compiler warnings are gone, apart from autoconf/automake ones and one
  that the XOVER range fix removes (GCC 16 with Fedora's flags: 56 before).
  Most were harmless (catch by value, unused variables, a never-NULL array
  checked against NULL). A few fixes change behaviour in cases that
  shouldn't happen: a corrupt active database with a broken record chain
  no longer dereferences NULL, a failed read of an upstream reply no longer
  checks an uninitialized status, updatenews no longer reads an undefined
  exit status when wait() fails, LimitGroupSize ignores a group whose
  last article number is below its first, and builds without syslog stop
  with an error if LogDirectory is too long for the log file name.
• New option AuthFailureDelay: the seconds NewsCache waits before it
  answers a failed login, multiplied by the number of failed logins in the
  connection. The default stays 15 (15, 30, 45 seconds); 0 answers at
  once, the maximum is 3600. It was fixed at 15 seconds.
• configure.in is now configure.ac and the manual doc/NewsCache.texi
  (was doc/NewsCache.texinfo), and the obsolete autoconf and automake
  macros are replaced, so autogen (autoreconf) and configure run without
  warnings (21 with autoconf 2.73 and automake 1.18 before). The version
  is set in AC_INIT now. PACKAGE and VERSION, the tarball name and the
  configure options are unchanged; config.h also defines PACKAGE_NAME,
  PACKAGE_VERSION and the like now, and configure --help names the issue
  tracker. Autoconf 2.69 or later is needed. Packagers who patch
  configure.in or refer to NewsCache.texinfo need to follow the renames.
• The release tarball has the notes in doc/ again (newscache-pam.txt,
  newscache-stunnel-ssl.txt, ...), which README.md points to, and the NNTP
  specifications there. doc/Makefile.am had no backslash after
  "EXTRA_DIST =", so its list was ignored, and it named two drafts that
  were no longer in the tree.
• Documentation: the manual's chapter "Access Control" described only
  libwrap; it now explains the AccessList (nothing is allowed by default,
  Client and Default entries, rights, groups, logins) with an example.
  The login example in newscache.conf-dist made the login optional
  ("allow read post"); it now uses "allow authentication" with the rights
  in the user file, as FEDORA.md does. debian/README.Debian still said an
  auth-file line needs the list field. newscache(8) lacked -n and what -c
  does. The FAQ says what to do after moving groups to another news
  server: remove their databases, or NewsCache may serve the old server's
  cached articles under the new server's numbers.