NewsCache 1.3.22

Upgrading

• The upstream news server's password was written to the log (see Bug
  fixes). Check old logs, and change the password if others could read
  them.
• The active database becomes .active64 at the first start; the old
  .active is kept as .active.v1. Older versions can still open the
  spool, but drop each group's cached articles the first time they use
  it; to go back to 1.3.21, rename .active.v1 to .active again.
• "ListenTo DEFAULT" now listens on IPv6 as well as IPv4. Check your
  firewall, or use "ListenTo 0.0.0.0" for IPv4 only.
• ARTICLE, HEAD, BODY and STAT by message-ID, LISTGROUP and XPAT now
  check the Read list of the client's entry, and clients that a login
  can't help get 502 instead of 480.
• LISTGROUP answers "211 count first last group", as RFC 3977 says.
• In the prefetch file of updatenews, a line starting with ! now
  excludes groups; it used to be ignored.
• newscache -p no longer prints the upstream servers' Password; put it
  back into a config made from -p output.
• The spool databases take the disk space of their full size instead of
  being sparse files (with History on, about 8 MB more for the history),
  so that a full disk is an error instead of a crash.
• Packagers: configure.in is now configure.ac (autoconf 2.69 or later)
  and the manual doc/NewsCache.texi; the end-to-end tests need Go 1.26
  and are skipped with an older Go.

New functions

• XPAT (RFC 2980 2.9): "XPAT header range| pattern" lists
  the articles whose header matches the wildmat pattern, in the format of
  XHDR. Like XHDR, it sees the overview fields of every article and all
  headers of the articles in the spool; for other headers of articles
  that are only in the overview, nothing matches. XPAT is in HELP and in
  LIST EXTENSIONS ("pat").
• The prefetch file of updatenews takes wildmat patterns: a line like
  "de.*" or "comp.lang.[cg]o" prefetches every matching group of the
  active list, and a line "!pattern" (or "!group") leaves the matching
  groups out, wherever they come from in the file. Plain group names work
  as before, even for groups the active list doesn't have. Empty lines and
  lines starting with # are skipped, and each group is prefetched once.
  A line starting with ! used to be ignored; it now excludes.
• HDR and LIST HEADERS (RFC 3977 8.5, 8.6). By article number HDR reads
  the overview: Subject, From, Date, Message-ID, References, Xref and the
  metadata :bytes and :lines; other headers get 503, as the RFC asks. By
  message-ID it fetches the article, so every header is there (and
  :bytes). An article without the header gets a line with an empty value,
  and a range without articles 423.
• Articles by message-ID from the spool, with the new option
  "History on" (off by default: then NewsCache behaves as before). It
  keeps a history of the articles in its spool (the directory .history in
  the spool: 16 files of 65536 buckets, keyed by the MD5 hash of the
  Message-ID, like go-pugleaf's and INN's history), and ARTICLE, HEAD,
  BODY and STAT , and the message-ID forms of XHDR, XPAT and
  HDR, serve an article from there. An article the spool knows only from
  its overview record is fetched by number and stored, so a crossposted
  article is fetched once (but not into a group the client may not read);
  in offline mode, or with the news server down, articles in the spool
  can be read by message-ID too. Every message-ID request used to go to
  the news server. A spool of an older version gets its entries as
  clients read its articles. newscacheclean drops the entries of groups
  whose database it removed, counts the history toward SpoolSize and
  removes it when it alone takes more than half of SpoolSize (or History
  is off); a damaged history file is renamed to .bad and started anew.
  A history written by a development build since #56 (256 buckets per
  file, "history 1") is moved aside and started anew.

Bug fixes

• Articles bigger than about 15 KB, which NewsCache stores in their own
  .art file, were never deleted from the spool after they expired on the
  upstream server: the database entry went, the file stayed. Two bugs in
  CNewsgroup: the file name was built as .artd instead of
  .art, and the overview update skipped every article after a range
  it had just fetched, so it never noticed the expired ones there.
• One group name longer than 511 characters in the upstream server's
  active list made NewsCache give up reading the list: clients saw no
  groups at all (LIST empty, GROUP 411 for every group), the database
  stayed locked and the rest of the list was left unread on the
  connection. Such lines are now logged and skipped.
• XOVER and XHDR accepted a range that doesn't start with a number
  ("XOVER -5", "XHDR subject -3") and used an uninitialized first article
  number for it; such ranges now get 501. For an open range ("XOVER 3-")
  the XHDR status line showed the last number as -1, and it showed no
  header name; it now reads
  "221 subject header follows [3-18446744073709551615]".
• The login examples added in 1.3.21 (FEDORA.md and the commented entry in
  the container config) did not require a login: with "allow read post"
  and "Authentication file:", clients that don't log in still read and
  post, and a login only adds rights. The examples now use
  "allow authentication" with "List/Read/PostTo !*" and give the rights in
  the user file, whose format (user:hash:list:read:postto:allow) FEDORA.md
  showed wrongly as user:password. The shipped configs were not affected:
  they let nobody in.
• Before logging in, a client of an entry with Authentication got
  "500 What?" for commands that only a login allows, e.g. every command
  with "allow authentication", or POST with "allow read". Newsreaders wait
  for 480 before they send AUTHINFO, so they couldn't log in. Such commands
  now get "480 Authentication required". Unknown commands, and entries
  without Authentication, still get 500.
• newscache -p and the xdebug command printed "allow authentication" as a
  bare "allow" (and "allow read authentication" as "allow read"). Read
  back as a config, a bare "allow" is allow none, so a login-only entry
  copied from the -p output let nobody in.
• Started as root (by hand, from inetd or from cron), newscache,
  newscacheclean and updatenews switched to Username and Groupname but
  kept root's supplementary groups, group 0 among them; with only a
  Username, also root's primary group. They now set the supplementary
  groups to the user's (initgroups) and, without a Groupname, use the
  user's primary group. A service that systemd starts as root has no
  supplementary groups, so there only the Username-only case was affected.
• In the allow field of an auth-file line (and the last field of
  "Authentication unix:" and "pam:"), "!read", "!post" and "!debug" set
  every other right instead of removing that one: "read,!post" gave a
  logged-in user debug (XDEBUG, which shows the configuration), and "!post"
  didn't take away the post right of the Client entry. They now remove only
  the named right.
• Article numbers above 2^31 were cut down to 32 bits. RFC 3977 stops at
  2^31-1, but the big binary providers passed that long ago (alt.binaries
  groups are above 6*10^10). The numbers went wrong in the upstream
  server's GROUP reply, in OVER and ARTICLE requests to it, in the
  overview records of the spool, in ARTICLE/HEAD/BODY/STAT/XOVER/XHDR and
  in newscacheclean, so such groups showed wrong numbers or an empty
  XOVER. Article numbers are 64-bit now.
  The active database is now the file .active64. An existing .active is
  converted once at the first start (each group keeps its creation time,
  which NEWGROUPS reports) and renamed to .active.v1, which can then be
  deleted.
  A number with trailing characters ("ARTICLE 12x") or one that doesn't
  fit in 64 bits now gets 501; atoi read the leading digits or overflowed.
  On 32-bit systems the spool can't index numbers above 4294967295; such
  articles are passed through from the upstream server without caching.
• STAT with the number of an article that doesn't exist (a gap in the
  group, or past its last article) answered "223 status",
  without a Message-ID, instead of 423. And a successful STAT
  didn't make that article the current one, so NEXT, LAST and a following
  HEAD or BODY without a number went on from the previous article
  (RFC 3977 6.2.4).
• XHDR copied the header name into a 512-byte buffer on the stack, but a
  command line can be 1024 characters long: a client with read access
  could overflow it with a long header name and crash its session (or
  worse, where the stack isn't protected). The name is now a string.
• An auth-file line with only user and password ("alice:$6$...") was
  refused, so that user could never log in; at least the List field had
  to follow. Such lines are valid now: fields missing at the end add
  nothing, like empty ones, so the user keeps the rights of the Client
  entry. The sample newscache.conf-dist also showed the file format
  without its List and allow fields in one place.
• "Authentication unix" never read /etc/shadow: configure's default
  (--with-shadow=auto) didn't enable shadow password support even where
  shadow.h exists, so the password was compared against the "x" in
  /etc/passwd and every login failed, silently. Shadow support is now on
  by default where shadow.h exists. Note that NewsCache checks passwords
  after it has switched to its Username (news), and only root may read
  /etc/shadow, so unix authentication still can't check local accounts;
  the same is true of PAM with pam_unix. NewsCache now logs why ("cannot
  read the shadow password", and PAM's error with a hint about pam_unix),
  and the manual, newscache.conf(5), newscache.conf-dist and FEDORA.md
  describe the limitation and what works instead: Authentication file:,
  or PAM modules that work without root (pa...

NewsCache 1.3.21

Upgrading

• The shipped configs no longer let anyone in. If you copy
  newscache.conf-dist or use the container image, add a Client entry for
  your clients' addresses or networks, or an entry with Authentication
  (login), before Default.
• A Client entry with a /prefix after something that isn't an IP address
  (e.g. a host name like news.example.org/24) is now a config error, so
  such a config no longer loads. Before, the entry silently never matched.
  Check with newscache -p before restarting.

Bug fixes

• The sample newscache.conf-dist allowed inetd clients (Client stdin, with
  debug) and clients from localhost, and the container config allowed
  loopback and all private networks. Both now only contain
  Default { allow none }, with the old entries as commented examples, so
  nobody may read or post until a Client entry (address or CIDR) or a login
  allows it. FEDORA.md does the same and shows how to allow clients.
• newscache -p and the xdebug command printed AccessList entries for an
  address or network (e.g. Client 192.168.0.0/255.255.0.0) as "Default",
  without the address; read back as a config, such an entry would have let
  every client in. They now print as Client 192.168.0.0/16.
• On hosts where getaddrinfo considers IPv6 unavailable (seen in Docker
  containers), an IPv6 Client entry such as 2001:db8::/32 was not parsed as
  an address and silently became a host name that never matched. Addresses
  are now parsed without AI_ADDRCONFIG. A /prefix after something that
  isn't an IP address (e.g. 300.1.1.1/8) is now a config error instead of
  an entry that silently never matches.

Other changes

• Dockerfile for Docker and Podman (also Docker Desktop and Podman Desktop on
  Windows): builds on Ubuntu 24.04, runs as user news in the foreground and
  logs to the container output. Its config reads from
  news.blueworldhosting.com over TLS and lets nobody in until you mount a
  config with a Client entry or a login. CI builds the image and checks both.
• CI also builds and runs make check on Linux ARM64, Alpine 3.24 (musl),
  Debian 13 and FreeBSD.
• README.md: how to run NewsCache in a container and on Windows (WSL 2).

Build

tar xzf NewsCache-1.3.21.tar.gz && cd NewsCache-1.3.21
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make && make check && sudo make install

Or as a container: docker build -t newscache . (see README.md).

Tested on Ubuntu 24.04, Fedora 43, 44 and 45, Alpine 3.24 and Debian 13; CI also builds and tests on Linux ARM64, FreeBSD and macOS. See README.md and FEDORA.md.

SHA-256: 4cc3652a4cbcfb63235ce6af3b2e965f14de1387854604e86afc3e4e7387bb7f

NewsCache 1.3.20

Upgrading: with Options ssl or starttls, NewsCache now verifies the server's
TLS certificate and drops the connection if it is invalid, expired or for
another name, or if STARTTLS is refused. Check the log after upgrading. For a
server with a private CA set TLSCAFile; Options noverify accepts any
certificate, without protection against a man-in-the-middle.

Bug fixes

• TLS to upstream servers (Options ssl and starttls) now verifies the
  server's certificate: it must chain to a trusted CA, be valid and match the
  host name in the Server line. Before, any certificate was accepted, so
  anyone in the middle could impersonate the server.
• STARTTLS no longer falls back to plain text. If the server refused
  STARTTLS, NewsCache continued without TLS and sent User/Password in plain
  text, which anyone in the middle could force; now the connection is
  dropped. A failed TLS handshake after STARTTLS also ends the connection.
• A NewsCache built without GnuTLS ignored Options ssl and starttls and
  talked to such a server in plain text, login included. It now refuses to
  connect to it and logs why.
• TLS uses the system's trust store (it was a fixed Debian path, missing on
  Fedora and macOS), sends the host name as SNI, has a handshake timeout and
  logs why a handshake or verification failed.
• newscache -p left the server options ssl, starttls and dontgenmsgid out of
  its output, although they were parsed and used.

Other changes

• New server settings: TLSCAFile (CA certificates for servers with a
  private CA) and Options noverify (accept any certificate, logged as a
  warning). The man page now documents ssl and starttls, and the sample
  config no longer lists not-ssl and not-starttls, which never existed.
• Bug reports and the home page point to
  https://github.com/go-while/NewsCache instead of the dead members.aon.at,
  linuxhacker.at and TU Wien sites and mailing lists: in the error messages
  that ask for a report, the man pages, README, INSTALL (whose socket++
  instructions were obsolete), the texinfo manual, doc/ and debian/.
• NVClasses: NVlist.{h,cc} is now NVListBase.{h,cc} (class NVListBase).
  NVlist.h and NVList.h clashed on case-insensitive file systems, so
  checkouts on Windows and macOS were broken. The spool format is unchanged.
• src/Config.{h,cc} is now src/NewsCacheConfig.{h,cc} (the class is still
  Config). On case-insensitive file systems, #include "config.h" in src/
  found src/Config.h instead of the header that configure generates.
• NewsCache builds on macOS: configure checks for crypt.h (macOS and the
  BSDs declare crypt() in unistd.h), and wildmat.c no longer uses a K&R
  function definition, which C23 compilers such as Xcode's clang reject.
• CI: build and make check on macOS, and a check on Windows and macOS that
  the tree checks out cleanly on a case-insensitive file system.
• The sample newscache.conf-dist and FEDORA.md use news.blueworldhosting.com
  (port 563, TLS) instead of the long-gone TU Wien servers, and README.md
  lists public news servers.
• FEDORA.md: all steps tested on Fedora 43, 44 and 45.
• Removed stamp-h.in (an automake leftover) and the TODO file from 2013;
  its still-open items are in REVIEW.md.

Build

tar xzf NewsCache-1.3.20.tar.gz && cd NewsCache-1.3.20
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make && make check && sudo make install

Tested on Ubuntu 24.04 (g++ 13.3) and Fedora 43, 44 and 45 (g++ 15.3/16.2, Fedora's rpm flags); CI also builds and tests on macOS. See README.md and FEDORA.md.

SHA-256: 113f1bffee6dfc2d6a12b56689c35007fa57bf3b9f75379abb1b2f293dfdc8ac

NewsCache 1.3.19

Bug fixes

• NewsCache crashed at startup ("buffer overflow detected" with
  _FORTIFY_SOURCE, memory corruption without) on hosts whose fully
  qualified domain name is 64 characters or longer, as on cloud VMs such
  as GitHub's runners. The host name buffers were MAXHOSTNAMELEN (64)
  bytes, which is the kernel host name limit, not the DNS limit; they are
  now 256 bytes and the copies are bounded. Found by the new CI.

Other changes

• CI (.github/workflows/ci.yml): make check on Ubuntu (default with the
  race detector and make distcheck, file logging with debug, and
  AddressSanitizer) and on Fedora 43, 44 and 45 with Fedora's packaging
  flags.
• README: says that NewsCache is developed on GitHub and that the
  Subversion repository is archived (SVN r81 added a note there).

Upgrading from 1.3.18 needs no config changes.

Build

tar xzf NewsCache-1.3.19.tar.gz && cd NewsCache-1.3.19
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make && make check && sudo make install

Tested on Ubuntu 24.04 (g++ 13.3) and Fedora 43, 44 and 45 (g++ 15.3/16.2, Fedora's rpm flags). See README.md and FEDORA.md.

SHA-256: f50d128eab8e705d94929263a4f1f82147b27368a41d0f1787301c6b8dfbf155

NewsCache 1.3.18

Bug fixes

• Authentication: a missing optional field was read past the end of the
  field list, and the client session crashed after a successful login.
  This happened with "Authentication unix" or "pam" without the
  ":list:read:postto:flags" fields (documented in newscache.conf(5)), and
  with auth-file lines shorter than user:password:list:read:postto:flags.
  Missing fields are now empty.
• MD5 password hashes in auth files (base64, 24 characters ending in "==")
  crashed the session on 64-bit systems. md5lib used unsigned long for its
  32-bit words, which is 64 bits there, so computing the hash overran a
  stack buffer (and gave a wrong hash). It now uses uint32_t. Affects every
  release since at least 1.2rc6 on 64-bit systems; crypt() hashes such as
  $6$ were not affected. Found by CodeQL.
• DATE and NEWGROUPS use gmtime_r/localtime_r, and NEWGROUPS checks their
  result instead of dereferencing a NULL pointer on failure.

Other changes

• CodeQL code scanning (.github/workflows/codeql.yml) runs on pushes and
  pull requests to main and weekly.

Tests

• End-to-end tests for file authentication: a short auth-file line, an MD5
  hash, and a wrong password.
• End-to-end tests for DATE and NEWGROUPS.

Upgrading from 1.3.17 needs no config changes.

Build

tar xzf NewsCache-1.3.18.tar.gz && cd NewsCache-1.3.18
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make && make check && sudo make install

Tested on Ubuntu 24.04 (g++ 13.3, also with AddressSanitizer) and Fedora 44 (g++ 16.2, Fedora's rpm flags). See README.md and FEDORA.md.

SHA-256: 3bef38454c2f63dbecef41f6284d6d0b94840e8f68905055def00a70cb15ebb9

NewsCache 1.3.17

First release from https://github.com/go-while/NewsCache, which continues
NewsCache with Christof Meerwald's agreement. It is SVN trunk r80 (1.3.16)
plus the fixes and tests below.

Bug fixes

• Access list: a Client entry with a plain IP address and no mask (for
  example "Client 10.1.2.3") had prefix length 0 and matched every client of
  that address family. It now means exactly that host. Affects 1.3.ipv6 to
  1.3.16 (since the IPv6 support); configs with a mask or a host name are not
  affected, and 1.2rc6 matched such entries by name only.
• Access list: entries for one exact host (a.b.c.d/32, /255.255.255.255,
  /128) never matched, including the example in newscache.conf(5). A prefix
  longer than the address is now a syntax error.
• SIGHUP with a broken newscache.conf no longer kills the daemon; it logs the
  error and keeps the running configuration.
• Upstream articles without an Xref header are served instead of failing with
  423/430.
• "400 too many users" is no longer followed by a NUL byte.
• "502 You have no permission to talk" ends with CRLF instead of a bare LF.
• Log file timestamps (builds without syslog) showed the previous month.
• newscacheclean: "ng == NULL | nr < firstnr" read an uninitialized variable
  (| instead of ||); the result was the same.
• The sample newscache.conf-dist loads again: BindFrom only takes numeric
  addresses since the IPv6 support, and the sample used a host name.
• make dist works again (a misspelled header name in NServer-0.9.1, there
  since 1.2rc6), and make distcheck passes: the test programs are now
  check_PROGRAMS, so make clean removes them.
• configure.in carries the release version again; it still said 1.3.ipv6,
  so 1.3.16 greeted clients with "NewsCache 1.3.ipv6".

Tests

• make check passes again (cArticle, cNewsgroupFilter and cConfig were broken).
• New end-to-end tests in tests/e2e (Go, using
  github.com/go-while/nntp-testkit) run the real newscache binary against a
  fake upstream server. make check runs them when Go is installed and skips
  them otherwise.

Documentation

• README.md: project history, building, and a check of the old documentation.
• FEDORA.md: building, testing and installing on Fedora, with a systemd unit
  and a cleanup timer.

Build

tar xzf NewsCache-1.3.17.tar.gz && cd NewsCache-1.3.17
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make && make check && sudo make install

Tested on Ubuntu 24.04 (g++ 13.3) and Fedora 44 (g++ 16.2, Fedora's rpm flags). See README.md and FEDORA.md.

SHA-256: b72ed76b6a747d630a2544dc3e41d770e8d3d815d8c73fc6f97212549749d2f6