Caching and Large Groups

Caching and Large Groups

What NewsCache stores

Everything lives under SpoolDirectory (default /var/cache/newscache):

Path	Contents
.active64	The active list: every group with its first and last article number, flag and creation time
.newsgroups	Group descriptions for LIST NEWSGROUPS. NewsCache doesn't fetch them; create the file by hand (see the manual, "Bugs and Misfeatures").
a/b/c/.db	The database of group a.b.c: overview records and articles up to about 15 KB
a/b/c/.art<number>	Articles bigger than about 15 KB, one file each. Written as .art<number>.tmp, flushed to the disk and renamed, so the file is either whole or the old one; a .tmp that a crash left behind is removed by newscacheclean by its age
.history/0 … .history/f	The message-ID history, only with History on
.active.refresh, .fetch.lock	Lock files, no data: only one session at a time fetches the active list, or a range of 10000 numbers of a group's overview, from the upstream server; the others serve what the spool holds meanwhile. Backups can skip them, newscacheclean leaves them alone

Articles are stored only when a client reads them, and only for servers with Options cached (the default). Options not-cached passes a server's articles through.

When NewsCache asks the upstream server

Data	Refreshed after
The active list (LIST)	Timeouts (first value, default 3600 s)
A group's numbers (GROUP) and overview	the server's GroupTimeout (default 600 s)
An article	never: once stored, it is served from the spool

Within these times, clients get the cached data. A short active-list timeout makes new groups appear sooner; a long one makes newsreaders start faster.

If one upstream's active-list reply is incomplete, the combined list is retried after at most 60 seconds (sooner if the configured active-list timeout is shorter), rather than being treated as fresh. See the reply limits on Upstream Servers and TLS.

Large groups

NewsCache asks the upstream server only for the article numbers a client asks for, in steps of 10000 numbers. A client that reads the newest 500 headers of a group makes NewsCache fetch the overview records of those 10000 numbers (LISTGROUP group first-last, RFC 3977, then OVER or XOVER for the records it doesn't have), not the whole group. Each step is compared with the server again only after GroupTimeout; within that time only newer articles are fetched. Articles the server no longer lists are removed from the spool then. When the server's last number goes down (its newest articles cancelled or expired, or a server farm whose frontends lag behind), only the cached articles above it are dropped.

• A group's index in the spool covers at most the newest 10000000 article numbers that clients have read (80 MB). Articles further back are still served, but passed through without being cached.
• Some providers return at most so many overview records per OVER, whatever the range; NewsCache asks again after the last record it got.
• A client's LISTGROUP takes an optional range (LISTGROUP group 100-200). NewsCache asks the upstream in adaptive steps and passes each list on as it arrives, without caching it: 10000 numbers per step in a dense group, increasing up to 100000000 numbers across gaps. If no number reaches the client for five minutes, the command fails. Each upstream reply also has its own limits; see Upstream Servers and TLS.
• LimitGroupSize n (per server) still exists: clients then see only the newest n article numbers of each group. It is no longer needed to make large groups work.

Up to 1.3.21, the first XOVER on a group fetched the list of the whole group, and the index had 8 bytes for every number from the group's first to its last: 57 GB for a binary group with numbers from 60000000000 to 67486767815. A group database never shrinks; to get the space of such an old .db back, remove the group's .db and .art* files (not its subdirectories, which hold other groups); NewsCache fetches what clients ask for again.

Going back to 1.3.21 or older: they open the new spools, but drop a group's cached articles the first time they use it (its index now starts above the server's first number, which they take for a renumbered group). The .art files of big articles stay until newscacheclean needs the space.

64-bit article numbers

Article numbers are 64-bit, so groups above 2^31 and 2^32 (big binary providers are above 6×10^10) work in every command and in the spool.

• Active list format: the active list is the file .active64 since 1.3.22. At the first start, an existing .active is converted once (every group keeps its stored first-seen time, which NEWGROUPS uses) and renamed to .active.v1, which can then be deleted.
• 32-bit systems: the spool can't index numbers above 4294967295; such articles are passed through without caching.

Articles by message-ID: the history

By default, every request by message-ID (ARTICLE <id> and the like) goes to the upstream server, as in earlier versions. With

History on

NewsCache keeps a history of the articles in its spool: for each Message-ID, the groups that hold it and the article number there. Then ARTICLE, HEAD, BODY and STAT with a message-ID, and the message-ID forms of XHDR, XPAT and HDR:

• serve the article from the spool when it is there (also in offline mode and with the upstream server down);
• fetch it by number, and store it, when the spool has only its overview record, so an article crossposted to several groups is fetched once;
• otherwise go to the upstream server by message-ID.

The client's Read list applies as always: an article is served if the client may read one of the groups in its Newsgroups header, and it is never fetched by number into a group the client may not read (it comes by message-ID then).

The history is filled when overview records and articles are stored, and when an article is read from the spool, so a spool of an older version gets its entries as clients read it. Entries go when NewsCache drops the article, and newscacheclean removes the entries of groups whose database it removed. Every lookup checks that the record at that number still has that Message-ID, so a stale entry can't serve a wrong article.

The layout follows go-pugleaf's history (INN's works similarly): 16 files .history/0 to .history/f. The MD5 hash of the Message-ID in hex picks the file (first digit) and one of its 65536 buckets (next four digits). A record holds the Message-ID and its locations, group:number,.... Each file has its own lock and a record marking the format (NewsCache history 2). The bucket table takes 512 KB per file, a record about 100 bytes, so 10 million entries take about 1 GB.

A damaged file, or one of another format (a development build before #63 used 256 buckets), is renamed to <n>.bad and started anew. The history can be deleted at any time while NewsCache is stopped; it is filled again as clients read.

The history counts toward SpoolSize. Its files never shrink, so newscacheclean removes them when the history alone takes more than half of SpoolSize, and after History off (they are unused then); it also removes .history/*.bad.

Cleaning the spool: newscacheclean

NewsCache itself doesn't delete articles to make room; newscacheclean does. It walks the spool, sorts group databases and article files by last access, removes outdated article files first and then the least recently used entries until the spool is below SpoolSize (in KB). The message-ID history counts toward SpoolSize (see above).

newscacheclean doesn't follow symbolic links in the spool, neither to directories nor to files: what they point to is not counted and not removed. It runs with the priority increment NiceClean (default 3).

newscacheclean          # clean
newscacheclean -s       # statistics
newscacheclean -t -p    # try only, and print the purge table

Option	Effect
-c file	Read this configuration file instead of the built-in one (usually /etc/newscache.conf)
-s	Show statistics: how many databases and articles were found and removed, and their sizes; DirCnt counts cached group directories. With -t, show the found counts (up to 1.3.22 -s -t printed zeros, and DirCnt was always zero).
-p	Print the sorted purge table
-t	Try only, remove nothing; with -p it shows what would be removed first
-v, -h	Print the version, or a summary of the options

Run it regularly: daily from cron, or with a systemd timer (the Debian package has newscacheclean.timer; FEDORA.md shows one). In a container: docker exec newscache newscacheclean.

Damaged spool files

NewsCache checks the offsets and sizes in a group's .db and in .active64 before it uses them. A damaged file is moved to name.bad (one error line in the log), a new one is started and filled from the upstream server again, and the command that ran into it is retried. newscacheclean goes on with the other groups and removes the .bad files. See doc/newscache-corrupt-db-files.txt.

Prefetching: updatenews

updatenews sends postings that are still queued, refreshes the active list if needed, and prefetches the groups in the PrefetchFile. Run it from cron.

Option	Effect
-c file	Read this configuration file instead of the built-in one (usually /etc/newscache.conf)
-a	Refresh the active list now
-o	Prefetch only the overview (of the newest 10000000 numbers of each group); with -g as well, -g wins
-g	Prefetch the articles too (same range)
-l, --lock-group	With -g, reserve each range of 10000 numbers while it is prefetched: a session that needs that range's overview waits for the prefetch instead of fetching it too; sessions reading what the spool holds don't wait. Up to 1.3.22 it locked the whole group, so nobody could read it. --lock-groups works too
-w n	Accepted for old command lines and ignored (with a log line): groups are prefetched one at a time. Up to 1.3.22 it failed with the usage text and did nothing.
-v, -h	Print the version, or a summary of the options

Without any of -a, -g and -o, updatenews works as with -g (-l doesn't count: -l alone works as -g -l; up to 1.3.22 it prefetched nothing). Once one of them is given, it prefetches groups only with -g or -o: -a alone only posts the queue and refreshes the active list.

The prefetch file has one entry per line:

# all of de.*, but not the binaries
de.*
!de.alt.binaries.*
comp.lang.[cg]o
at.test

• A group name prefetches that group.
• A wildmat pattern (*, ?, [...]) prefetches every matching group of the active list, in alphabetical order.
• !pattern (or !group) leaves the matching groups out, wherever they come from in the file.
• Empty lines and lines starting with # are skipped; each group is prefetched once. The last line counts even without a newline at its end.

updatenews uses each server's ConnectTimeout. When its spool can't be created or written, it logs the error and exits with status 1. A configured PrefetchFile that cannot be opened, or is a directory or FIFO, also causes an error and a nonzero exit. An empty file or /dev/null means no groups to prefetch.

Offline reading

Server option	Behaviour
Options offline	NewsCache never contacts the server; clients get only what updatenews fetched.
Options semioffline	NewsCache contacts the server when needed, except for the group list; fetch that with updatenews -a.

Postings made while offline are queued in SpoolDirectory/.artSpool and sent by the next updatenews run. They stay there until a news server takes them, also when it still can't be reached; only articles the server refuses go to .badArticles, for the administrator. If a posting can't be queued (a full disk), the client gets 441 instead of 240. See the manual, section "Offline Mode".