Upstream Servers and TLS

Upstream Servers and TLS

NewsCache reads articles from one or more upstream news servers, listed in NewsServerList. Clients see one merged list of groups and configure only NewsCache. The per-server options are listed on Configuration.

One server

NewsServerList {
  Server news.example.net 563 {
    Read *
    PostTo *
    GroupTimeout 300
    Options ssl
  }
}

The Server line takes the host name (or address) and the port, as a number or a service name (nntp, nntps).

Several servers

Each group is read from the server whose Read list matches it most specifically, whatever the order of the entries; postings go to the server whose PostTo list matches.

NewsServerList {
  Server news.blueworldhosting.com 563 {
    Read *
    PostTo *
    GroupTimeout 300
    Options ssl
  }
  # eternal-september.* matches more specifically than *
  Server news.eternal-september.org 563 {
    Read eternal-september.*
    PostTo eternal-september.*
    GroupTimeout 300
    User joe
    Password secret
    Options ssl
  }
  # a local news server, passed through without caching
  Server news.internal.example 119 {
    Read local.*
    PostTo local.*
    Options not-cached
  }
  # groups nobody should get
  NoServer {
    Read alt.binaries.*
    PostTo alt.binaries.*
  }
}

• The Read list of a server is also sent to it as LIST ACTIVE patterns, so narrow lists save bandwidth.
• A posting to groups of several servers is sent to each of them; before sending or queuing any copy, NewsCache adds any missing Message-ID, Date and Injection-Date fields, so all copies have the same values. Existing fields are preserved. For a single server, NewsCache does not add the dates; the injecting server handles them.
• After changing the servers or their Read lists, remove the active database (SpoolDirectory/.active64) or run updatenews -a, so the group list is fetched again.
• When a group moves to another server (its Read list changes), also remove that group's cached files: .db and .art* in its spool directory (de.comp.os → SpoolDirectory/de/comp/os/; leave the subdirectories, they belong to other groups), best with NewsCache stopped. The spool doesn't record which server a group came from: if the new server's article numbers overlap the old server's, NewsCache would serve the old server's cached articles under the new numbers (manual, FAQ).
• The Xref header of articles and of overview records keeps only the groups of the server the article came from: the other entries hold another server's numbers, and newsreaders use Xref to mark crossposts read.
• Local newsgroups: run a small news server with only the local groups and add it as another Server (see the manual, section "Local Newsgroups").

Logging in to the upstream server

User and Password send AUTHINFO USER / AUTHINFO PASS to the server when NewsCache connects, and again when the server asks for the login in the middle of a session (480, a login that timed out); then the command is sent once more. With Options starttls, the login always waits for STARTTLS, even when the server answers 480 to MODE READER before it; MODE READER is sent again after the login. The log shows AUTHINFO to the server without its arguments. Up to 1.3.21 the password was logged at level info (in syslog's news facility, which Debian's rsyslog also writes to /var/log/syslog, or in newscache.log): check old logs, and change the password if others could read them. newscache -p and xdebug dump configuration don't show the password; they print # Password set, not shown instead. Keep the configuration file readable only for root and the news group:

chown root:news /etc/newscache.conf
chmod 0640 /etc/newscache.conf

TLS

NewsCache uses GnuTLS (build with --with-gnutls, the default when it is found). A build without GnuTLS doesn't connect to servers with ssl or starttls at all (it logs "built without GnuTLS") rather than falling back to plain text.

Option	Connection
Options ssl	TLS from the start, usually port 563 (nntps)
Options starttls	Plain connection, then STARTTLS, usually port 119. If the server refuses STARTTLS or the handshake fails, NewsCache gives up the connection; it never continues in plain text.

Certificate verification

With ssl or starttls, the server's certificate must:

• chain to a trusted CA: the system trust store, or the CA certificates in TLSCAFile (a PEM file) for servers with a private CA;
• be valid now (not expired);
• match the host name or IP address in the Server line.

The host name is also sent as SNI (not when it is an IP address). If the check fails, the connection fails and the log says why: a line TLS handshake with <host> failed: <reason>, and for certificate errors a line starting with TLS: that gives GnuTLS's explanation (untrusted, expired, wrong name, ...).

Options noverify skips the check: the connection is still encrypted, but anybody in the middle can pretend to be the server. NewsCache logs a warning for every such connection. Use it only for a server with a self-signed certificate, and prefer TLSCAFile.

To see a server's certificate and its dates:

openssl s_client -connect news.example.net:563 -servername news.example.net </dev/null |
    openssl x509 -noout -subject -issuer -dates

Example: private CA

  Server news.internal.example 563 {
    Read *
    PostTo *
    TLSCAFile /etc/ssl/internal-ca.pem
    Options ssl
  }

TLS for clients

With GnuTLS, NewsCache can offer implicit TLS to newsreaders on its own TLSListenTo listener; see TLS for Clients. This is configured separately from TLS to upstream servers. A TLS proxy is still an option; NewsCache then sees the proxy's address, so the AccessList must allow it.

Connection behaviour

Option	Meaning
ConnectTimeout seconds	Give up connecting after this long (default 0: the system's own timeout, often about two minutes). updatenews uses it too.
ConnectBackoff seconds	After a failed connect, don't try this server again for this long.
ReadTimeout seconds	Give up the connection when the server sends or accepts nothing for this long, on each read and write (default 300; 0 disables this per-I/O timeout). Also over TLS. Some whole replies and TLS handshakes have separate limits below.
Retries n	Reconnect this often when the server closes the connection or runs into ReadTimeout (default 3).
BindFrom address	Local address for the connection (numeric).
Commands not-over, not-listgroup	Don't use OVER/XOVER or LISTGROUP with servers that handle them badly.

Upstream servers must have an overview database (OVER or XOVER). An upstream TLS handshake has a whole-handshake deadline: ReadTimeout if nonzero, otherwise ConnectTimeout if nonzero, otherwise 60 seconds. Partial handshake data does not extend it.

A reply of 400 from the server counts as a closed connection, and the command is sent again on a new one. A 503 ("feature not supported" in RFC 3977) is an ordinary reply, unless the server has closed the connection behind it, as old INN versions do with 503 Timeout on an idle connection. A server that answers LIST OVERVIEW.FMT with an error gets the standard overview format.

Limits and failures

• An article from the server may have at most MaxArticleSize bytes (global option, default 1048576), and other lines (status lines, overview records, the active list, LISTGROUP) at most 16 KiB. More is an error: NewsCache closes the connection to that server; an oversized article is not stored.
• Each upstream LIST ACTIVE reply is limited to 2000000 lines, 256 MiB and five minutes. Each OVER or LISTGROUP reply is limited to the number of lines in its requested range and two minutes. These are whole-reply limits, independent of ReadTimeout. An incomplete reply is not marked fresh, and its upstream connection is closed rather than reused. An incomplete combined active list is retried after at most 60 seconds (sooner if its configured timeout is shorter).
• A client's LISTGROUP is split into adaptive upstream steps; a command that has delivered no number to the client for five minutes fails. See Caching and Large Groups.
• When the server fails while NewsCache fetches something for a client (it can't be reached, it closes the connection in the middle, it runs into ReadTimeout, an article is too large), the client gets 403 (403 upstream failure for an article by message-ID, 403 internal error for one by number) and the session goes on. Up to 1.3.22 the reply was 503, or 423 for an article by number. An article or overview range that the server cut off is not stored as complete.
• With several servers, a server that can't be reached is skipped for ARTICLE <message-id>; the article comes from the next one.
• GROUP for a group the spool doesn't know yet answers 403 when the server can't be reached, not 411; a group the spool knows is served from it while the server is down.