Session Storage Server 2.12.0

This is a major Storage Server update. Storage servers running 2.12 now talk to each other over QUIC instead of oxenmq, swarm members keep each other's message stores in sync much more reliably, and the database layer has been reworked to remove some long-standing locking problems and make the most common write requests far cheaper.

This release is optional, but recommended, for service nodes. (It is required when also upgrading oxen-core to the new 11.7.0 release).

2.12.0 is designed to run in a mixed network with 2.11.x nodes during the upgrade period: everything a 2.11 peer can observe is version-gated, and 2.11 peers continue to see exactly the traffic they always have.

Node-to-node traffic over QUIC

When both QUIC nodes run 2.12+, storage servers now send each other forwarded requests, swarm data and onion request hops over QUIC rather than oxenmq. This uses the same QUIC listener (UDP, on the same port number as the oxenmq TCP port: 22020 by default) that 2.11 already uses for clients and already had reachability-tested by other nodes, so no new ports or firewall changes are needed. As before, make sure both TCP and UDP are open on that port, along with the HTTPS port (TCP 22021 by default).

Node-to-node connections are authenticated against the service node list, opened on demand and kept alive while in use, and closed when the other node leaves the network. Each connection carries separate streams for forwarded commands, bulk swarm data and onion hops, so a large data transfer between two nodes no longer delays client requests that are passing between them. Node-to-node work also has its own worker queue so it cannot starve client requests, and bulk data transfers are processed on a dedicated single-threaded queue (bulk SQLite inserts do not benefit from running in parallel).

Every node keeps reachability-testing other nodes' oxenmq ports, so a node with a closed oxenmq port is still caught while older nodes depend on it. That testing stops at the next network hard fork (not yet scheduled for mainnet), once 2.12+ is required everywhere. oxenmq will then no longer be needed for node-to-node traffic, and a later release will remove the public oxenmq listener.

Swarm synchronization

Keeping messages consistent across a swarm has been substantially reworked:

• Failed forwards are retried. If a store fails to reach another swarm member, it is now queued and retried instead of being lost. A failed store is retried as a delivery of the stored message rather than a replay of the client's signed request, so a message deleted in the meantime is not brought back. Other failed request types are retried after 15 and 45 seconds and dropped after 15 minutes. Pending retries are kept in the database, so they survive a restart.
• Swarm membership is remembered across restarts. A node now stores which swarm it was in, so it can tell when its swarm changed while it was offline and send its messages to members that joined in the meantime.
• Paced, resumable data transfers. When a node joins a swarm, the existing members send it the swarm's messages in acknowledged 1 MB batches (five in flight at a time) instead of one huge transfer. Progress is saved per peer, so a transfer that is interrupted by a restart or a dropped connection picks up where it left off.
• Fewer unnecessary transfers. A node asks its swarm for messages only when it has just joined the swarm, or when it starts up with none of the swarm's messages (for example, a fresh, wiped or copied database). Restarting into the same swarm with its data intact no longer triggers a transfer, and no longer logs every swarm member as "joining".
• Mixed 2.11/2.12 swarms. Data transfers in both directions between 2.11 and 2.12 nodes are handled, including a 2.11 node joining a swarm of 2.12 nodes.

Database changes and first start

The first start after upgrading runs a one-time database upgrade, mostly spent rebuilding an index. Its duration depends on how many messages the node stores and how fast its disk is: in testing it took about 4 seconds for 565,000 messages and 30 seconds for 1.2 million. The storage server reports its progress to systemd while it starts, and keeps extending systemd's start timeout during the upgrade, so a slow disk will not get the service killed and restarted partway through. Expect systemctl status oxen-storage-server to show "activating" until it finishes.

Note that a 2.12 database cannot be opened by 2.11 without manual changes, so plan on not downgrading after upgrading.

Database performance and reliability improvements in this release:

• Fixed a writer starvation problem. Under heavy read load (many clients polling at once), a lock around the database could keep a write waiting for a very long time, or indefinitely. That lock has been removed, and the "database is locked" errors it was working around have been fixed at their actual source.
• Much cheaper expiry updates. Analysis of live mainnet nodes showed that expire requests made up most database writes: Session clients currently attach an expiry refresh for their config messages to every poll, and every swarm member applies it. Multi-message expiry updates, deletes and expiry lookups now look up each message directly instead of scanning every message the account holds. In addition, refreshes that only nudge an already near-maximum expiry are now held in memory and written at most once every 30 minutes (and at shutdown). Clients see the requested expiry immediately, and swarm members stay in agreement, but the amount of database writing is drastically reduced.
• Message retrieval now pages straight off an index instead of sorting the account's messages on every poll, which significantly reduces disk reads and writes, and other common queries (namespace counts, message totals, swarm-range lookups) have been optimized to be cheaper.

Stability fixes

• Fixed a deadlock in reachability testing that could stall the storage server until systemd's watchdog killed and restarted it (typically seen as minutes of "No space to queue incoming command" log messages before a restart).
• Worked around a bug in libzmq that could abort the storage server when a connection died with a full receive buffer and a heartbeat was then sent on it.
• Fixed a use-after-free crash (in libquic) when the first send on a new outbound connection failed, e.g. with "network unreachable", during a reachability test. A related bug that could leave a reachability test round unfinished, so that the node was neither passed nor failed, is also fixed.
• Fixed a possible crash during shutdown caused by work still being processed after the QUIC server had been destroyed.
• Fixed a crash on systems where libcurl is linked against OpenSSL's ngtcp2 crypto backend: storage server now uses the GnuTLS build of libcurl to avoid the symbol conflict.
• Requests and responses between 10,000,000 bytes and 10 MiB (such as large attachment downloads routed through two 2.12 nodes) now pass correctly over node-to-node QUIC; anything too large now gets an error reply instead of no reply at all.
• Fixed an error thrown when deleting messages by timestamp across all namespaces (also present in 2.11.3).
• Startup no longer throws away the first service node list it receives from oxend. If oxend is synced, the storage server takes the node list from it before opening its listeners, and only contacts the bootstrap seed nodes when oxend is behind. Previously a node could reject peer connections after a restart until the next block arrived.

HTTPS and TLS

• The HTTPS listener now uses an ECDSA (P-256) certificate instead of RSA-2048, which makes each TLS handshake much cheaper for the server. The new certificate is generated automatically as cert_ecdsa.pem/key_ecdsa.pem in the data directory. The old cert.pem, key.pem and dh.pem files are no longer used and can be deleted. This changes nothing for Session clients.
• Fewer TLS cipher suites are offered (AES-128-GCM and ChaCha20-Poly1305), and the unused Diffie-Hellman parameters are no longer generated.
• uWebSockets remains the HTTPS server. Experimental support for libmicrohttpd as an alternative backend has been added for builds that enable it (with a new --https-backend option to choose between them), but it is not used in release builds.
• Request bodies are now limited to 10 MiB however they are sent; previously a chunked upload could get around the limit.

Configuration and logging

• --log-level (and log-level in the config file) now accepts per-category levels as well as a single level, e.g. log-level=info,swarm=debug. The default is now info,quic=warning,oxenmq=warning, which keeps the QUIC and oxenmq libraries quieter than before.
• New --skip-bootstrap-nodes flag to skip contacting the seed nodes at startup (intended for private test networks).
• The QUIC listener's address is now logged at startup, like the HTTPS and oxenmq listeners.

Client-facing changes

These do not require any operator action, but are relevant to client developers:

• Subscribers to swarm notifications now get a monitor_ended notification when the subscribed account moves to a different swarm, and subscribing to an account that is not in the node's swarm now returns an error. QUIC subscriptions are removed when the connection closes rather than an hour later.
• Namespace -3741 is reserved for clients to check whether they are still talking to the right swarm: stores into it are refused, and retrievals from it return immediately.
• 421 (wrong swarm) responses now include the account pubkey they refer to, so clients can match each redirect in a batch request to its account.
• AES-CBC onion request encryption has been removed. Session clients never used it; AES-GCM (the default) and XChaCha20-Poly1305 are unaffected.
• An onion request hop that names the receiving node as its own next hop is now handled directly instead of failing. A bug in some Session versions could build such a path about 0.08% of the time, and would result in a failed request.

Upgrading

As usual, upgrades are available from our APT repository (the old deb.oxen.io address continues to work). Static binaries for other users are available below.

Debian 11 ("bullseye") and Ubuntu 20.04 ("focal") are no longer supported, as they have reached end of life and no longer receive security updates. Packages are now provided for Debian 12+ and Ubuntu 22.04+. Operators still on those distributions should upgrade as soon as possible.

Expect the first start after the upgrade to take longer than usual while the database upgrade runs (see above).

Full Changelog: v2.11.3...v2.12.0