Skip to content

Clustering

Jump to bottom
Dejan Strbac edited this page Nov 12, 2026 · 1 revision

ALPS keeps no database, so several ALPS processes can serve the same mail server side by side behind a load balancer. This page explains what the nodes share through the [cluster] gossip, what stays on one node and why the load balancer needs session affinity, the settings every node must share, how certificates work across nodes, how to restart and upgrade, and how to check that the cluster has formed. The full reference is docs/CLUSTER.md. Every key is listed on Config-Cluster, and Setup-Cluster walks through a deployment.

When to turn on the cluster

Turn on [cluster] whenever several ALPS nodes serve one site, not only when they use Let's Encrypt. Several nodes can run without it, but each one is then a standalone instance:

  • It counts sign-in rate limits on its own, so attempts made on different nodes count towards separate limits.
  • Its sessions learn that an account enrolled a security key on another node only from their own periodic read of the account's record, which a session in use makes at most once a minute.
  • With Let's Encrypt, it requests and renews certificates on its own. Nothing stops several nodes from requesting the same certificate at about the same time, so several nodes that use Let's Encrypt need [cluster].

What the gossip carries

With [cluster] enabled = true (default false), the nodes join one gossip cluster. It carries three things:

  1. Leader election. One node is the leader (see Leader election). Only Let's Encrypt uses the leader, and nothing else in ALPS depends on which node leads.
  2. Sign-in rate-limit events. Every attempt a node records (a success, refused credentials or an error) is sent to all the other nodes, so the limits and lockouts apply across the whole cluster. A node can receive an event more than once but counts it once. Lockouts themselves are not sent: each node locks an address or an account when its own shared counts reach a limit, using its own [server.rate_limit] values. Delivery is best effort. See Cluster Synchronization.
  3. Security-key enrolment notices. When an account enrols a security key on one node, that node tells the others, and the account's sessions there read the account's 2FA record on their next request. The notice only prompts the read; the record on the mail server decides. If a notice is lost, the periodic read still finds the key.

The cluster carries nothing else:

Recovery codes in a cluster. With [cluster] enabled, recovery codes are offered only when the account's 2FA record is kept in the mail server's IMAP METADATA. With the maildir provider, or with an IMAP server that does not support METADATA, a cluster offers no recovery codes. See Mail-Server-Requirements and WEBAUTHN.md.

The rate-limit counts are kept in each node's memory and never written anywhere. A restarted node starts with empty counts: the other nodes do not resend past attempts, so it counts only the attempts it records or receives from then on (Where the State Lives).

Leader election

  • The leader is the member with the smallest node_id, compared as plain strings. node-10 therefore sorts before node-2. Pad the numbers (node-01, node-02, ...) if you want the order to follow them.
  • Each node works out the leader again every second and whenever a node joins or leaves. A node that sees no other member is its own leader.
  • Leadership is not handed over. When the leader stops, the node with the next smallest name leads. When the node with the smallest name rejoins, it leads again.
  • At startup a node tries to join the nodes in peers. A peer that is not up yet is not an error: ALPS logs a warning and starts anyway. Once a node reaches any one member, it learns about the rest of the cluster.
  • Every 30 seconds, a node contacts each peer in its peers list that is not among its members, and logs cluster rejoined peers when it reaches them. List all the other nodes in peers, so that a node can find its way back whichever of them is up.
  • Write each peer as the IP:port that node advertises, such as 10.0.0.2:7946. A peer written another way, as a host name for example, never matches a member's address, so the node contacts it again every 30 seconds and logs cluster rejoined peers each time (#952).

A network split leaves each side with its own leader, and with Let's Encrypt each side's leader runs the certificate warm passes. Once the network is back, the nodes find the peers they are missing within about 30 seconds, and the sides become one cluster again, as long as some node's peers lists a node on the other side. Check afterwards that every node names the same leader (see Checking the cluster).

Node-local state and session affinity

Each node keeps the following in its own memory (and, for uploads, in its own temp directory). None of it is shared with the other nodes:

  • each session, with its IMAP connections and its cache (Sessions-and-Sign-in);
  • the session's uploaded attachments (ATTACHMENTS.md);
  • a sign-in waiting for its security key, and any security-key sign-in or enrolment in progress;
  • the session's /events stream, when [server] push_new_mail is on.

Session affinity

The load balancer must send every request from a browser to the node that signed it in (session affinity, or sticky sessions). Use an affinity cookie that the load balancer sets itself, or source-IP affinity. ALPS issues a new alps_session value at sign-in, at a session restore and at an account switch, so affinity keyed on that cookie's value would have to follow those changes.

ALPS has no health endpoint. GET / returns the built-in page with 200 and needs no session, so it serves as a liveness probe, but it says nothing about the mail server or the cluster (TLS.md).

A request on another node

When a request reaches a node that does not hold its session (after a failover or a restart, or without affinity):

  • With login_key set, and the same on every node, the node decrypts the alps_login_token cookie and opens a new session for the user, with a new alps_session cookie, and logs session restored. The new session keeps the "Stay signed in after closing the browser" choice and the second-factor state, and signs in to the mail server again. It starts without the old session's uploads: at the next save or send, the composer lists the files to add again. A security-key sign-in or enrolment begun on the other node has to start over. The old session stays on its node until it reaches one of its limits.
  • Without login_key, or with a different one on this node, no session can be restored. The node clears the browser's session cookies (and a login token it cannot decrypt) and answers 401, so the user signs in again. The browser no longer holds its old session either, so it is signed out on its original node too.

A browser that keeps moving between nodes opens a new session, and a new mail-server connection, on every move. See Session Restoration.

Settings every node must share

Setting On every node, and why
[cluster] enabled true. A node without it runs as a standalone instance.
[cluster] secret_key (or CLUSTER_SECRET_KEY) The same key. Nodes with different keys do not form one cluster. On a fully trusted private network only, no key and allow_insecure = true everywhere instead.
[server] login_key The same, so that any node can restore a session from the login token. It also encrypts the stored credentials of linked accounts and the separate CalDAV and CardDAV passwords users save in Settings, so account switching and those connections need the same key everywhere. Keep it secret. Changing it signs everyone out, linked accounts must be linked again, and users enter their separate CalDAV and CardDAV passwords again.
[webauthn] rpid and origins The same. A security key is bound to the RP ID it was enrolled under, and origins lists every browser-facing origin the load balancer serves.
[server.rate_limit] The same. Each node applies its own values to the shared counts.
[provider], [smtp], the [plugin.<name>] sections and [server] dav_allow_private_servers The same, so that a session on any node reaches the same servers and offers the same features. A node without dav_allow_private_servers refuses the users' own CalDAV and CardDAV connections that need it.
With Let's Encrypt: [tls.letsencrypt] domains, directory_url and storage_provider = "s3", and [tls.letsencrypt.s3] bucket and prefix The same. See Certificates in a cluster.
The ALPS version The same. Upgrade every node together; mixing versions is not supported (see Restarts and upgrades).
The clock Kept in sync, for example with NTP. A node counts a rate-limit event at the sending node's time, ignores events more than 2 hours old, and takes a time ahead of its own clock as now (Cluster Synchronization).

node_id must be different on every node. Its default, the host name, works when every host has its own name.

bind, port, peers (each node lists the others) and [server] addr may differ per node. When the load balancer is an HTTP proxy, list it in [server] trusted_proxies on every node, and list only your own proxies there. Each node then sees the browser's address and scheme, and the rate limits count per browser address rather than per proxy (Client Addresses Behind a Proxy, Setup-Behind-a-Reverse-Proxy).

The [cluster] section

The section has seven keys: enabled, node_id, bind, port, secret_key, allow_insecure and peers. Config-Cluster lists each one with its default, and has an example for a three-node cluster. Plan for these when you lay out the network:

  • secret_key encrypts and authenticates all gossip (AES-256), and must be the same on every node. The CLUSTER_SECRET_KEY environment variable, when set, takes its place and keeps the key out of the config file. allow_insecure = true lets the cluster run without a key, unencrypted and unauthenticated; leave it off unless the network between the nodes is fully trusted and private.
  • The gossip port, 7946 by default, must be reachable from every other node over both TCP and UDP. Open it between the ALPS nodes only.
  • bind takes an IP address, IPv4 or IPv6, with an optional port (10.0.0.1:7946, [fd00::1]:7946) that overrides port. A host name or an interface name stops startup with cluster.bind: must be an IP address with an optional port. With an unspecified address such as the default 0.0.0.0, the node advertises a private address of the host, detected automatically, so set bind when the host has no private address or more than one.

Certificates in a cluster

With [tls] provider = "file", each node loads its own cert_file and key_file, and the cluster plays no part in certificates. With provider = "letsencrypt":

  • storage_provider must be "s3" (the default). "file" stops startup with TLS letsencrypt storage_provider must be "s3" when clustering is enabled, because each node's cache_dir is its own.
  • Only the leader runs the certificate warm passes, and only the leader's certificate manager may store certificates, challenge tokens and the ACME account key. A follower's attempt to store one is refused and logged as cluster cache: certificate request BLOCKED - not cluster leader (leader will handle it).
  • The first warm pass runs 60 to 120 seconds after startup, which leaves time for the nodes to find each other, and then every 12 hours. A node that becomes leader runs one as soon as it notices (it checks every 15 seconds), but passes are never less than 15 minutes apart.
  • Followers read certificates from S3: when a handshake needs one they do not hold, when a certificate's renewal comes due, and once at startup.
  • Every node answers the ACME challenges, HTTP-01 on acme_http_addr (default ":80") and TLS-ALPN-01 on its HTTPS listener, and reads the leader's challenge tokens from S3. Let's Encrypt's validation request can therefore reach any node.
  • Every node needs the same S3 bucket and prefix, the same domains and the same directory_url.

Certificates in a Cluster covers the less common cases, such as a follower that has to order a certificate itself. Config-TLS lists the keys.

Startup and shutdown

ALPS starts the cluster before the TLS manager and the HTTP server. With [cluster] enabled, ALPS exits with status 1 when:

  • bind is not an IP address with an optional port (this is checked with the rest of the configuration, before the log is opened, so the message goes to standard error);
  • there is no secret key and allow_insecure is not set;
  • the key does not decode to 32 bytes (failed to decode cluster secret key);
  • the gossip port cannot be bound;
  • with bind = "0.0.0.0", the host has no private address to advertise.

Except for the bad bind and the bad key, the log line starts with failed to initialize cluster. These lines go to the [logging] output. ALPS does not fall back to running as a standalone node.

Startup errors found after the cluster has started, such as a [tls] setting that is not valid or an HTTP address already in use (HTTP listener failed: ...), also stop ALPS with status 1. By then the node has joined the cluster, so it leaves it before it exits, as at a shutdown, and the other nodes log node joined cluster and then node left cluster for it. Under the shipped systemd unit, which starts ALPS again 5 seconds after a failure, this repeats until the configuration is fixed, so look for the error in the [logging] output of a node that keeps joining and leaving.

On SIGINT or SIGTERM, ALPS stops serving HTTP, the ACME challenge listener included (waiting up to 30 seconds for open requests), closes its sessions (waiting up to 5 seconds for their cleanup), stops the certificate manager (with Let's Encrypt up to 5 seconds, or 15 with S3 storage), and leaves the cluster, waiting up to 5 seconds so that the other nodes see it leave. A node with Let's Encrypt, which in a cluster always uses S3, therefore takes up to 55 seconds to stop, and one without it up to 40. The shipped systemd unit's TimeoutStopSec=60 covers both (SIGINT and SIGTERM). A node that is killed drops out once the others' failure detection notices it is gone. Either way the remaining nodes work out the leader again.

Restarts and upgrades

Same binary. Nodes that keep the same ALPS binary can be restarted one at a time, to apply a configuration change for example. The other nodes go on serving, and with the same login_key on every node, the browsers of the restarted node are restored on whichever node they reach. A change to one of the settings every node must share takes effect node by node, so the nodes disagree on it until the last one has restarted. A new secret_key needs every node restarted: until then, the nodes with the new key and those with the old one form separate clusters, each with its own leader.

New binary. A new ALPS binary is not rolled out node by node. Upgrade every ALPS instance that serves the same accounts together, in a cluster or standalone:

  1. Stop every node that runs the old binary.
  2. Wait until each one has exited.
  3. Start the nodes with the new binary.

Mixing binaries is not supported (Restarting and Upgrading, Recovery Codes).

The nodes can start in any order, and none has to go first. Until a node has joined the others it is its own leader; the node with the smallest node_id leads as soon as it has joined. The upgrade ends every session, as any restart does. With login_key set and unchanged, browsers are restored from their login token on the new nodes; without it, every user signs in again (What a Restart Loses). Running-in-Production has the wider checklist.

Checking the cluster

ALPS has no status endpoint for the cluster. Its state shows in the log. The cluster's lines, and the certificate manager's, go through [logging] like every other line: to its output, in its format, and only when their level reaches its level. The lines below are at INFO or WARN. The gossip library's own messages are logged at DEBUG, so they show only with [logging] level = "debug". See Config-Logging-and-Cache.

Line Meaning
gossip encryption enabled (AES-256) The node has a secret key. Without one, under allow_insecure, a warning says that gossip encryption is disabled.
cluster started Logged once at startup, with node_name, bind_addr, bind_port, advertise_addr (empty with bind = "0.0.0.0"), advertise_port, members, leader and is_leader.
node joined cluster, node left cluster A member came or went, with its name and address. A node also logs its own join at startup.
cluster leader changed With old_leader, new_leader and is_leader.
failed to join some peers (may be first node) No listed peer answered at startup. Normal for the first node to start.
cluster rejoined peers A node reached peers from its peers list that were not among its members.
TLS certificate cache wrapped with cluster-aware leader gating With Let's Encrypt, at startup.

A healthy third node joining two running ones logs these lines, shown here with format = "console" and the time= field left out. With the default JSON format, each line is a JSON object with the same fields.

level=INFO msg="gossip encryption enabled (AES-256)"
level=INFO msg="node joined cluster" node=node-c addr=10.0.0.3:7946
level=INFO msg="cluster leader changed" old_leader="" new_leader=node-c is_leader=true
level=INFO msg="node joined cluster" node=node-b addr=10.0.0.2:7946
level=INFO msg="node joined cluster" node=node-a addr=10.0.0.1:7946
level=INFO msg="cluster leader changed" old_leader=node-c new_leader=node-a is_leader=false
level=INFO msg="cluster started" node_name=node-c bind_addr=0.0.0.0 bind_port=7946 advertise_addr="" advertise_port=7946 members=3 leader=node-a is_leader=false

The node briefly names itself leader before it has joined the others. That is expected. What to check:

  • The last node to start reports members= equal to the number of nodes.
  • Every node's latest cluster leader changed line names the same new_leader, and only that node says is_leader=true.
  • A node that says members=1 and is_leader=true while the others are up has not joined. If its failed to join some peers line says no installed keys could decrypt the message, its secret_key differs from the others'. Only the node that tried to join logs this line. The node it tried reports the refused connection only in the gossip library's own messages, which show with [logging] level = "debug". Other errors in that line point to a peer that is not up, a peer address that is wrong, or a gossip port blocked on TCP, which the join uses. A gossip port blocked only on UDP does not show there: the nodes still form one cluster, but the rate-limit events and enrolment notices do not arrive. Config-Cluster shows how to check UDP.

To check affinity and login_key together, sign in through the load balancer, stop the node that holds the session, and reload the page. It should keep working, and the node it moved to logs session restored.

Tests

For contributors: the cluster package has its own Go tests (cluster/*_test.go: leader election, failover, the key check, message delivery, the bind check and a split that heals). They use ports the system assigns, so two runs at once do not collide. The browser suite can start a second ALPS process with [cluster] turned on in both (E2E_CLUSTER=1). See Go-Tests, Browser-Tests and E2E.md.

See also

These pages describe ALPS as it is on the main branch; the reference documents live in the repository's docs/ directory.

Home

Architecture

Tests

Configuration

Common Setups

Clone this wiki locally