Setup Cluster

This page sets up two ALPS nodes behind a load balancer: [cluster] with a shared secret key and peers, the same login_key and settings on every node, session affinity at the load balancer, and the checks that show the nodes have joined, that a browser moves between nodes without signing in again, and that the sign-in rate limits are shared. A second part covers nodes that get their own certificates from Let's Encrypt and share them through S3, and the last parts cover restarts, upgrades and troubleshooting. Clustering explains how the cluster works, and Config-Cluster lists every key.

What you get

• Two nodes, node-1 (10.0.0.11) and node-2 (10.0.0.12), serving https://webmail.example.com behind HAProxy (10.0.0.10). HAProxy ends TLS and keeps each browser on one node with a cookie of its own.
• If a node stops, HAProxy sends its browsers to the other node, which opens a new session from the browser's login token. Users stay signed in, as long as their login token is less than 30 days old. Uploads not yet sent and security-key steps in progress are lost and have to be done again.
• Sign-in attempts are counted across both nodes: failures on one node count towards the limits on the other.
• With the Let's Encrypt variant, only one node (the leader) orders certificates, and the other nodes read them from S3.

More nodes work the same way: give each one its own node_id and bind, list all the others in peers, and add a server line for it at the load balancer. node_id values are compared as plain strings, so node-10 sorts before node-2 and would lead.

Before you start

• Two Linux hosts for ALPS and one for the load balancer, on a private network. The DNS name webmail.example.com points at the load balancer.
• The same ALPS binary on every node, at /usr/local/bin/alps, run by the systemd unit from dist/alps.service (see Running-in-Production).
• A mail server that every node can reach, here imaps://imap.example.com:993 and smtps://smtp.example.com:465 (Mail-Server-Requirements).
• A certificate for webmail.example.com on the load balancer, as one PEM file with the chain and the key.
• Firewall rules between the nodes:
  • the gossip port, 7946, open over both TCP and UDP between the ALPS nodes, and to nothing else;
  • the ALPS port, 1323, open from the load balancer (and, for the checks below, from an admin host).
• A test account on the mail server. The rate-limit check below blocks it for about 15 minutes.

Generate two keys, once, and use the same two on every node:

openssl rand -base64 32   # login_key: restores sessions on any node
openssl rand -base64 32   # [cluster] secret_key: encrypts and authenticates the gossip

Keep both secret: login_key encrypts the login tokens, which carry the users' credentials, and secret_key admits a node to the cluster. Make /etc/alps/config.toml readable by the alps user and by no other user.

Configuration

node-1: /etc/alps/config.toml

[server]
addr = "10.0.0.11:1323"
# The same on every node. Any node can then restore a session from the
# browser's login token. Linked accounts need it too.
login_key = "PASTE-THE-LOGIN-KEY"
# The load balancer only. ALPS then takes the browser's address and scheme
# from its X-Forwarded-* headers.
trusted_proxies = ["10.0.0.10/32"]

[logging]
output = "stderr"
format = "console"
level = "info"

# The same on every node: a security key is bound to the RP ID it was
# enrolled under, wherever the browser lands.
[webauthn]
rpid = "webmail.example.com"
origins = ["https://webmail.example.com"]

[provider]
type = "imap"

[provider.imap]
server = "imaps://imap.example.com:993"

[smtp]
server = "smtps://smtp.example.com:465"

# List every plugin you use, the same on every node.
[plugin.base]
enabled = true

[cluster]
enabled = true
node_id = "node-1"            # Unique per node; the smallest one leads
bind = "10.0.0.11"            # This node's private IP address
port = 7946                   # Gossip, TCP and UDP
secret_key = "PASTE-THE-CLUSTER-SECRET-KEY"
peers = ["10.0.0.12:7946"]    # Every other node

node-2: /etc/alps/config.toml

The same file, with these four lines changed:

[server]
addr = "10.0.0.12:1323"

[cluster]
node_id = "node-2"
bind = "10.0.0.12"
peers = ["10.0.0.11:7946"]

Everything else must be the same on both nodes: login_key, [webauthn], secret_key, [server.rate_limit] (left at its defaults here), [provider], [smtp], the plugins (and [server] dav_allow_private_servers when you set it) and trusted_proxies. A node applies its own rate-limit values to the shared counts, so different values give different limits per node. Settings Every Node Must Share has the full list.

To keep the gossip key out of the config file, leave secret_key out and set the environment variable CLUSTER_SECRET_KEY instead. When both are set, the variable wins.

bind must be an IP address, IPv4 or IPv6, which a port may follow (10.0.0.11:7946, [fd00::11]:7946). A host name stops ALPS before it opens its log, with Error generating options: cluster.bind: must be an IP address with an optional port, got "node1.example.com" on standard error. ALPS warns at startup about every key it does not know, such as Unknown configuration key: cluster.peer (ignored), so check the log after each change.

systemd drop-in, on every node

ALPS takes up to 40 seconds to stop, and up to 55 with Let's Encrypt: it drains HTTP for up to 30 seconds, waits up to 5 for the sessions' cleanup, stops the certificate workers, then leaves the cluster, waiting up to 5 seconds so that the other nodes see it go. The shipped unit's TimeoutStopSec=60 covers that (systemd). A unit copied from an earlier release has TimeoutStopSec=30: copy the current dist/alps.service, or add TimeoutStopSec=60 to the drop-in.

The drop-in is needed only to read the gossip key from a file. Run systemctl edit alps and add:

[Service]
# Optional: the gossip key from a file only root can read, instead of secret_key.
# The file holds one line: CLUSTER_SECRET_KEY=...
# EnvironmentFile=/etc/alps/cluster.env

HAProxy: /etc/haproxy/haproxy.cfg

global
    log stdout format raw local0

defaults
    mode http
    log global
    option httplog
    timeout connect 5s
    timeout client  60s
    timeout server  60s

frontend https-in
    bind :443 ssl crt /etc/haproxy/certs/webmail.example.com.pem
    http-request set-header X-Forwarded-For   %[src]
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Host  %[req.hdr(host)]
    http-request set-header X-Real-IP         %[src]
    default_backend alps

backend alps
    balance roundrobin
    # Session affinity: HAProxy sets its own cookie and keeps each browser on one node.
    cookie SERVERID insert indirect nocache httponly secure
    # If that node is down, send the browser to another one.
    option redispatch
    option httpchk GET /
    server node1 10.0.0.11:1323 check cookie node1
    server node2 10.0.0.12:1323 check cookie node2

• Affinity. Sessions live in the memory of the node that opened them, so every request from a browser has to reach the same node. The SERVERID cookie does that. Do not key affinity on alps_session: ALPS issues a new value at sign-in, at a session restore and at an account switch. Source-IP affinity also works.
• Forwarded headers. set-header replaces any copy of the header the browser sent, so each header carries only what HAProxy saw. ALPS reads them only from the address in trusted_proxies.
• Timeouts. With [server] push_new_mail on, /events stays open as long as the page. timeout server must be longer than its 20-second heartbeat.
• Health check. ALPS has no health endpoint. GET / answers 200 without a session, so it shows that ALPS is up, but not that the mail server is reachable. Each check writes an access-log line on the node (see Troubleshooting).

The HAProxy settings are explained in docs/TLS.md and on Setup-Behind-a-Reverse-Proxy.

Start it

On every node, in any order:

systemctl daemon-reload
systemctl enable --now alps

Then start HAProxy. No node has to go first: the node with the smallest node_id leads as soon as it has joined the others.

Check it works

The nodes have joined

The cluster has no status endpoint. Its state is in the log. These lines follow [logging] like the rest, so with the file above they go to standard error as console text, and under systemd they are in the journal:

journalctl -u alps | grep -E 'gossip|cluster started|node (joined|left)|leader changed|failed to join|rejoined'

On node-1, started first:

time=… level=INFO msg="gossip encryption enabled (AES-256)"
time=… level=INFO msg="node joined cluster" node=node-1 addr=10.0.0.11:7946
time=… level=INFO msg="cluster leader changed" old_leader="" new_leader=node-1 is_leader=true
time=… level=WARN msg="failed to join some peers (may be first node)" error="1 error occurred:\n\t* failed to join 10.0.0.12:7946: dial tcp 10.0.0.12:7946: connect: connection refused\n\n"
time=… level=INFO msg="cluster started" node_name=node-1 bind_addr=10.0.0.11 bind_port=7946 advertise_addr=10.0.0.11 advertise_port=7946 members=1 leader=node-1 is_leader=true
time=… level=INFO msg="node joined cluster" node=node-2 addr=10.0.0.12:7946

On node-2:

time=… level=INFO msg="gossip encryption enabled (AES-256)"
time=… level=INFO msg="node joined cluster" node=node-2 addr=10.0.0.12:7946
time=… level=INFO msg="cluster leader changed" old_leader="" new_leader=node-2 is_leader=true
time=… level=INFO msg="node joined cluster" node=node-1 addr=10.0.0.11:7946
time=… level=INFO msg="cluster leader changed" old_leader=node-2 new_leader=node-1 is_leader=false
time=… level=INFO msg="cluster started" node_name=node-2 bind_addr=10.0.0.12 bind_port=7946 advertise_addr=10.0.0.12 advertise_port=7946 members=2 leader=node-1 is_leader=false

The failed to join warning on the first node is normal: its peer was not up yet. Each node names itself leader for a moment before it has joined the other. What to check:

• the last node to start says members= with the number of nodes;
• every node's latest cluster leader changed line names the same new_leader, and only that node says is_leader=true.

A browser moves between nodes

1. Sign in at https://webmail.example.com. The browser now holds a SERVERID cookie naming its node, for example node2.
2. On that node, run systemctl stop alps. The other node logs node left cluster node=node-2.
3. Reload the page. It keeps working, and the other node logs session restored:

time=2026-11-12T10:03:25.547Z level=INFO msg="session restored" user=test@example.com ip=203.0.113.25 session=a5b6a6c4f565

4. Start the node again with systemctl start alps. It rejoins, and node-1 stays leader.

The same check from a shell, through the load balancer:

U=https://webmail.example.com
curl -s -c jar.txt -b jar.txt -H "Origin: $U" -H 'Content-Type: application/json' \
  -d '{"username":"test@example.com","password":"the-password"}' \
  -o /dev/null -w '%{http_code}\n' "$U/session"            # 200
grep SERVERID jar.txt                                       # the node that holds the session
curl -s -c jar.txt -b jar.txt -o /dev/null -w '%{http_code}\n' "$U/mailboxes"   # 200
# stop ALPS on that node, then:
curl -s -c jar.txt -b jar.txt -o /dev/null -w '%{http_code}\n' "$U/mailboxes"   # 200, from the other node
grep SERVERID jar.txt                                       # now the other node

Send the Origin a browser would send, the address you reach ALPS at. A different one is refused with 403 Invalid origin.

The log line ip= should show the browser's address. If it shows the load balancer's, see Troubleshooting.

Rate limits are shared

From an admin host, sign in with wrong passwords directly on each node, at least a minute after any other sign-in from that host. With the default limits, an account with 5 refused attempts in 15 minutes is refused with a wait of 15 minutes, on every node:

try() {  # node, password
  curl -s -w ' %{http_code}' -H "Origin: http://$1" -H 'Content-Type: application/json' \
    -d "{\"username\":\"ratelimit-test@example.com\",\"password\":\"$2\"}" "http://$1/session" | tr -d '\n'
  echo
}
for i in 1 2 3 4; do try 10.0.0.11:1323 wrong; done   # 4 x {"error":"Failed to login"} 401
try 10.0.0.12:1323 wrong                               # {"error":"Failed to login"} 401
sleep 61                                               # past the per-minute limit of the admin host's address
try 10.0.0.11:1323 'the-right-password'
try 10.0.0.12:1323 'the-right-password'

Both nodes answer with the right password:

{"error":"Too many failed login attempts for this account","retry_after":900} 429

node-2 saw only one failure itself. It refuses because it also counted the four failures node-1 sent it. Without [cluster], each node would count only its own. Lockouts are not sent between nodes: each node locks an account or an address itself, at the first attempt it receives once its shared counts have reached an hourly limit, and logs it, for example Username ratelimit-test@example.com locked out for 15m0s after 10 failed attempts in 1 hour. docs/RATELIMIT.md has the limits and how the counts are shared.

The test account stays blocked until those failures are 15 minutes old. The counts live only in memory, so restarting every node also clears them.

Variant: the nodes get certificates from Let's Encrypt

In the setup above, the load balancer holds the certificate. ALPS can instead get its own certificates from Let's Encrypt. In a cluster that needs:

• S3 storage. storage_provider must be "s3" (the default); "file" stops startup. The leader stores the ACME account key, the challenge tokens and the certificates in S3, and the other nodes read them from there.
• The same domains, directory_url, S3 bucket and prefix on every node.
• A load balancer that passes TCP through. ALPS ends TLS itself, so port 443 must reach the nodes unchanged, with source-IP affinity for the sessions. Port 80 must reach the nodes too, for the HTTP-01 challenge.
• The browser's address. Through a TCP pass-through, ALPS sees the address of the connection, and it does not read the PROXY protocol. The load balancer must keep the browser's source address (no source NAT). Otherwise every browser shares the load balancer's address for the rate limits, and the address limits apply to all users together.

Change these sections on every node (trusted_proxies goes, since nothing sends forwarded headers):

[server]
addr = ":443"
login_key = "PASTE-THE-LOGIN-KEY"

[tls]
enabled = true
provider = "letsencrypt"

[tls.letsencrypt]
email = "admin@example.com"
domains = ["webmail.example.com"]
cache_dir = "/var/lib/alps/cert-cache"   # A local copy on each node
acme_http_addr = ":80"

[tls.letsencrypt.s3]
bucket = "alps-certs"
region = "us-east-1"
prefix = "alps/"
access_key = "PASTE-THE-S3-ACCESS-KEY"   # Leave both out to use the AWS credential chain
secret_key = "PASTE-THE-S3-SECRET-KEY"
# endpoint = "https://s3.example.com"    # Only for S3-compatible storage other than AWS

Create /var/lib/alps, owned by alps (install -d -o alps -g alps -m 700 /var/lib/alps), and add ReadWritePaths=/var/lib/alps to the systemd drop-in. The shipped unit already allows binding ports 80 and 443.

The load balancer has two jobs:

• Port 443 reaches the nodes as TCP, unchanged, with the browser's own source address, and keeps each address on one node. That takes a layer-4 load balancer that does no source NAT, such as Linux IPVS in direct-routing mode with source hashing, or a cloud network load balancer that keeps client addresses. HAProxy in TCP mode does not do this: it opens its own connections to the nodes, so ALPS would see HAProxy's address on every connection. Setting up such a load balancer is outside this page.
• Port 80 carries only ACME HTTP-01 validation. The source address does not matter there, so HAProxy can do it (/etc/haproxy/haproxy.cfg):

global
    log stdout format raw local0

defaults
    log global
    timeout connect 5s
    timeout client  60s
    timeout server  60s

# Port 80 carries only ACME HTTP-01 validation. Send it to node-1, the
# leader, with the others as backups, in node_id order.
frontend http-in
    mode http
    option httplog
    bind :80
    default_backend alps-acme

backend alps-acme
    mode http
    server node1 10.0.0.11:80 check
    server node2 10.0.0.12:80 check backup

Nothing else is served on port 80: ALPS answers 404 on every other path there and does not redirect to HTTPS (docs/TLS.md).

Send port 80 to the leader, which places the certificate orders and answers their validations. ALPS tries TLS-ALPN-01 on port 443 first and HTTP-01 on port 80 second, so with port 80 on the leader an order succeeds whichever node the port-443 validation reaches. When node-1 is down, node-2 leads and HAProxy sends port 80 to it.

Check the certificates

• About 60 to 120 seconds after the nodes start, the leader runs its first certificate pass. Only node-1 logs that it stored the certificate:

time=… level=INFO msg="cluster cache: cluster leader storing certificate" name=acme_account+key
time=… level=INFO msg="cluster cache: certificate stored by leader" name=webmail.example.com

The same pass then logs msg="cert warmer: certificate is valid" domain=webmail.example.com with the certificate's not_after, and so does every later pass, every 12 hours, on whichever node leads. cert warmer: certificate is due for renewal at two passes in a row means that renewals are failing (Setup-Lets-Encrypt-Standalone).

• A follower reads it from S3 at its first handshake for the name:

time=… level=INFO msg="FallbackCache: certificate found in S3 - syncing to local cache" name=webmail.example.com

• Every node serves the same certificate. Run this against each node and compare the serial:

openssl s_client -connect 10.0.0.12:443 -servername webmail.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -serial -issuer -dates

• The bucket holds, under alps/, the account key acme_account+key and one object per certificate, named after the domain. prefix = "alps" without the slash gives the same keys.

In testing, two nodes placed one order for webmail.example.com in all: through restarts of either node, a stop of both, a leader handover, and a node that had lost contact with the cluster. Each time a node found the certificate in its cache_dir or in S3. To rehearse without touching production limits, set directory_url to the Let's Encrypt staging directory on every node; ALPS then keeps the staging certificates apart from the production ones, in a cache_dir of their own and under alps/acme-<8 hex characters>/ in the bucket (docs/TLS.md).

Certificates in a Cluster covers renewal and the less common cases, and Setup-Lets-Encrypt-Standalone the single-node setup.

Restarts and upgrades

A configuration change, same binary: restart the nodes one at a time. The other node goes on serving, and the browsers of the restarted node are restored on whichever node they reach. A change to a shared setting is in effect node by node, so the nodes disagree on it until the last one has restarted. A new secret_key needs every node restarted; until then, the nodes with the old key and those with the new one form two clusters, each with its own leader.

A new ALPS binary: upgrade every node together. Mixing binaries is not supported (Restarting and Upgrading).

1. On every node: systemctl stop alps.
2. Wait until it has stopped everywhere: systemctl is-active alps prints inactive on every node.
3. Install the new binary at /usr/local/bin/alps on every node.
4. On every node, in any order: systemctl start alps.
5. Check membership and the leader as in The nodes have joined.

The upgrade ends every session. With login_key set and unchanged, each browser is restored from its login token at its next request; without it, every user signs in again. The rate-limit counts start from zero.

Troubleshooting

A node stays alone: members=1 and is_leader=true

Look at its failed to join some peers line:

• no installed keys could decrypt the message: its secret_key (or CLUSTER_SECRET_KEY) differs from the others'. Only the node that tried to join logs this; the node it tried logs nothing.
• connection refused or a timeout: the peer is not up, the address in peers is wrong, or the gossip port is closed on TCP or UDP.

A node alone is its own leader and counts rate limits on its own. Every 30 seconds each node tries again the peers in its peers list that are not among its members, and logs cluster rejoined peers when it reaches one. A lone node, or a group of nodes cut off from the others, therefore joins the rest once the network lets it, and the group again has one leader.

A node that was killed and started again quickly may not show as node left and node joined on the others. Check its own cluster started line instead.

Every request shows the load balancer's address

The node logs ip=10.0.0.10 for browser requests too, not only for the health checks. With an HTTP load balancer, trusted_proxies does not list its address, or it does not send X-Forwarded-For. With a TCP pass-through, the load balancer replaces the source address; see the Let's Encrypt variant. Until it is fixed, all users share one address's rate limits.

Users are signed out when they move to another node

The login_key differs between the nodes, or is not set. The other node cannot read the login token, clears the cookies and answers 401. Set the same key on every node. Changing the key signs everyone out once, linked accounts have to be linked again, and separate CalDAV and CardDAV passwords saved in Settings have to be entered again (Sessions-and-Sign-in).

The access log fills with GET / from the load balancer

Each health check writes a line. Lengthen the check interval in the load balancer (inter on the HAProxy server lines), or set [logging] access = false if the load balancer keeps its own access log.

Startup stops with S3 bucket validation failed

With Let's Encrypt, ALPS checks the bucket at startup and exits with status 1 when it cannot use it. It tries five times within 30 seconds of validating S3 bucket access, then logs one ERROR line with the S3 client's last error, for example:

time=… level=ERROR msg="Failed to initialize TLS manager: S3 bucket validation failed after 5 attempts: operation error S3: HeadBucket, exceeded maximum number of attempts, 3, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Head \"https://s3.example.com/alps-certs\": dial tcp 192.0.2.20:443: connect: connection refused"

When the 30 seconds run out first, the line reads S3 bucket validation cancelled after <n> attempts: context deadline exceeded, followed by the same kind of error. A connection error points at the endpoint or the network, and an HTTP status such as 403 at the keys, the bucket name or the region. Check them from the node itself with an S3 client. The node has joined the cluster by then: it leaves it again before it exits, and the other nodes log node left cluster.

A follower fails handshakes right after a new cluster starts

time=… level=WARN msg="cluster cache: certificate request BLOCKED - not cluster leader (leader will handle it)" name=acme_account+key
time=… level=ERROR msg="TLS: failed to get certificate" server_name=webmail.example.com error="only cluster leader can request new certificates"

The leader has not stored an ACME account key or a certificate yet. Its first pass runs 60 to 120 seconds after startup. After such a failure, the follower fails the next handshakes for that name with error="acme/autocert: missing certificate" for up to a minute before it looks again, so once the certificate is in S3, the follower serves it within a minute.

The leader logs failed to obtain certificate ... no viable challenge type found

Every validation failed. On a follower, acme/autocert: no token cert for lines at the same moment mean the validation reached it instead of the leader; send port 80 to the leader as shown above. A port-80 proxy must pass the Host header without a port: with Host: webmail.example.com:80, ALPS answers 403. After a failure, the leader's next pass is 12 hours later. A handshake for the domain that reaches the leader starts a new attempt sooner, at most once a minute.

Tested with

ALPS main a5443047, on 2026-11-12: two nodes as the Linux arm64 build in Debian 13.7 (trixie) containers on Docker 29.9.2 (Docker Desktop, arm64), HAProxy 2.8.31, sora as the IMAP and SMTP server, and for the Let's Encrypt variant Pebble 2.5.1 as the ACME server and versitygw 1.11.0 as the S3 store. Checked in the lab: joining, leader handover, session restore on the other node (failover through HAProxy, and cookies reused on the other node directly), shared rate limits with the defaults, an account lockout reached from both nodes (with lowered limits), CLUSTER_SECRET_KEY, a mismatched key, stopping and starting both nodes, one order for two nodes, a follower serving the leader's certificate, and HTTP-01 sent to the leader. The systemd drop-in and an upgrade between two different binaries were checked against the docs only.

A second run on the same day followed this page from the top with two macOS builds of the same commit on one host, each node with its own HTTP and gossip port, behind the HAProxy configuration above: joining, failover through HAProxy with the session restored, the rate-limit check exactly as written, a third node joining, CLUSTER_SECRET_KEY alone and over a different secret_key, a mismatched key, a different login_key (401, cookies cleared), a lone node rejoining after 30 seconds, stopping both nodes and starting them in the other order, and, for the Let's Encrypt variant, the follower's BLOCKED lines, the port-80 Host check and the startup S3 check. That run issued no certificate.

Updated from the code at main 92ec0e11 without a rerun: the retries that now also join a group of nodes split from the others, the check that bind is an IP address, the startup S3 check's message, which now names its cause, the node leaving the cluster when its startup fails after it joined, and the warm pass's line for each certificate. The node-1 file, alone and with the Let's Encrypt sections, was started once with a 92ec0e11 build on macOS, with loopback addresses and an unreachable S3 endpoint, to check that ALPS knows every key and to read the S3 message.