Config Cluster

This page lists every key of the [cluster] section, the secret key and CLUSTER_SECRET_KEY, the peers list, the settings that must match on every node, and how to check a node's configuration in the log. Clustering explains what the cluster carries, Setup-Cluster walks through a deployment, and docs/CLUSTER.md is the full reference.

Example

Three nodes, 10.0.0.1 to 10.0.0.3. This is the first node's section:

[cluster]
enabled = true
node_id = "node-1"                          # Unique per node (default: the host name)
bind = "10.0.0.1"                           # This node's IP address (default: "0.0.0.0")
port = 7946                                 # Gossip port, TCP and UDP (default: 7946)
secret_key = "base64-encoded-32-byte-key"   # Generate: openssl rand -base64 32
peers = ["10.0.0.2:7946", "10.0.0.3:7946"]  # The other ALPS nodes
# allow_insecure = false                    # Only for a fully trusted private network

On the other nodes, only node_id, bind and peers change. secret_key is the same on all three.

Keys

Key	Type	Default	What it does
enabled	Boolean	false	Joins this node to the gossip cluster.
node_id	String	The host name	This node's name in the cluster. Must be unique and should stay the same across restarts, because the smallest name leads.
bind	String	"0.0.0.0"	The IP address to listen on for gossip, written as "IP" or "IP:port". A port written here wins over port. Anything else stops startup.
port	Integer	7946	The gossip port, used on both TCP and UDP. 0 also means 7946.
secret_key	String	None	The key that encrypts and authenticates the gossip: standard base64 of exactly 32 bytes, the same on every node. Required unless allow_insecure = true. CLUSTER_SECRET_KEY takes precedence.
allow_insecure	Boolean	false	Lets the cluster run without a secret key.
peers	Array of strings	None	The other nodes' gossip addresses, as "IP:port". Joined at startup, and every 30 seconds after that, each one that is not a member at the time.

ALPS ignores keys it does not know, and logs a warning for each at startup. A misspelt key such as peer = [...] therefore leaves its default in place, and the log says Unknown configuration key: cluster.peer (ignored). Compare the section with config.example.toml.

enabled and startup

With enabled = true, ALPS starts the cluster before the TLS manager and the HTTP server. If the cluster cannot start, ALPS exits with status 1. It does not fall back to running as a standalone node. These stop startup:

Cause	Message starts with
A bind that is not an IP address, or a port outside 1 to 65535	Error generating options: cluster.bind: must be an IP address with an optional port, got "node1.example.com", or cluster.bind: port must be between 1 and 65535, got ...
No secret key, and allow_insecure not set	failed to initialize cluster (clustering is enabled in config, refusing to run ungated): cluster secret_key is required
A key that is not valid base64	failed to decode cluster secret key: failed to decode secret key: illegal base64 data at input byte ...
A key that does not decode to 32 bytes	failed to decode cluster secret key: secret key must be 32 bytes when decoded, got ...
A bind address the host does not have, or a gossip port already in use	failed to initialize cluster (...): failed to create memberlist: ...
bind = "0.0.0.0" on a host with no private address	failed to initialize cluster (...): failed to create memberlist: failed to get final advertise address: no private IP address found, and explicit IP not provided
[tls] enabled = true and provider = "letsencrypt", with a storage_provider other than "s3" (the default)	TLS letsencrypt storage_provider must be "s3" when clustering is enabled

ALPS checks bind while it reads the file, before the log is opened, so that message goes to stderr as plain text. It logs the others at level ERROR to the [logging] output, in the format it sets: by default JSON on standard output, with the text above in msg. With Let's Encrypt, every node needs the same S3 storage. See Config-TLS and Certificates in a cluster.

ALPS makes the storage_provider check, and every later startup step, after the node has joined its peers. A node that fails at one of them leaves the cluster again before it exits, as at a normal shutdown, so the other nodes log node joined cluster and then node left cluster for it, and work out the leader again. See Startup and shutdown.

ALPS reads [cluster] only at startup. systemctl reload alps sends SIGHUP, which only reopens the log file (SIGHUP), so restart the node to apply a change. Restarts and upgrades covers which changes can be made one node at a time.

A cluster node takes up to 40 seconds to stop, or up to 55 with Let's Encrypt, since it waits for the leave to go out. The shipped systemd unit's TimeoutStopSec=60 allows for that (SIGINT and SIGTERM).

node_id

node_id defaults to the host name. The leader is the member with the smallest node_id, and only Let's Encrypt uses the leader (Leader election).

• Names are compared as plain strings, so node-10 sorts before node-2. Pad the numbers (node-01, node-02) if you want the order to follow them.
• Keep each node's name stable. Renaming a node can move the leadership.
• Set node_id explicitly when the hosts may share a name, for example containers that are all given the same host name.

At the default log level, ALPS does not report two nodes with the same node_id. The second node starts and logs cluster started as usual, but the other nodes keep only one member under that name. If the shared name is the smallest one, both nodes report is_leader=true. To check, compare node_name in each node's cluster started line. With a duplicate, the members= counts stay lower than the number of nodes. With [logging] level = "debug", the gossip library's own lines appear too, at level DEBUG, and the nodes that hear of both log [ERR] memberlist: Conflicting address for node-1. Mine: ... Theirs: ....

bind and port

bind takes an IP address, with or without a port:

• bind = "10.0.0.1" listens on that address and port port.
• bind = "10.0.0.1:7950" listens on port 7950, whatever port says. An IPv6 address with a port goes in brackets, "[fd00::1]:7950".
• port = 0, or no port, means 7946.

The node listens on that port over both TCP and UDP. Every node must reach every other node's gossip port over both. Open the port between the ALPS nodes only.

The address the node gives its peers depends on bind:

• With a specific address, the node advertises that address.
• With 0.0.0.0 (the default), the node advertises a private address of the host, detected automatically. If the host has no private address, ALPS exits. If it has more than one, set bind to the one the peers can reach.

ALPS has no separate setting for the advertised address. The other nodes must reach each node at its bind address, or at its detected private address. A node behind address translation cannot be reached at the address it advertises, for example a container that the other nodes reach only through a port published on its host. Run such nodes on a network where their own addresses are reachable, or on the host network.

Nodes may use different gossip ports. A peer entry then needs that node's port (see peers).

Host names and interface names do not work in bind. ALPS refuses them before it starts, with Error generating options: cluster.bind: must be an IP address with an optional port, got "eth0" on stderr and exit status 1. Set bind to the node's own address, or leave it at 0.0.0.0.

secret_key and CLUSTER_SECRET_KEY

The key encrypts and authenticates all gossip traffic (AES-256). It must be standard base64 that decodes to exactly 32 bytes, and the same on every node. Generate one with:

openssl rand -base64 32

The output is 44 characters long and ends in =. Keep the =: without it, the value is not valid base64 and ALPS exits with illegal base64 data at input byte 40. URL-safe base64 (with - or _) is not accepted either.

When the CLUSTER_SECRET_KEY environment variable is set to a non-empty value, ALPS uses it instead of secret_key. This keeps the key out of the config file:

• A set, non-empty CLUSTER_SECRET_KEY always wins. An invalid value stops ALPS, even when secret_key holds a valid key.
• An empty CLUSTER_SECRET_KEY= counts as unset, and secret_key applies.

The shipped systemd unit (dist/alps.service) sets no environment variables. One way to add the key is a drop-in, created with systemctl edit alps, that reads it from a file only root can read:

[Service]
EnvironmentFile=/etc/alps/cluster.env

Here /etc/alps/cluster.env holds one line, CLUSTER_SECRET_KEY=<the key>, and is owned by root with mode 600. systemd reads it before it starts ALPS, so the alps user does not need to read it.

Keep the key secret, like [server] login_key (Config-Server). Nodes with different keys do not form one cluster: at startup, the node that tries to join logs failed to join some peers (may be first node) with no installed keys could decrypt the message. To change the key, give every node the new key and restart them all. Until the last one has restarted, the nodes with the new key and the nodes with the old one form separate clusters, each with its own leader (secret_key and CLUSTER_SECRET_KEY).

allow_insecure

Without a secret key, the cluster does not start and ALPS exits, unless allow_insecure = true. With allow_insecure = true and no key, the gossip runs unencrypted and unauthenticated, and ALPS logs this warning at startup (here in the default JSON format, time field left out):

{"level":"WARN","msg":"gossip encryption DISABLED (allow_insecure set) - cluster communication is INSECURE"}

With a key set, the key is used and the gossip is encrypted, whatever allow_insecure says.

Nodes with a key and nodes without one do not form one cluster. A keyless node that tries to join a keyed one logs remote state is encrypted and encryption is not configured.

Leave allow_insecure off in production. Set it only on a fully trusted private network, and set a key wherever you can.

peers

peers lists the other nodes' gossip addresses, best as "IP:port":

• Startup. A node tries to join its peers at startup. If none of them answers, ALPS logs failed to join some peers (may be first node) and starts anyway. That is normal for the first node to start. Once a node reaches any one member, it learns about the rest of the cluster.
• Retry. Every 30 seconds, a node compares its peers with the addresses of its current members, and tries to join each peer that is missing. It logs cluster rejoined peers when it reaches one. A retry that fails logs only at debug level, as cluster rejoin attempt failed. A node that started before all of its peers therefore joins them once one of them is up, and the sides of a network split, however many nodes each side has, find each other again within about 30 seconds of the network coming back.
• Write IP:port. A peer counts as a member when its entry is exactly the address and port that member advertises, such as "10.0.0.2:7946". An entry by host name, or without a port, never matches, so it is joined again every 30 seconds, each time with a cluster rejoined peers line (#952).
• Host names. A host name is looked up at each attempt. A name that resolves to several addresses joins every one of them.
• No port. An entry without a port uses this node's own gossip port.
• Own address. Listing the node's own address does no harm, so every node can carry the same full list.
• No peers. A node without peers never joins anyone itself, and never tries again. It becomes a member only when a node that lists it joins it, which that node does at its own startup and every 30 seconds while the node is missing from its members.

Give every node peers, and list every other node, so that a node can find its way back whichever of them is up. Leader election has the details.

peers is used for nothing else. Client addresses behind a load balancer depend only on [server] trusted_proxies.

Settings every node must share

Every node needs [cluster] enabled = true and the same secret key, a node_id of its own, and the same [server] login_key (it also encrypts the separate CalDAV and CardDAV passwords users save) and dav_allow_private_servers, [webauthn] rpid and origins, [server.rate_limit] values, [provider], [smtp] and [plugin.<name>] sections, Let's Encrypt settings and ALPS version, with clocks kept in sync. Settings every node must share has the table, with why each one matters.

bind, port, peers and [server] addr may differ per node. When the load balancer is an HTTP proxy, list it in [server] trusted_proxies on every node, and list only your own proxies there (Setup-Behind-a-Reverse-Proxy).

Checking the configuration

The cluster's lines go through [logging] like every other line: to its output, in its format, and only when their level reaches its level. Under the shipped systemd unit with the default output, they are in journalctl -u alps (Config-Logging-and-Cache). Each node logs a cluster started line at startup, shown here with format = "console" and the time left out:

level=INFO msg="cluster started" node_name=node-3 bind_addr=0.0.0.0 bind_port=7946 advertise_addr="" advertise_port=7946 members=3 leader=node-1 is_leader=false

• node_name is different on every node.
• bind_addr and bind_port are what you meant. advertise_addr is empty with bind = "0.0.0.0", because the detected address is not shown. The node's own node joined cluster line, a few lines earlier, shows the address it advertises.
• The last node to start reports members= equal to the number of nodes.
• Every node's latest leader or new_leader names the same node, and only that node reports is_leader=true.

When no peer answered, the failed to join some peers line says why:

Error in the line	Meaning
no installed keys could decrypt the message	This node's secret key differs from the peer's.
remote state is encrypted and encryption is not configured	This node has no key, and the peer has one.
connection refused	Nothing listens at that address and port: the peer is not up yet, or the port in peers is wrong.
no such host	The peer's host name does not resolve (yet).

Only the node that tried to join logs these. The node it tried logs nothing about it.

UDP

A node that the others reach over TCP but not over UDP still joins and stays listed as a member, because the membership checks fall back to TCP. ALPS sends its own messages, the rate-limit events and the security-key enrolment notices, over UDP. Those messages then do not arrive, and at the default log level no node logs a warning. With [logging] level = "debug", the gossip library's own lines appear, at level DEBUG, and the node that probes another over UDP in vain logs [WARN] memberlist: Was able to connect to <node> over TCP but UDP probes failed, network may be misconfigured. A cluster that forms is therefore no proof that UDP works.

To check UDP end to end, with rate limiting on (the default), send six sign-ins with a wrong password within one minute: five straight to node A's own HTTP address, then one to node B's, not through the load balancer. Run this from a machine that is not listed in trusted_proxies, and use usernames that do not exist on your mail server:

NODE_A=http://10.0.0.1:1323   # node A's own [server] addr
NODE_B=http://10.0.0.2:1323   # node B's own [server] addr
for i in 1 2 3 4 5; do
  curl -s -o /dev/null -w "A: %{http_code}\n" --data-urlencode "username=cluster-check-$i@example.com" \
    --data-urlencode "password=wrong" "$NODE_A/session"
done
curl -s -o /dev/null -w "B: %{http_code}\n" --data-urlencode "username=cluster-check-6@example.com" \
  --data-urlencode "password=wrong" "$NODE_B/session"

Use https:// when the nodes serve TLS themselves. Node A refuses each attempt with 401. Node B should answer 429 (Too many login attempts, please wait a minute), because it counted node A's five attempts towards the address's per-minute limit (ip_requests_per_minute, 5 by default; send that many to node A if you changed it). If node B answers 401, node A's events did not reach it: check UDP on the gossip port in both directions.

Wait a minute before trying again. These attempts count towards that address's limits like any others: each run adds at least five refused attempts to the address's hourly count (ip_requests_per_hour, 20 by default), and once that count is reached, the address is locked out for lockout_minutes (15 by default). Run the check only a few times an hour. The mail server also sees the failed sign-ins, from the nodes' addresses. See Config-Rate-Limiting-and-WebAuthn.

Checking the cluster lists the other cluster lines and has a sample log of a healthy join.

See also

• Clustering: what the cluster carries, node-local state, certificates, restarts and upgrades.
• Setup-Cluster: a deployment from start to finish.
• Configuration-Overview and Running-in-Production.
• docs/CLUSTER.md and the [cluster] table in CONFIGURATION.md.