High Availability Runbook

High Availability Runbook

Source of truth: hmailserver/docs/HighAvailabilityRunbook.md, kept in the repository and copied here. This wiki copy adds the diagrams, the probe-lifecycle table, the load-balancer configurations and the worked commands (sections marked wiki addition); everything else is the repository document verbatim. Checked against 6.2.28 on 8 September 2026. The repository text was re-copied from master 92b8f9f63 on 26 September 2026, and is the same at the tag v6.3.4 (51e2dbda0), the release; what it now says about stored settings, the stored-secrets key, the drain and UpgradeOnStart is new in 6.3.4, and the page says what 6.3.3 does instead. Re-copied on 5 October 2026 from the document as the tag v6.3.6 (e3c448aa6) has it (HighAvailabilityRunbook.md at v6.3.6): from 6.3.6 the five settings of the live update's trust live in the file only, as PasswordPepper does, and ActiveActive.md designs a topology that is not built. Re-copied on 18 October 2026 from the document as the tag v6.3.7 (31d656c0c), the release, tagged and published on 17 October 2026, has it (HighAvailabilityRunbook.md at v6.3.7): behind a TCP load balancer each of the SMTP, IMAP and POP3 listeners can take the client's address from a PROXY protocol header sent by a peer on its own list, with what that asks of the balancer's checks of the listener, and the /readyz check of the metrics listener sends no such header.

This runbook describes a supported, operationally simple active/passive high-availability topology for hMailServer. It deliberately contains no clustering code: hMailServer runs as a single active node at a time, and failover is performed by external infrastructure (a shared database, shared message storage, a floating virtual IP, and a health check). This keeps the mail server itself simple and avoids split-brain message corruption.

For an active/active or sharded design you would need a different storage and locking model; that is out of scope here. ActiveActive.md designs one for two or more nodes on one database. It is not built.

---

1. Topology

                 ┌─────────────────────────────┐
   clients ─────►│  Virtual IP (VIP) / load     │
   (SMTP/IMAP/   │  balancer with health check  │
    POP/REST)    └──────────────┬───────────────┘
                                │  (routes to whichever node is "ready")
                ┌───────────────┴───────────────┐
                │                                │
        ┌───────▼────────┐              ┌────────▼───────┐
        │  Node A (active)│              │ Node B (passive)│
        │  hMailServer    │              │  hMailServer    │
        │  service RUNNING│              │  service STOPPED │
        └───────┬─────────┘              └────────┬────────┘
                │                                  │
                └──────────────┬───────────────────┘
                               │
              ┌──────────────────────────────────────┐
              │  Shared database (MSSQL/MySQL/PG)    │
              │  Shared message store ([Directories] │
              │  DataFolder)                         │
              └──────────────────────────────────────┘

Exactly one node runs the hMailServer service at any time. Both nodes are configured identically and point at the same database and the same message-store directory.

Wiki addition. The same picture with the control plane shown — which arrow decides where the traffic goes, and where the state that must not be written twice lives:

flowchart TB
    C["Clients: SMTP 25 and 587, IMAP 143 and 993, POP3 110 and 995, REST"]
    LB["VIP or load balancer"]
    C --> LB
    LB -->|"mail and API traffic"| A["Node A - service RUNNING"]
    LB -.->|"health check on /readyz"| A
    LB -.->|"health check on /readyz - refused, so never routed to"| B["Node B - service STOPPED"]
    A --> DB[("Shared database")]
    A --> FS[("Shared message store - DataFolder")]
    B -.->|"only once it becomes active"| DB
    B -.->|"only once it becomes active"| FS
    A -->|"/metrics"| M["Prometheus and alerting"]

Loading

The dotted arrows are the whole design. Node B is configured and ready but is not running a service, so its /readyz connection is refused, so the load balancer never routes to it — no clustering protocol, no quorum, no split-brain arbitration inside hMailServer.

Who owns what:

Layer	Owned by	Failure here looks like
Deciding which node is live	Your VIP or load balancer, driven by /readyz	Traffic to a node that cannot serve, or to both
Making sure only one node writes	You, through fencing	Duplicate deliveries and corrupted mailbox state
Reporting whether a node can serve	hMailServer's probes	A node kept in rotation during a database outage — which is what the readiness probe's staleness ceiling exists to prevent
Keeping the state consistent	The shared database and the shared store, on one failover boundary	Rows whose message files are missing

---

2. Shared components

2.1 Shared database

• Use an external database server (Microsoft SQL Server, MySQL, or PostgreSQL), not the built-in SQL Compact / internal database, which is local-only.
• Both nodes use the same hMailServer.INI [Database] connection settings.
• The database server should itself be made highly available (e.g. SQL Server Always On / failover cluster, managed RDS/Cloud SQL with HA), or hosted on the same shared-storage layer as the message store.

2.2 Shared message store

• hMailServer stores message files under the data directory, which is hMailServer.INI → [Directories] → DataFolder (there is no setting called DataDirectory; that is the name of the accessor in the code). Both nodes must see the same directory on shared storage (SAN/NAS/clustered file system, or a cloud file share). Note that this key lives in the per-node ini file, not in the shared database, so it is one of the per-node values section 4 leaves you to keep in step. The database row for each message references its on-disk path; the database and the message store must therefore stay consistent with each other, so keep them on the same failover boundary.
• Ensure both nodes' service accounts have identical read/write access to the share.

2.3 Virtual IP / load balancer

• Clients connect to a floating VIP (or a load balancer / DNS name) rather than a node's real address.
• The VIP must point at the node that is ready (see health checks below).
• Only ever direct traffic to one node at a time. If you use a load balancer that can see both nodes, gate routing strictly on the readiness probe so the passive node (service stopped → probe fails/refused) never receives traffic.
• A TCP load balancer hides each client's address from the node: every connection comes from the balancer, and the IP ranges, the auto-ban and an account's list of devices act on it. Each of the SMTP, IMAP and POP3 listeners can take the client's address from a PROXY protocol header (HAProxy: send-proxy or send-proxy-v2 on the server line) sent by a peer on a list of its own: SMTPProxyProtocolEnabled and SMTPProxyProtocolTrustedIPs, IMAPProxyProtocolEnabled and IMAPProxyProtocolTrustedIPs, POP3ProxyProtocolEnabled and POP3ProxyProtocolTrustedIPs (the Control Panel's Protocols page has the IMAP and POP3 pairs). A listed peer must then send the header on every connection to that listener, its health checks of the listener included. HAProxy sends it on its checks by itself when the server line has send-proxy or send-proxy-v2 and the check names no port or addr - with send-proxy a version 1 header that names the balancer itself, with send-proxy-v2 a version 2 header with the LOCAL command - and the node takes either as a connection of the balancer's own. A check of the listener itself that names a port or an addr needs check-send-proxy, or it waits for a greeting that never comes and times out: the balancer marks the node down, and the node's error log says nothing. Only a check that sends before the server speaks, with no header, is an invalid header (HM6280 in the error log). A plain TCP check closes before the header is due and is only noted in the TCP/IP log.
• The /readyz check of section 3 is not a check of those listeners. It names the metrics listener's port, and must not have check-send-proxy: that listener reads no PROXY header, and answers a request with one in front 404 Not Found (with TLS on, the handshake fails and the connection is closed unanswered), which marks the node down. Without check-send-proxy, a check that names a port sends no header.

---

3. Health checks and readiness gating

hMailServer exposes Kubernetes-style probes on the metrics listener. Enable it on both nodes. On 6.3.3 these are lines of each node's hMailServer.INI, as shown (see section 4 for keeping the two files' [Settings] sections the same). Since 6.3.4 they are stored settings: set them in the Control Panel, over /api/v1/settings/ini, with hmctl, or on a Linux node whose REST listener is not yet on with hmailserver --set-setting NAME=VALUE, then reinitialise the server (the Control Panel's or the Deck's Apply now, or POST /api/v1/server/reinitialize) or restart the service: the listener's port and address are bound when it starts, so they wait for one of those (SettingsApply.md). They are shown below in the [Settings] shape they have always been written in, and that section of each node's hMailServer.ini still holds a copy — but since schema 6042 it is a copy the server maintains, not the place to change them.

[Settings]
MetricsServerPort=8080
MetricsServerBindAddress=0.0.0.0    ; reachable by the load balancer / health checker
MetricsServerAuthToken=<32+ random characters>   ; see below - required for /metrics on a non-loopback bind
ShutdownDrainSeconds=30             ; let in-flight sessions finish on a graceful stop

The three probes never require a credential, and the exposition now does. On a non-loopback bind, /metrics answers 503 until MetricsServerAuthToken (or the MetricsServerAuthUsername/MetricsServerAuthPassword pair) is set, and the 503 body names the settings that would open it. /livez, /readyz and /healthz are served unauthenticated in every configuration, because a load balancer cannot present a credential and a health check cannot hold a secret — so the VIP configuration in this runbook keeps working exactly as written, with or without the token.

The reason the exposition is treated differently: on 0.0.0.0 it publishes queue depth, session counts and authentication-failure counts to anything that can reach the port. Add the token, and give the scraper an Authorization: Bearer <token> header. If you want the metrics port encrypted as well, set MetricsServerCertificateFile and MetricsServerPrivateKeyFile; without them the listener stays plain HTTP and says so in the application log rather than refusing to start.

Two properties of this listener to design the health check around, both deliberate:

• MetricsServerBindAddress takes an IP literal and nothing else. It is parsed with inet_pton, so 0.0.0.0, 127.0.0.1, :: and a specific IPv6 address work, while a host name or localhost is rejected — the listener logs MetricsServer: Invalid bind address and does not start, which takes the probes with it. :: serves both families (the listener clears IPV6_V6ONLY and says so in the log if it cannot); a specific IPv6 literal serves IPv6 only. If your health check gets a refused connection on a node whose service is running, this is the first thing to check.
• It is a single accept loop serving one connection at a time. Probes are cheap and answered before anything that can refuse, but a scrape and a probe are still serialised behind each other. Keep the probe interval and its timeout comfortably apart (the listener bounds a request read at 5s and a response write at 15s), and do not point a sub-second health check at it.

Probes (HTTP):

Path	Meaning	Use for
/livez	Process is alive (200 whenever the listener is up).	Liveness restarts.
/readyz	200 only when the server is Running and the database is connected. Returns 503 while the server is stopping (not ready: server not in running state), while it is draining (not ready: draining - maintenance mode, entered with POST /api/v1/server/drain and by the live update before it stops the service, in which the SMTP, IMAP and POP3 listeners are closed and the delivery queue paused while what is in flight finishes; this listener stays up), or if the database connection is lost. During startup it is not 503 but refused: this listener is brought up only after the state has already gone to Running (it is the first of the optional listeners started, ahead of the REST API and web services), so there is nothing listening until the server is ready. All three read as unhealthy to a load balancer, which is all that matters here.	VIP / load-balancer routing.
/api/v1/server/drain (REST listener)	GET answers {"draining","since","sessions","deliveries","drained"}; POST enters the drain and DELETE leaves it, reopening the listeners on the same ports. drained is true once nothing is in flight, which is when a stop would be instantaneous.	Planned failover, section 5.
/healthz	JSON: status (ok/unavailable), state, database (up/down) and uptime_seconds. 200 when running with the database up, 503 otherwise. Per-protocol session counts are not in it — they are hmailserver_sessions on /metrics, behind its credential.	Dashboards / debugging.

Configure the VIP/load balancer health check against /readyz. Because the passive node's service is stopped, its /readyz connection is refused (unhealthy) and it will never be routed to. During a graceful stop, the active node flips /readyz to 503 before tearing down listeners, so the load balancer drains it cleanly.

The check names the metrics listener's port, and sends no PROXY protocol header even where the mail listeners take one (section 2.3): this listener reads none, and answers a request with one in front 404 Not Found - or, with MetricsServerCertificateFile set, fails the TLS handshake and closes the connection. In HAProxy, for example, server n1 10.0.0.1:143 send-proxy-v2 check port 8080 with option httpchk GET /readyz, and no check-send-proxy on the line.

3.1 What each probe says at every point in a node's life (wiki addition)

This is the table to design the health check's thresholds around. "Refused" means the TCP connection is refused because nothing is listening.

Node's condition	/livez	/readyz	/healthz	The load balancer's view
Service stopped (the passive node)	refused	refused	refused	Unhealthy. Never routed to
Service starting, before the state reaches Running	refused	refused	refused	Unhealthy. The listener is created only after the state is Running, so start-up is silence rather than a 503
Running, database answering	200 alive	200 ready	200, "status":"ok"	Healthy. Routed to
Running, database stopped answering	200 alive	503 not ready: database did not answer the last readiness probe	503, "database":"down"	Shed within roughly 5–20 s, depending on which of the three readiness conditions trips first
Running, database hanging rather than failing	200 alive	503 not ready: database readiness probe has not completed recently	503	Shed after 20 s. This is the case a naive probe misses entirely
Graceful stop, inside the drain window	200 alive	503 not ready: server not in running state	503, "state":"stopping"	Shed while existing sessions finish — the point of ShutdownDrainSeconds
Graceful stop, past the drain	refused	refused	refused	Unhealthy

/livez staying 200 through a database outage is deliberate: liveness restarts a process that is wedged, and killing a healthy mail server because its database blinked makes the outage worse.

3.2 Health-check configuration (wiki addition)

The probe is HTTP/1.0 and closes the connection, and the listener serves one connection at a time — so give the check a timeout of a second or two and an interval of five seconds or more, and never a sub-second check.

HAProxy (TCP mode for the mail ports, health-checked over HTTP on the metrics port):

backend mail_smtp
    mode tcp
    option httpchk GET /readyz
    http-check expect status 200
    default-server inter 5s fall 2 rise 2
    server nodeA 10.0.0.11:25 check port 8080
    server nodeB 10.0.0.12:25 check port 8080 backup

nginx stream (nginx Plus is required for active health checks; open-source nginx relies on max_fails/fail_timeout against the mail port itself):

stream {
    upstream mail_smtp {
        server 10.0.0.11:25 max_fails=2 fail_timeout=10s;
        server 10.0.0.12:25 backup;
    }
    server { listen 25; proxy_pass mail_smtp; }
}

AWS target group / Azure Load Balancer: protocol HTTP, port 8080, path /readyz, healthy threshold 2, unhealthy threshold 2, interval 5 s, timeout 2 s, success codes 200.

Kubernetes, if you run the service in a container:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Add scheme: HTTPS when MetricsServerCertificateFile and MetricsServerPrivateKeyFile are set — TLS applies to the whole port, probes included.

Verify the whole gate before you rely on it:

# On the ACTIVE node - all three must answer, with no credential.
Invoke-RestMethod http://10.0.0.11:8080/livez
Invoke-RestMethod http://10.0.0.11:8080/readyz
Invoke-RestMethod http://10.0.0.11:8080/healthz

# On the PASSIVE node - this must be REFUSED, not 200.
Test-NetConnection 10.0.0.12 -Port 8080

# From the load balancer's own vantage point, watch readiness flip during a stop.
while ($true) {
    $t = Get-Date -Format "HH:mm:ss"
    try   { "$t $((Invoke-WebRequest http://10.0.0.11:8080/readyz -UseBasicParsing -TimeoutSec 2).StatusCode)" }
    catch { "$t $($_.Exception.Response.StatusCode.value__) $($_.ErrorDetails.Message)" }
    Start-Sleep -Seconds 2
}

---

4. Normal operation

• Active node (A): hMailServer service set to Automatic and running.
• Passive node (B): hMailServer service set to Manual (or Disabled) and stopped. Keep the binaries in sync with A (same version - the schema pin refuses a mismatch in both directions) and point it at the same database and data directory.
• Apply configuration changes on the active node. The configuration lives in the shared database - since schema 6042 that includes the server settings that were the [Settings] section of hMailServer.INI, which are rows of hm_inisettings - and B reads it when it starts. The [Settings] section of B's own INI is only the server's copy: B brings it up to date at that start, naming each setting changed since as HM5804 in its error log, which is the copy catching up rather than a fault (a setting added since is written in with an application-log note). Do not edit [Settings] on B to make it match; the edit would be undone, and a line B has for a setting A has deleted is adopted back into the shared store when B starts.
• What is per node, and is yours to keep in step by hand or with your configuration-management tooling: [Directories], [Database] and [Security] in each node's INI (the paths, the connection, the administrator password and its second factor); PasswordPepper, a [Settings] key that lives in the file only and must be identical on both nodes or no account whose password is an Argon2id or scrypt hash signs in on B; from 6.3.6 the five settings of the live update's trust, where a site sets them, which live in the file only as well (The Update Feed, "Where the trust is read from"); the stored-secrets key, [Security] StoredSecretsKey, which every account's second factor and the [Settings] secrets in the shared database are sealed under once the installation has one, and which must be the database's key on both nodes or every second factor refuses every code on B and SRS, BATV, HS256 tokens and /metrics stop working there (Secrets at Rest). It is made once, by an elevated step - --create-stored-secrets-key, or the installer on a new installation - never by a service's start, and the database records which key it is, so B can neither make one of its own beside A's nor have one imported that is not the database's. Carry it with hMailServer.exe --export-stored-secrets-key FILE on A and --import-stored-secrets-key FILE on B, never by copying the line, which is sealed under A's own DPAPI, and keep the exported file with the database backups; and the certificate and DKIM key files at the paths the database names. Do not copy A's INI over B's wholesale - on Windows [Database] Password is normally sealed with the machine's own DPAPI key, and a [SettingsOverride] section, if A has one, is per node on purpose.
• Wiki addition. On 6.3.3 the [Settings] section is not in the shared database. What the database shares there is what the COM Settings object stores (hm_settings), the domains, the accounts and the rest; the [Settings] section is each node's file, mirrored in hm_inisettings as one shared record of what "the file" last held, so B starting with an older section takes its own old values and writes them over A's (Warm Standby has the mechanism). On 6.3.3, copy A's [Settings] section into B's INI after every change to it. The store in the database is schema 6042, since 6.3.4; with it too, [LDAP], [SendingLimits] and [SendingLimitsOverrides] stay in each node's file, because the store has not taken them over (RelocatingAnInstallation.md, step 4), so keep them the same on both nodes by hand.

---

5. Planned failover (maintenance)

The drain as a state of its own (step 1) and UpgradeOnStart (step 5) are new in 6.3.4. On 6.3.3, step 1 is a plain stop of A with ShutdownDrainSeconds set: A reports /readyz = 503 and waits up to the drain window for the sessions in flight before it shuts down.

1. Drain A: POST /api/v1/server/drain on A (manage_server; the Deck's session - its update view has a Drain button - or the administrator password). A closes its SMTP, IMAP, POP3 and ManageSieve listeners - a new connection is refused at the socket, which every MTA retries on its own - pauses its delivery queue, and reports /readyz = 503 draining, so the load balancer stops sending it anything. The sessions and deliveries in flight finish at their own pace; A's REST, metrics and web-services listeners stay up throughout (the Deck, the webmail, CalDAV and CardDAV keep answering; see Upgrading Guide for what a person sees), so you can watch: GET /api/v1/server/drain until drained is true. Nothing is cut, and nothing is accepted only to be cut, which is what a plain stop with ShutdownDrainSeconds used to do.
2. Stop A: stop the hMailServer service on A. It finds the doors already closed and, with ShutdownDrainSeconds set, waits up to the window for anything still in flight - nothing, if drained was true - before shutting down. A stop after a drain does not wait again for what has already drained.
3. Verify A is down: /readyz on A is refused.
4. Move the VIP to B (or let the load balancer health check do it).
5. Start B: start the hMailServer service on B. Wait until /readyz on B returns 200. If this is an upgrade and B's binaries are newer than the shared schema, B refuses to start (HM5011) unless its hMailServer.ini says [Database] UpgradeOnStart=1, in which case B applies the chain at its own start - see Upgrading Guide. Take the database backup before this step, not after. An upgrade to a version that seals stored secrets makes no stored-secrets key: sealing stays off until you make one on one node (--create-stored-secrets-key), export it and import it on the other (Upgrading Guide). A node that starts without the database's key once there is one says so as HM7495 at each start, seals nothing and makes no key of its own. One node at a time: start B and let it finish the chain before A's binaries are replaced and A is started again; A then reads the new version and does nothing. Two nodes that start together are serialised on the version row and the second finds each step done, but a warm-standby pair should never need that.
6. Check the key on B before the VIP is B's for good: B's error log has no HM7495, HM7492 or HM7493 from this start. /readyz is 200 either way - a node without the key serves mail, and only its users with a second factor, and SRS, BATV and HS256, fail - so this is the check that finds it. If B says HM7495, stop it, run hMailServer.exe --import-stored-secrets-key FILE on B with the file exported from A (or kept with the backups), and start it again.
7. Confirm: send a test message through the VIP and confirm delivery; check /healthz reports database: up and state: running.

Reverse the steps to fail back - step 6 included, on A: after an upgrade the installer starts A on its own before you can import anything, so when A comes back check its error log for HM7495, HM7492 and HM7493 before the VIP moves back, and import the key there first if it asks for it. To take A off the air for a moment without a failover at all - a certificate on disk to swap, a disk to grow - step 1, then DELETE /api/v1/server/drain to reopen the same ports: that is maintenance mode, and nothing was stopped.

Wiki addition. The same five steps as a timeline, with what each participant sees:

sequenceDiagram
    autonumber
    participant OP as Operator
    participant LB as Load balancer
    participant A as Node A - active
    participant B as Node B - passive
    participant DB as Shared database and store
    Note over A: state Running, /readyz 200
    OP->>A: Stop-Service hMailServer
    A->>A: state goes to Stopping
    LB->>A: GET /readyz
    A-->>LB: 503 not ready - server not in running state
    LB->>LB: mark A unhealthy, stop sending new connections
    Note over A: drain window - up to ShutdownDrainSeconds
    A->>A: existing SMTP, IMAP and POP3 sessions finish
    A->>A: metrics listener stops, then the mail listeners
    A->>DB: releases every lock as the service exits
    OP->>A: confirm /readyz is now REFUSED
    OP->>B: Start-Service hMailServer
    B->>DB: connect, check schema version, unlock the queue
    B->>B: state goes to Running, listeners up
    LB->>B: GET /readyz
    B-->>LB: 200 ready
    LB->>LB: mark B healthy, route traffic
    OP->>B: send a test message through the VIP, read it back

Loading

Step 12 — unlock the queue — is the one to understand: at start-up the server sets messagelocked = 0 on every queued message, because locks held by a node that crashed would otherwise strand that mail for ever. It is also exactly why two nodes must never run at once. See Warm Standby for what happens if they do.

The commands, in order:

# 1-3. On A: drain and stop. The service does not return until the drain has run.
Stop-Service hMailServer
Get-Content "C:\Program Files\hMailServer\Logs\hmailserver_$(Get-Date -Format yyyy-MM-dd).log" -Tail 20 |
  Select-String "Graceful shutdown"

# 4. Prove A is down from the load balancer's point of view.
Test-NetConnection 10.0.0.11 -Port 8080     # TcpTestSucceeded must be False

# 5. On B: start, then wait for readiness rather than assuming it.
Start-Service hMailServer
do {
    Start-Sleep -Seconds 2
    $ready = $false
    try { $ready = (Invoke-WebRequest http://10.0.0.12:8080/readyz -UseBasicParsing -TimeoutSec 2).StatusCode -eq 200 } catch {}
} until ($ready)
Invoke-RestMethod http://10.0.0.12:8080/healthz

The drain lines to look for in the application log are Graceful shutdown: draining N active session(s), up to Ns... followed by either Graceful shutdown: all sessions drained. or Graceful shutdown: drain window elapsed with N session(s) still active; stopping anyway. With ShutdownDrainSeconds=0 (the default) none of them appear, because there is no wait at all.

---

6. Unplanned failover (node A fails)

1. The load balancer health check against /readyz on A fails; stop routing to A (automatic if the LB is health-check driven).
2. Fence A to guarantee it cannot still be writing to the shared store: power it off / isolate it from the storage and database network. This prevents split-brain (two active nodes writing the same message store).
3. Move the VIP to B.
4. Start the hMailServer service on B and wait for /readyz = 200.
5. When A is repaired, bring it back as the new passive node (service stopped) before any future failover.

Wiki addition. If A comes back as a new machine rather than a repaired one, its hMailServer.INI is a fresh file, and what that file must hold before A's service first starts depends on the release. On 6.3.3, copy B's [Settings] section into it (and write its own [Database] and [Directories]): a fresh file with no [Settings] keys makes the mirror drop every agreeing row from the shared hm_inisettings and run on defaults. Since 6.3.4, the [Settings] values come from the shared database and are written into the new file at that first start; what has to be written by hand is the per-node sections of section 4 and the PasswordPepper line, which no database and no backup holds - without it, every account whose password is hashed with Argon2id or scrypt is refused on A.

Split-brain safety: never let both nodes run the service against the shared store at the same time. Always fence the failed node before starting the standby. The shared database is the source of truth for message metadata; two active writers can corrupt mailbox state.

Wiki addition. The decision, with the fencing step where it has to be — before anything is started, not after:

flowchart TD
    A["Health check on A fails"] --> B{"Is A definitely dead?"}
    B -->|"powered off, or you can see it is"| D["Fence complete"]
    B -->|"unreachable, but you cannot prove it"| C["FENCE IT: power off the VM, pull the switch port, revoke its storage and database access"]
    C --> D
    B -->|"still running, database or store unreachable to it"| C
    D --> E["Move the VIP to B"]
    E --> F["Start the service on B"]
    F --> G{"/readyz on B is 200?"}
    G -->|"no"| G1["Read ERROR_hmailserver on B: schema mismatch, database credentials, or DataFolder not mounted"]
    G -->|"yes"| H["Send a test message through the VIP and read it back"]
    H --> I["When A is repaired, bring it back STOPPED, as the new passive node"]

Loading

Why the fence is not optional, in this product specifically. If A is alive but partitioned from you rather than from the storage, starting B gives two servers polling the same hm_messages table. Both unlock the whole queue at start-up, both select where messagelocked = 0 and messagenexttrytime <= now, and both take the lock after selecting — so they race that window and deliver the same messages. Every recipient gets duplicates and nothing anywhere looks wrong. The same shape applies to external fetch accounts, where a POP3 account collected twice with delete-after-download splits the mail arbitrarily between two stores. Warm Standby documents both mechanisms.

Recovery-time expectations, so the runbook can be held to a number:

Step	Typically
Health check declares A unhealthy	2 intervals — 10 s at the settings above
Fencing	However long your platform takes. This is usually the whole recovery time
VIP move	Seconds, or a DNS TTL if you use DNS instead of a VIP
B starts and reaches /readyz = 200	Seconds to a minute: connect to the database, check the schema, unlock the queue, bind the listeners
Mail queued at sending servers during the gap	Delivered on their retry. Sending servers retry for days; nothing is lost, it is delayed

---

7. Validation checklist

• Both nodes use the same external database and the same [Directories] DataFolder on shared storage.
• MetricsServerPort is enabled and reachable by the health checker on both nodes; the VIP health check targets /readyz.
• MetricsServerAuthToken is set on both nodes, and the scraper sends it. Without it /metrics answers 503 on a non-loopback bind — the probes and therefore the failover still work, so this fails quietly as missing dashboards, not as an outage.
• ShutdownDrainSeconds is set so planned failovers drain gracefully.
• PasswordPepper is identical in both nodes' hMailServer.INI (or absent from both), and neither node has a [SettingsOverride] section.
• Both nodes hold the database's stored-secrets key: the key id each prints when it exports it is the same, the setup checklist's stored_secrets_key item is done on both, and neither node's error log has HM7495, HM7492 or HM7493.
• The exported stored-secrets key is kept with the database backups, apart from them: without it a database restored after both nodes are lost opens no second factor and no [Settings] secret.
• Passive node's service is stopped and set to Manual/Disabled.
• (Wiki addition.) On 6.3.3, both nodes' [Settings] sections are identical. Since 6.3.4, both nodes carry the same PasswordPepper line (if any) and neither carries a [SettingsOverride] section.
• A documented fencing step exists for unplanned failover.
• A test message sent through the VIP is delivered after a planned failover in both directions.

7.1 The seven ways this is got wrong (wiki addition)

Each row is a real configuration that looks correct and is not.

What was done	Why it looks fine	What actually happens
MetricsServerBindAddress=mail.example.com	It is the node's own name	The value is parsed with inet_pton and must be an IP literal. The listener logs MetricsServer: Invalid bind address and does not start, so every probe is refused and the load balancer takes a healthy node out of rotation
Health check pointed at port 25 instead of /readyz	A TCP connect succeeds, so the node "is up"	A node whose database has gone away still accepts TCP on 25 and then rejects or defers every message. Only /readyz knows the difference
Health check interval of 1 s with a 1 s timeout	Faster detection	The listener serves one connection at a time; a scrape and a probe serialise. Use 5 s and 2 s
Both nodes' services set to Automatic	"Either can take over"	Both start after a power cut, both unlock the queue, and every recipient gets duplicates
Passive service set to Disabled rather than Manual	Safer	It cannot be started by the runbook without an extra step, which is the step that will be forgotten at 3 a.m.
Shared store on a share that only one node can reach	It works today, from the active node	The failover finds a store it cannot open. Test the passive node's access before you need it
Fencing skipped because "A is obviously down"	It stopped answering	Unreachable is not the same as dead. This is the one mistake that corrupts data rather than causing downtime

7.2 A rehearsal, end to end (wiki addition)

Do this once when you build the pair, and again after any change to either node. Below, A is 10.0.0.11 and B is 10.0.0.12.

# 0. Both nodes agree about the world.
foreach ($n in "10.0.0.11","10.0.0.12") {
    # Same build, same schema. Read it from whichever node is active.
    try { (Invoke-RestMethod "http://${n}:8080/metrics") -split "`n" | Select-String '^hmailserver_build_info' }
    catch { "$n is not serving metrics (expected on the passive node)" }
}

# 1. Baseline: send a message through the VIP and read it back.
# 2. Planned failover A -> B, following section 5.
# 3. Repeat the baseline. It must pass unchanged.
# 4. Planned failover B -> A.
# 5. Repeat the baseline a third time.

The test that matters most is step 5: a pair that fails over in one direction and not the other is the usual outcome of configuring one node by hand.

---

8. What hMailServer provides vs. what you provide

Provided by hMailServer	Provided by your infrastructure
/livez / /readyz / /healthz readiness gating	Virtual IP / load balancer + health check
Graceful shutdown drain (ShutdownDrainSeconds)	Shared database with its own HA
Shared-database + shared-store single-active design	Shared message storage (SAN/NAS/cloud)
/metrics for alerting on the active node	Fencing of a failed node

---

9. Verified against the code

Checked 13 August 2026. Every setting named on this page exists in IniFileSettings::LoadSettings with the default stated (MetricsServerPort 0, MetricsServerBindAddress 127.0.0.1, MetricsServerAuthToken / MetricsServerAuthUsername / MetricsServerAuthPassword / MetricsServerCertificateFile / MetricsServerPrivateKeyFile all empty, ShutdownDrainSeconds 0). The probe behaviour is MetricsServer::HandleClient_, which answers /livez, /readyz and /healthz before any branch that can refuse — a deliberate invariant recorded at that code, and the reason the VIP configuration here does not change when a credential is added. /metrics closing with 503 rather than 401 on a non-loopback bind with no credential is MetricsServer::Start plus BuildMetricsUnavailableResponse_; the loopback test is IsLoopbackAddress_, which accepts the whole of 127.0.0.0/8. The drain order — state to Stopping first, so /readyz is already 503, then, since 23 September 2026, the doors closed (IOService::StopAccepting, SMTPDeliveryManager::Pause) before the bounded wait, which now waits for the deliveries in flight as well as the sessions, then the listeners come down — is Application::StopServers. Maintenance mode is Application::EnterDrain / LeaveDrain behind RestApiServer::HandleServerDrain_, and StateDraining in ServerStatus; the draining reason is MetricsServer::IsReady_. The /healthz body is BuildHealthBody_. The bind-address parsing and the request/response deadlines are in MetricsServer::Start, ReadRequest_ and Send_.

There is regression coverage for the parts that would fail silently: test/RegressionTests/Infrastructure/HealthProbes.cs, test/RegressionTests/Infrastructure/MetricsSecurity.cs, and for the drain test/RegressionTests/Infrastructure/DrainAndUpgrade.cs beside ShutdownDrain.cs.

Wiki addition, checked 8 September 2026 against 6.2.28. The probe-lifecycle table in §3.1 comes from MetricsServer::IsReady_ and IsDatabaseAnswering_ (the three readiness conditions and their exact reason strings), the DatabaseProbeInterval (5 s), DatabaseProbeFailureInterval (30 s) and DatabaseProbeStaleness (20 s) constants at the top of MetricsServer.cpp, and Application::StartServers, which creates the metrics listener only after the state has been set to Running and before the REST API and web services — which is why start-up is a refused connection rather than a 503. The drain log lines quoted in §5 are Application::StopServers, which also shows the 200 ms poll and the metrics listener being stopped immediately after the drain window. The queue-unlock at start-up is PersistentMessage::UnlockAll, called from SMTPDeliveryManager::DoWork, and the uncoordinated selection it races is LoadPendingMessageList_; the fetch-account twin is PersistentFetchAccount::UnlockAll. The schema check that refuses a mismatched pair in both directions is REQUIRED_DB_VERSION in Constants.h, enforced by Application::OnDatabaseConnected. Further coverage: test/RegressionTests/Infrastructure/MetricsReadiness.cs and test/RegressionTests/Infrastructure/ShutdownDrain.cs.

Wiki addition, checked 15 September 2026. What sections 3, 4, 6 and 7 say about the [Settings] section on two nodes is IniSettingStore::Synchronize in v6.3.3 (the three-way merge, whose inisettingfilevalue is one shared row, and its second pass, which drops an agreeing row whose key is missing from the file) and in the tree at 3233b0cc4 (the stored value wins, IsFileOnlyName keeps PasswordPepper in the file, and IniFileSettings::LoadDatabaseSettings applies [SettingsOverride]).