High Availability Runbook

Source of truth: hmailserver/docs/HighAvailabilityRunbook.md, kept in the repository and copied here. This wiki copy adds the diagrams, the probe-lifecycle table, the load-balancer configurations and the worked commands (sections marked wiki addition); everything else is the repository document verbatim. Checked against 6.2.28 on 8 September 2026.

This runbook describes a supported, operationally simple active/passive high-availability topology for hMailServer. It deliberately contains no clustering code: hMailServer runs as a single active node at a time, and failover is performed by external infrastructure (a shared database, shared message storage, a floating virtual IP, and a health check). This keeps the mail server itself simple and avoids split-brain message corruption.

For an active/active or sharded design you would need a different storage and locking model; that is out of scope here.

1. Topology

                 ┌─────────────────────────────┐
   clients ─────►│  Virtual IP (VIP) / load     │
   (SMTP/IMAP/   │  balancer with health check  │
    POP/REST)    └──────────────┬───────────────┘
                                │  (routes to whichever node is "ready")
                ┌───────────────┴───────────────┐
                │                                │
        ┌───────▼────────┐              ┌────────▼───────┐
        │  Node A (active)│              │ Node B (passive)│
        │  hMailServer    │              │  hMailServer    │
        │  service RUNNING│              │  service STOPPED │
        └───────┬─────────┘              └────────┬────────┘
                │                                  │
                └──────────────┬───────────────────┘
                               │
              ┌──────────────────────────────────────┐
              │  Shared database (MSSQL/MySQL/PG)    │
              │  Shared message store ([Directories] │
              │  DataFolder)                         │
              └──────────────────────────────────────┘

Exactly one node runs the hMailServer service at any time. Both nodes are configured identically and point at the same database and the same message-store directory.

Wiki addition. The same picture with the control plane shown — which arrow decides where the traffic goes, and where the state that must not be written twice lives:

flowchart TB
    C["Clients: SMTP 25 and 587, IMAP 143 and 993, POP3 110 and 995, REST"]
    LB["VIP or load balancer"]
    C --> LB
    LB -->|"mail and API traffic"| A["Node A - service RUNNING"]
    LB -.->|"health check on /readyz"| A
    LB -.->|"health check on /readyz - refused, so never routed to"| B["Node B - service STOPPED"]
    A --> DB[("Shared database")]
    A --> FS[("Shared message store - DataFolder")]
    B -.->|"only once it becomes active"| DB
    B -.->|"only once it becomes active"| FS
    A -->|"/metrics"| M["Prometheus and alerting"]

The dotted arrows are the whole design. Node B is configured and ready but is not running a service, so its /readyz connection is refused, so the load balancer never routes to it — no clustering protocol, no quorum, no split-brain arbitration inside hMailServer.

Who owns what:

Layer	Owned by	Failure here looks like
Deciding which node is live	Your VIP or load balancer, driven by `/readyz`	Traffic to a node that cannot serve, or to both
Making sure only one node writes	You, through fencing	Duplicate deliveries and corrupted mailbox state
Reporting whether a node can serve	hMailServer's probes	A node kept in rotation during a database outage — which is what the readiness probe's staleness ceiling exists to prevent
Keeping the state consistent	The shared database and the shared store, on one failover boundary	Rows whose message files are missing

2. Shared components

2.1 Shared database

Use an external database server (Microsoft SQL Server, MySQL, or PostgreSQL), not the built-in SQL Compact / internal database, which is local-only.
Both nodes use the same hMailServer.INI [Database] connection settings.
The database server should itself be made highly available (e.g. SQL Server Always On / failover cluster, managed RDS/Cloud SQL with HA), or hosted on the same shared-storage layer as the message store.

2.2 Shared message store

hMailServer stores message files under the data directory, which is hMailServer.INI → [Directories] → DataFolder (there is no setting called DataDirectory; that is the name of the accessor in the code). Both nodes must see the same directory on shared storage (SAN/NAS/clustered file system, or a cloud file share). Note that this key lives in the per-node ini file, not in the shared database, so it is one of the values the "keep the INI in sync" step below is for. The database row for each message references its on-disk path; the database and the message store must therefore stay consistent with each other, so keep them on the same failover boundary.
Ensure both nodes' service accounts have identical read/write access to the share.

2.3 Virtual IP / load balancer

Clients connect to a floating VIP (or a load balancer / DNS name) rather than a node's real address.
The VIP must point at the node that is ready (see health checks below).
Only ever direct traffic to one node at a time. If you use a load balancer that can see both nodes, gate routing strictly on the readiness probe so the passive node (service stopped → probe fails/refused) never receives traffic.

3. Health checks and readiness gating

hMailServer exposes Kubernetes-style probes on the metrics listener. Enable it on both nodes:

[Settings]
MetricsServerPort=8080
MetricsServerBindAddress=0.0.0.0    ; reachable by the load balancer / health checker
MetricsServerAuthToken=<32+ random characters>   ; see below - required for /metrics on a non-loopback bind
ShutdownDrainSeconds=30             ; let in-flight sessions finish on a graceful stop

The three probes never require a credential, and the exposition now does. On a non-loopback bind, /metrics answers 503 until MetricsServerAuthToken (or the MetricsServerAuthUsername/MetricsServerAuthPassword pair) is set, and the 503 body names the settings that would open it. /livez, /readyz and /healthz are served unauthenticated in every configuration, because a load balancer cannot present a credential and a health check cannot hold a secret — so the VIP configuration in this runbook keeps working exactly as written, with or without the token.

The reason the exposition is treated differently: on 0.0.0.0 it publishes queue depth, session counts and authentication-failure counts to anything that can reach the port. Add the token, and give the scraper an Authorization: Bearer <token> header. If you want the metrics port encrypted as well, set MetricsServerCertificateFile and MetricsServerPrivateKeyFile; without them the listener stays plain HTTP and says so in the application log rather than refusing to start.

Two properties of this listener to design the health check around, both deliberate:

MetricsServerBindAddress takes an IP literal and nothing else. It is parsed with inet_pton, so 0.0.0.0, 127.0.0.1, :: and a specific IPv6 address work, while a host name or localhost is rejected — the listener logs MetricsServer: Invalid bind address and does not start, which takes the probes with it. :: serves both families (the listener clears IPV6_V6ONLY and says so in the log if it cannot); a specific IPv6 literal serves IPv6 only. If your health check gets a refused connection on a node whose service is running, this is the first thing to check.
It is a single accept loop serving one connection at a time. Probes are cheap and answered before anything that can refuse, but a scrape and a probe are still serialised behind each other. Keep the probe interval and its timeout comfortably apart (the listener bounds a request read at 5s and a response write at 15s), and do not point a sub-second health check at it.

Probes (HTTP):

Path	Meaning	Use for
`/livez`	Process is alive (200 whenever the listener is up).	Liveness restarts.
`/readyz`	200 only when the server is `Running` and the database is connected. Returns 503 while the server is stopping or draining, or if the database connection is lost. During startup it is not 503 but refused: this listener is brought up only after the state has already gone to `Running` (it is the first of the optional listeners started, ahead of the REST API and web services), so there is nothing listening until the server is ready. Both read as unhealthy to a load balancer, which is all that matters here.	VIP / load-balancer routing.
`/healthz`	JSON: `status` (`ok`/`unavailable`), `state`, `database` (`up`/`down`) and `uptime_seconds`. 200 when running with the database up, 503 otherwise. Per-protocol session counts are not in it — they are `hmailserver_sessions` on `/metrics`, behind its credential.	Dashboards / debugging.

Configure the VIP/load balancer health check against /readyz. Because the passive node's service is stopped, its /readyz connection is refused (unhealthy) and it will never be routed to. During a graceful stop, the active node flips /readyz to 503 before tearing down listeners, so the load balancer drains it cleanly.

3.1 What each probe says at every point in a node's life (wiki addition)

This is the table to design the health check's thresholds around. "Refused" means the TCP connection is refused because nothing is listening.

Node's condition	`/livez`	`/readyz`	`/healthz`	The load balancer's view
Service stopped (the passive node)	refused	refused	refused	Unhealthy. Never routed to
Service starting, before the state reaches `Running`	refused	refused	refused	Unhealthy. The listener is created only after the state is `Running`, so start-up is silence rather than a 503
Running, database answering	200 `alive`	200 `ready`	200, `"status":"ok"`	Healthy. Routed to
Running, database stopped answering	200 `alive`	503 `not ready: database did not answer the last readiness probe`	503, `"database":"down"`	Shed within roughly 5–20 s, depending on which of the three readiness conditions trips first
Running, database hanging rather than failing	200 `alive`	503 `not ready: database readiness probe has not completed recently`	503	Shed after 20 s. This is the case a naive probe misses entirely
Graceful stop, inside the drain window	200 `alive`	503 `not ready: server not in running state`	503, `"state":"stopping"`	Shed while existing sessions finish — the point of `ShutdownDrainSeconds`
Graceful stop, past the drain	refused	refused	refused	Unhealthy

/livez staying 200 through a database outage is deliberate: liveness restarts a process that is wedged, and killing a healthy mail server because its database blinked makes the outage worse.

3.2 Health-check configuration (wiki addition)

The probe is HTTP/1.0 and closes the connection, and the listener serves one connection at a time — so give the check a timeout of a second or two and an interval of five seconds or more, and never a sub-second check.

HAProxy (TCP mode for the mail ports, health-checked over HTTP on the metrics port):

backend mail_smtp
    mode tcp
    option httpchk GET /readyz
    http-check expect status 200
    default-server inter 5s fall 2 rise 2
    server nodeA 10.0.0.11:25 check port 8080
    server nodeB 10.0.0.12:25 check port 8080 backup

nginx stream (nginx Plus is required for active health checks; open-source nginx relies on max_fails/fail_timeout against the mail port itself):

stream {
    upstream mail_smtp {
        server 10.0.0.11:25 max_fails=2 fail_timeout=10s;
        server 10.0.0.12:25 backup;
    }
    server { listen 25; proxy_pass mail_smtp; }
}

AWS target group / Azure Load Balancer: protocol HTTP, port 8080, path /readyz, healthy threshold 2, unhealthy threshold 2, interval 5 s, timeout 2 s, success codes 200.

Kubernetes, if you run the service in a container:

livenessProbe:
  httpGet: { path: /livez, port: 8080 }
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 2

Add scheme: HTTPS when MetricsServerCertificateFile and MetricsServerPrivateKeyFile are set — TLS applies to the whole port, probes included.

Verify the whole gate before you rely on it:

# On the ACTIVE node - all three must answer, with no credential.
Invoke-RestMethod http://10.0.0.11:8080/livez
Invoke-RestMethod http://10.0.0.11:8080/readyz
Invoke-RestMethod http://10.0.0.11:8080/healthz

# On the PASSIVE node - this must be REFUSED, not 200.
Test-NetConnection 10.0.0.12 -Port 8080

# From the load balancer's own vantage point, watch readiness flip during a stop.
while ($true) {
    $t = Get-Date -Format "HH:mm:ss"
    try   { "$t $((Invoke-WebRequest http://10.0.0.11:8080/readyz -UseBasicParsing -TimeoutSec 2).StatusCode)" }
    catch { "$t $($_.Exception.Response.StatusCode.value__) $($_.ErrorDetails.Message)" }
    Start-Sleep -Seconds 2
}

4. Normal operation

Active node (A): hMailServer service set to Automatic and running.
Passive node (B): hMailServer service set to Manual (or Disabled) and stopped. Keep the binaries and hMailServer.INI in sync with A (same version, same settings, same database + data directory).
Apply configuration changes on the active node; because configuration lives in the shared database, B picks them up automatically when it becomes active. Keep the hMailServer.INI (which holds the database connection + local settings) in sync manually or via your configuration-management tooling.

5. Planned failover (maintenance)

Drain A: stop the hMailServer service on A. With ShutdownDrainSeconds set, A first reports /readyz = 503 (the load balancer stops sending new connections) and waits up to the drain window for active SMTP/IMAP/POP sessions to finish before shutting down.
Verify A is down: /readyz on A is refused.
Move the VIP to B (or let the load balancer health check do it).
Start B: start the hMailServer service on B. Wait until /readyz on B returns 200.
Confirm: send a test message through the VIP and confirm delivery; check /healthz reports database: up and state: running.

Reverse the steps to fail back.

Wiki addition. The same five steps as a timeline, with what each participant sees:

sequenceDiagram
    autonumber
    participant OP as Operator
    participant LB as Load balancer
    participant A as Node A - active
    participant B as Node B - passive
    participant DB as Shared database and store
    Note over A: state Running, /readyz 200
    OP->>A: Stop-Service hMailServer
    A->>A: state goes to Stopping
    LB->>A: GET /readyz
    A-->>LB: 503 not ready - server not in running state
    LB->>LB: mark A unhealthy, stop sending new connections
    Note over A: drain window - up to ShutdownDrainSeconds
    A->>A: existing SMTP, IMAP and POP3 sessions finish
    A->>A: metrics listener stops, then the mail listeners
    A->>DB: releases every lock as the service exits
    OP->>A: confirm /readyz is now REFUSED
    OP->>B: Start-Service hMailServer
    B->>DB: connect, check schema version, unlock the queue
    B->>B: state goes to Running, listeners up
    LB->>B: GET /readyz
    B-->>LB: 200 ready
    LB->>LB: mark B healthy, route traffic
    OP->>B: send a test message through the VIP, read it back

Step 12 — unlock the queue — is the one to understand: at start-up the server sets messagelocked = 0 on every queued message, because locks held by a node that crashed would otherwise strand that mail for ever. It is also exactly why two nodes must never run at once. See Warm Standby for what happens if they do.

The commands, in order:

# 1-3. On A: drain and stop. The service does not return until the drain has run.
Stop-Service hMailServer
Get-Content "C:\Program Files\hMailServer\Logs\hmailserver_$(Get-Date -Format yyyy-MM-dd).log" -Tail 20 |
  Select-String "Graceful shutdown"

# 4. Prove A is down from the load balancer's point of view.
Test-NetConnection 10.0.0.11 -Port 8080     # TcpTestSucceeded must be False

# 5. On B: start, then wait for readiness rather than assuming it.
Start-Service hMailServer
do {
    Start-Sleep -Seconds 2
    $ready = $false
    try { $ready = (Invoke-WebRequest http://10.0.0.12:8080/readyz -UseBasicParsing -TimeoutSec 2).StatusCode -eq 200 } catch {}
} until ($ready)
Invoke-RestMethod http://10.0.0.12:8080/healthz

The drain lines to look for in the application log are Graceful shutdown: draining N active session(s), up to Ns... followed by either Graceful shutdown: all sessions drained. or Graceful shutdown: drain window elapsed with N session(s) still active; stopping anyway. With ShutdownDrainSeconds=0 (the default) none of them appear, because there is no wait at all.

6. Unplanned failover (node A fails)

The load balancer health check against /readyz on A fails; stop routing to A (automatic if the LB is health-check driven).
Fence A to guarantee it cannot still be writing to the shared store: power it off / isolate it from the storage and database network. This prevents split-brain (two active nodes writing the same message store).
Move the VIP to B.
Start the hMailServer service on B and wait for /readyz = 200.
When A is repaired, bring it back as the new passive node (service stopped) before any future failover.

Split-brain safety: never let both nodes run the service against the shared store at the same time. Always fence the failed node before starting the standby. The shared database is the source of truth for message metadata; two active writers can corrupt mailbox state.

Wiki addition. The decision, with the fencing step where it has to be — before anything is started, not after:

flowchart TD
    A["Health check on A fails"] --> B{"Is A definitely dead?"}
    B -->|"powered off, or you can see it is"| D["Fence complete"]
    B -->|"unreachable, but you cannot prove it"| C["FENCE IT: power off the VM, pull the switch port, revoke its storage and database access"]
    C --> D
    B -->|"still running, database or store unreachable to it"| C
    D --> E["Move the VIP to B"]
    E --> F["Start the service on B"]
    F --> G{"/readyz on B is 200?"}
    G -->|"no"| G1["Read ERROR_hmailserver on B: schema mismatch, database credentials, or DataFolder not mounted"]
    G -->|"yes"| H["Send a test message through the VIP and read it back"]
    H --> I["When A is repaired, bring it back STOPPED, as the new passive node"]

Why the fence is not optional, in this product specifically. If A is alive but partitioned from you rather than from the storage, starting B gives two servers polling the same hm_messages table. Both unlock the whole queue at start-up, both select where messagelocked = 0 and messagenexttrytime <= now, and both take the lock after selecting — so they race that window and deliver the same messages. Every recipient gets duplicates and nothing anywhere looks wrong. The same shape applies to external fetch accounts, where a POP3 account collected twice with delete-after-download splits the mail arbitrarily between two stores. Warm Standby documents both mechanisms.

Recovery-time expectations, so the runbook can be held to a number:

Step	Typically
Health check declares A unhealthy	2 intervals — 10 s at the settings above
Fencing	However long your platform takes. This is usually the whole recovery time
VIP move	Seconds, or a DNS TTL if you use DNS instead of a VIP
B starts and reaches `/readyz` = 200	Seconds to a minute: connect to the database, check the schema, unlock the queue, bind the listeners
Mail queued at sending servers during the gap	Delivered on their retry. Sending servers retry for days; nothing is lost, it is delayed

7. Validation checklist

Both nodes use the same external database and the same [Directories] DataFolder on shared storage.
MetricsServerPort is enabled and reachable by the health checker on both nodes; the VIP health check targets /readyz.
MetricsServerAuthToken is set on both nodes, and the scraper sends it. Without it /metrics answers 503 on a non-loopback bind — the probes and therefore the failover still work, so this fails quietly as missing dashboards, not as an outage.
ShutdownDrainSeconds is set so planned failovers drain gracefully.
Passive node's service is stopped and set to Manual/Disabled.
A documented fencing step exists for unplanned failover.
A test message sent through the VIP is delivered after a planned failover in both directions.

7.1 The seven ways this is got wrong (wiki addition)

Each row is a real configuration that looks correct and is not.

What was done	Why it looks fine	What actually happens
`MetricsServerBindAddress=mail.example.com`	It is the node's own name	The value is parsed with `inet_pton` and must be an IP literal. The listener logs `MetricsServer: Invalid bind address` and does not start, so every probe is refused and the load balancer takes a healthy node out of rotation
Health check pointed at port 25 instead of `/readyz`	A TCP connect succeeds, so the node "is up"	A node whose database has gone away still accepts TCP on 25 and then rejects or defers every message. Only `/readyz` knows the difference
Health check interval of 1 s with a 1 s timeout	Faster detection	The listener serves one connection at a time; a scrape and a probe serialise. Use 5 s and 2 s
Both nodes' services set to Automatic	"Either can take over"	Both start after a power cut, both unlock the queue, and every recipient gets duplicates
Passive service set to Disabled rather than Manual	Safer	It cannot be started by the runbook without an extra step, which is the step that will be forgotten at 3 a.m.
Shared store on a share that only one node can reach	It works today, from the active node	The failover finds a store it cannot open. Test the passive node's access before you need it
Fencing skipped because "A is obviously down"	It stopped answering	Unreachable is not the same as dead. This is the one mistake that corrupts data rather than causing downtime

7.2 A rehearsal, end to end (wiki addition)

Do this once when you build the pair, and again after any change to either node. Below, A is 10.0.0.11 and B is 10.0.0.12.

# 0. Both nodes agree about the world.
foreach ($n in "10.0.0.11","10.0.0.12") {
    # Same build, same schema. Read it from whichever node is active.
    try { (Invoke-RestMethod "http://${n}:8080/metrics") -split "`n" | Select-String '^hmailserver_build_info' }
    catch { "$n is not serving metrics (expected on the passive node)" }
}

# 1. Baseline: send a message through the VIP and read it back.
# 2. Planned failover A -> B, following section 5.
# 3. Repeat the baseline. It must pass unchanged.
# 4. Planned failover B -> A.
# 5. Repeat the baseline a third time.

The test that matters most is step 5: a pair that fails over in one direction and not the other is the usual outcome of configuring one node by hand.

8. What hMailServer provides vs. what you provide

Provided by hMailServer	Provided by your infrastructure
`/livez` / `/readyz` / `/healthz` readiness gating	Virtual IP / load balancer + health check
Graceful shutdown drain (`ShutdownDrainSeconds`)	Shared database with its own HA
Shared-database + shared-store single-active design	Shared message storage (SAN/NAS/cloud)
`/metrics` for alerting on the active node	Fencing of a failed node

9. Verified against the code

Checked 13 August 2026. Every setting named on this page exists in IniFileSettings::LoadSettings with the default stated (MetricsServerPort 0, MetricsServerBindAddress 127.0.0.1, MetricsServerAuthToken / MetricsServerAuthUsername / MetricsServerAuthPassword / MetricsServerCertificateFile / MetricsServerPrivateKeyFile all empty, ShutdownDrainSeconds 0). The probe behaviour is MetricsServer::HandleClient_, which answers /livez, /readyz and /healthz before any branch that can refuse — a deliberate invariant recorded at that code, and the reason the VIP configuration here does not change when a credential is added. /metrics closing with 503 rather than 401 on a non-loopback bind with no credential is MetricsServer::Start plus BuildMetricsUnavailableResponse_; the loopback test is IsLoopbackAddress_, which accepts the whole of 127.0.0.0/8. The drain order — state to Stopping first, so /readyz is already 503, then the bounded wait, then the listeners come down — is Application::StopServers. The /healthz body is BuildHealthBody_. The bind-address parsing and the request/response deadlines are in MetricsServer::Start, ReadRequest_ and Send_.

There is regression coverage for the parts that would fail silently: test/RegressionTests/Infrastructure/HealthProbes.cs and test/RegressionTests/Infrastructure/MetricsSecurity.cs.

Wiki addition, checked 8 September 2026 against 6.2.28. The probe-lifecycle table in §3.1 comes from MetricsServer::IsReady_ and IsDatabaseAnswering_ (the three readiness conditions and their exact reason strings), the DatabaseProbeInterval (5 s), DatabaseProbeFailureInterval (30 s) and DatabaseProbeStaleness (20 s) constants at the top of MetricsServer.cpp, and Application::StartServers, which creates the metrics listener only after the state has been set to Running and before the REST API and web services — which is why start-up is a refused connection rather than a 503. The drain log lines quoted in §5 are Application::StopServers, which also shows the 200 ms poll and the metrics listener being stopped immediately after the drain window. The queue-unlock at start-up is PersistentMessage::UnlockAll, called from SMTPDeliveryManager::DoWork, and the uncoordinated selection it races is LoadPendingMessageList_; the fetch-account twin is PersistentFetchAccount::UnlockAll. The schema check that refuses a mismatched pair in both directions is REQUIRED_DB_VERSION in Constants.h, enforced by Application::OnDatabaseConnected. Further coverage: test/RegressionTests/Infrastructure/MetricsReadiness.cs and test/RegressionTests/Infrastructure/ShutdownDrain.cs.

hMailServer 6.3.2 · AGPL-3.0-or-later · Repository · Report a documentation error

Hmail Server — full index

Start here

1. Install and run

2. Secure it

3. Operate it

4. Extend it

5. Contribute to it

Look it up — from any journey

Uh oh!

High Availability Runbook

High Availability Runbook

1. Topology

2. Shared components

2.1 Shared database

2.2 Shared message store

2.3 Virtual IP / load balancer

3. Health checks and readiness gating

3.1 What each probe says at every point in a node's life (wiki addition)

3.2 Health-check configuration (wiki addition)

4. Normal operation

5. Planned failover (maintenance)

6. Unplanned failover (node A fails)

7. Validation checklist

7.1 The seven ways this is got wrong (wiki addition)

7.2 A rehearsal, end to end (wiki addition)

8. What hMailServer provides vs. what you provide

9. Verified against the code

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally