Monitoring and Health

Monitoring and Health

This page began as a chapter of the 6.2.10 manual and has been corrected and extended for 6.2.28. The Control Panel's pages are grouped differently now, so the paths below use today's groups; the TLS ports 465, 993 and 995 exist only after you create them on the TCP/IP ports page (a fresh install seeds 25, 587, 110 and 143); and everything added since 6.2.10 is in Changes-Since-6210. Where a value here disagrees with the Settings Reference, which is generated from the code, the reference is right.

16.0 If you do only three things

1. Point a health check at /readyz. It is the one signal that distinguishes "the process is up" from "this server can actually accept and deliver mail", because it is proved by a real database round trip rather than by a flag.
2. Alert on the delivery queue's oldest message, not its depth. A deep queue draining normally is a busy server; a shallow queue whose oldest message is four hours old is a destination nobody is getting mail to.
3. Alert on certificate expiry. hmailserver_tls_certificate_expiry_seconds minus time() is how many seconds of TLS you have left, and expiry is the failure that takes every protocol down at once, on a schedule, silently.

Everything else on this page is detail behind those three.

16.1 The live view

The Control Panel's Monitoring & troubleshooting group is the human-facing half of everything below. Every page here reads the running service over COM, so it shows what the server thinks right now, not what a log said earlier.

Page	What it shows	Reach for it when
Dashboard (top level)	Uptime, messages processed, in-queue count with a Backlog badge, spam blocked, viruses removed, plus throughput and session charts. The live view refreshes every 2 seconds; the 24-hour, 7-day and 30-day ranges read the stored metric history (§16.6)	You want one screen that says whether today is normal
Server status	Version and architecture, state, start time, uptime, database type/host/name/schema version, the five statistics, the update check's verdict, and a computed Warnings list	First look after any change, and after every restart
Delivery queue	Every message waiting to go out: id, created, from, recipients, next try time, tries, file. Buttons: View source, Deliver now, Remove, Refresh	Mail is not arriving at the far end
Diagnosing stalled mail	The built-in version of Diagnosing Stalled Mail, with a one-click debug-logging toggle	Mail is accepted and then does not move
Message trace	What happened to a message — accepted, delivered, failed, quarantined — searchable by address or by queue id. Empty and inert unless MessageTraceEnabled=1	"Did we ever receive it, and what did we do with it?"
Live logs	A real-time, colour-coded tail of what the server is writing	You are reproducing a problem right now
Diagnostics	The built-in self-test (§16.8)	Anything is broken and you do not yet know what
MX query	An MX lookup through the server's own resolver	You suspect DNS rather than mail

Server status. The Warnings card is computed in the Control Panel from what the server reports, and it is the part worth reading first: a missing HostName, an IP range that allows remote-to-remote delivery without requiring authentication (an open relay), a 127.0.0.1 range that has been auto-banned, and the count of currently expiring auto-ban ranges.

The delivery queue. NextTry and Tries are the two columns that matter: a message whose try count is climbing is being deferred by the far end, and one whose next try is an hour away is waiting out Minutes between retries. Both of those are normal. A queue in which every message shares one destination domain is not.

16.2 Logs on disk

Logs live in the Logs directory (C:\Program Files\hMailServer\Logs by default). Monitoring & troubleshooting → Logging controls what is written.

hMailServer writes several different files, and knowing which one to open saves most of the time you would otherwise spend on a problem:

File	Holds	Rotates
hmailserver_<date>.log	Everything: application lines, and the protocol conversations unless they are split out	Daily, by name
ERROR_hmailserver_<date>.log	Only entries the server itself classed as an error. Read this first. Its mere existence is a diagnostic — the built-in self-test fails when any ERROR* file is present	Daily, by name
hmailserver_SMTP_<date>.log, hmailserver_IMAP_<date>.log, hmailserver_POP3_<date>.log	The per-protocol conversations, when SepSvcLogs=1	Daily, by name
hmailserver_backup.log	Every backup and restore run, line by line — see Backup and Restore	Never. One growing file; LogDeleteDays matches it by name but only deletes a file nothing has written to for that many days
hmailserver_events.log	Event-script output	Never, as above
hmailserver_awstats.log	The AWStats-format feed, when AWStats logging is on	Never, as above
hMailServer_messagestore_consistency.report	The most recent message-store consistency scan: one line per message whose file is missing, as messageid<TAB>account<TAB>expected-path	Overwritten each run

The settings that govern all of this:

Setting	Default	What it does	What it looks like when it is wrong
LogLevel	9	Bit mask of which log types are written	Set too low and the log you are grepping is simply not being produced
Debug messages (Logging page)	off	Adds the LOG_DEBUG lines, including the accept-pipeline stage timings	Left on: a busy server fills the disk in days, and a full disk stops mail (§16.9)
SepSvcLogs	0	Splits SMTP, IMAP and POP3 into their own files	Off, so a search for one protocol wades through all three
LogDeleteDays	0	Deletes files named hmailserver_*.log or ERROR_hmailserver_*.log whose last write is older than this many days, and nothing else in the directory. 0 keeps everything for ever	Left at 0 on a server with debug logging on — the usual cause of a full log volume
JsonLogging	0	Writes JSON lines, for ingestion into Elasticsearch, Loki or Splunk	Overridden while Logging.LogFormat is NCSA: your pipeline gets NCSA lines and no error
SlowQueryLogMilliseconds	0 (off)	Logs, and counts in hmailserver_db_slow_queries_total, any database statement slower than this	Off, so a database that has become slow leaves no trace but the latency histogram
WindowsEventLogLevel	2	How much also goes to the Windows event log	—

Turn debug logging on to diagnose, and off again afterwards. The stage-timing lines Diagnosing Stalled Mail depends on are debug lines; the slow ones (ten seconds or more) are written at application level as well, so a server with only application logging still records the important half.

16.3 The metrics listener

One small HTTP listener serves both the Prometheus exposition and three Kubernetes-style health probes. It is off until you give it a port.

[Settings]
MetricsServerPort=9100
MetricsServerBindAddress=127.0.0.1

Configure it under Monitoring & troubleshooting → API & monitoring, or in hMailServer.INI directly. The service reads these at start, so restart it.

Four rules decide what the listener will serve — the port, the bind address, the credential and the TLS pair — and they are worth understanding before you expose it, because three of them fail silently by design. The credential has two spellings, so it takes two rows:

Setting	Default	Effect	What it looks like when it is wrong
MetricsServerPort	0	0 means the listener is never created	No port, no probes, no metrics — and no complaint anywhere
MetricsServerBindAddress	127.0.0.1	An IP literal only, parsed with inet_pton. 0.0.0.0, 127.0.0.1, :: and a specific IPv6 address all work	A host name or localhost is rejected: the log says MetricsServer: Invalid bind address, the listener does not start, and the probes go with it. A refused health check against a running service is nearly always this
MetricsServerAuthToken	empty	Sent by the scraper as Authorization: Bearer <token>	Absent on a non-loopback bind: /metrics answers 503, not 401, and the body names the settings that would open it. The probes keep answering, so failover keeps working and this shows up as missing dashboards rather than as an outage
MetricsServerAuthUsername + MetricsServerAuthPassword	empty	HTTP Basic. When both schemes are configured, either one satisfies the check	As above
MetricsServerCertificateFile + MetricsServerPrivateKeyFile	empty	Both set makes the whole port HTTPS, probes included	Set but unusable (missing file, mismatched key): the listener still starts and still serves the probes in the clear, and /metrics answers 503. It never downgrades the exposition to plaintext

:: serves both address families (the listener clears IPV6_V6ONLY); a specific IPv6 literal serves IPv6 only. "Loopback" means the whole of 127.0.0.0/8 and ::1, not just 127.0.0.1.

This is the whole dispatch, drawn from the request handler:

flowchart TD
    A["TCP connection accepted"] --> B{"TLS configured?"}
    B -->|"yes"| C["TLS handshake"]
    B -->|"no"| D["Read request header block"]
    C -->|"fails"| X["Close, no reply. Never falls back to HTTP"]
    C -->|"succeeds"| D
    D --> E{"Request path"}
    E -->|"/livez"| L["200 alive"]
    E -->|"/readyz"| R{"Running AND database answering?"}
    E -->|"/healthz"| H["JSON body, 200 or 503"]
    E -->|"/metrics"| M{"May metrics be served in this configuration?"}
    E -->|"anything else"| N["404"]
    R -->|"yes"| R1["200 ready"]
    R -->|"no"| R2["503 not ready, plus the reason"]
    M -->|"non-loopback bind, no credential"| M1["503 metrics unavailable"]
    M -->|"TLS asked for, could not be prepared"| M1
    M -->|"available"| M2{"Credential correct, or none required?"}
    M2 -->|"no"| M3["401, and the refusal counter increments"]
    M2 -->|"yes"| M4["200, the exposition"]

Loading

The three probes are answered before any branch that can refuse. That ordering is deliberate and load-bearing: a Kubernetes httpGet probe, an ELB or ALB health check and a Docker HEALTHCHECK have nowhere to keep a bearer token, and an authenticated probe would report a healthy server as failing and get it killed or fenced out of a cluster.

Nothing on the request path touches the database or the disk. A second thread — the refresher — owns every database read and every certificate read, publishes the results into a small cache, and the request thread copies out of that cache under a mutex that is never held across I/O:

sequenceDiagram
    participant S as Scraper
    participant W as Listener thread
    participant C as Cache
    participant R as Refresher thread
    participant DB as Database
    R->>DB: select * from hm_dbversion
    DB-->>R: one row
    R->>C: publish probe result and success tick
    S->>W: GET /metrics
    W->>C: read counters, queue figures, certificate series
    W->>C: set the flag asking for a queue refresh
    C-->>W: values
    W-->>S: 200, the exposition
    R->>C: is a queue refresh wanted, and is it due?
    R->>DB: count, and min of messagecreatetime, over hm_messages
    DB-->>R: depth and oldest
    R->>C: publish depth and oldest age

Loading

The refresher's cadences, all fixed in the code: the database probe every 5 s, backing off to 30 s after a probe that failed; the delivery-queue aggregates at most every 10 s, and only when a scrape has asked for them since the last refresh; the certificate files every 5 minutes. Reading /metrics therefore costs no database work at all, and an installation with the port open and nothing scraping it pays nothing for the two queue queries.

Two more properties to design a monitoring system around:

• It is a single accept loop serving one connection at a time. Probes are cheap and are answered first, but a scrape and a probe still serialise behind each other. Keep the probe interval and its timeout comfortably apart — a request's header read is bounded at 5 s and a response write at 15 s — and do not point a sub-second health check at it.
• Monitoring traffic is never counted in the mail TLS metrics. A failed handshake on this port, a port scan, a plain-HTTP request to an HTTPS port: none of it moves hmailserver_tls_handshake_failures_total, which describes TLS on the mail protocols and is what interoperability alerts are built on. A scraper that cannot handshake is already reported by Prometheus's own synthetic up == 0.

Refused credentials are counted, not logged: one line per attempt would fill the application log with something nobody can act on, so hmailserver_metrics_unauthorized_requests_total turns "somebody is probing the monitoring port" into a rate you can alert on. Nothing about what was presented is recorded anywhere.

16.4 Every metric the listener exposes

Thirty metric families, in the order they appear in the response. Counters are monotonic since server start — a restart resets them, which is what the start timestamp is for.

Mail flow

Metric	Type	Meaning, and the trap in it
hmailserver_processed_messages_total	counter	One per delivery pass started, not one per message received. A message deferred three times and then delivered counts four. Rate it; do not read it as inbound volume
hmailserver_messages_delivered_total	counter	Delivery passes that completed with no failed recipient
hmailserver_messages_deferred_total	counter	Passes rescheduled for a later attempt: a temporary failure at the far end, greylisting, or a message held back because the virus scanner could not examine it (AVFailAction=1)
hmailserver_messages_bounced_total	counter	Delivery-failure notifications actually queued to the sender. One NDR can cover several failed recipients, so this is not a recipient count
hmailserver_delivery_queue_messages	gauge	Rows in hm_messages in the delivering state. Refreshed at most every 10 s
hmailserver_delivery_queue_oldest_message_age_seconds	gauge	Age of the oldest queued message; 0 when the queue is empty or the timestamp could not be read. The single most useful mail-flow signal on this page
hmailserver_spam_messages_total	counter	Messages classified as spam, on the SMTP accept path and on external fetch
hmailserver_viruses_removed_total	counter	Viruses removed during delivery preprocessing
hmailserver_domain_messages_received_total{domain}	counter	Per locally hosted domain, counted once per message per domain rather than per recipient. Only present when MetricsPerDomainEnabled=1
hmailserver_domain_messages_sent_total{domain}	counter	As above, for mail accepted from each hosted domain

Only domains this server hosts are ever labelled, so no amount of inbound mail from strangers can mint time series. The pair is still opt-in, because an installation hosting several thousand domains should choose to pay for the cardinality rather than discover it.

Sessions, authentication and TLS

Metric	Type	Meaning, and the trap in it
hmailserver_sessions{protocol}	gauge	Concurrent sessions; protocol is smtp, imap or pop3
hmailserver_auth_success_total	counter	Successful mailbox authentications
hmailserver_auth_failures_total	counter	Failed ones. A rising rate against a flat success rate is a password-guessing run; pair it with Access & abuse protection → Auto-ban
hmailserver_tls_handshakes_total	counter	Completed TLS handshakes on the mail protocols
hmailserver_tls_handshake_failures_total	counter	Failed ones. A step change after a certificate replacement means a client population that cannot chain to the new issuer
hmailserver_tls_certificate_expiry_seconds{certificate}	gauge	notAfter as a Unix timestamp, per configured certificate, re-read every 5 minutes. Where two certificates share a name the earliest expiry wins; at most 128 series, soonest first. Absent entirely when no certificate is configured, so alert on the value, and only alert on absent() if you know you have one
hmailserver_metrics_unauthorized_requests_total	counter	Requests to /metrics refused for a missing or wrong credential. Always 0 when no credential is configured, so any value above zero is worth reading

When no certificate row exists, the server falls back to the ACME certificate at <data directory>\ACME\fullchain.pem and labels it ACME (automatic). That is exactly the install shape where a silent renewal failure is the thing worth alerting on.

The server itself

Metric	Type	Meaning, and the trap in it
hmailserver_start_time_seconds	gauge	Unix timestamp at which the listener started. Uptime is time() - hmailserver_start_time_seconds; restarts are changes(hmailserver_start_time_seconds[1h])
hmailserver_build_info{version,architecture,database_schema_version}	gauge	Always 1; the information is in the labels. The schema version is the compiled-in requirement, which the server refuses to start without matching, so it is the running schema by construction
hmailserver_state{state}	gauge	One series per state — unknown, stopped, starting, running, stopping — exactly one of which is 1. Alert on hmailserver_state{state="running"} == 0 rather than on a magic number
hmailserver_workqueue_depth{queue}	gauge	Tasks accepted but not yet started, per queue
hmailserver_workqueue_blocking_tasks_waiting{queue}	gauge	Tasks marked as possibly blocking that are waiting for a slot. These hold no worker thread. A number that stays above zero means the AsyncQueueReservedThreads cap is doing its job and something those tasks depend on is slow
hmailserver_messagestore_missing_files	gauge	Messages whose backing file was missing at the last consistency scan. Always 0 unless MessageStoreConsistencyCheck=1, which is off by default
hmailserver_command_processing_seconds	histogram	Time to handle one protocol command line. Bucket bounds, in seconds: 0.0001, 0.00025, 0.0005, 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.5, 1.0

The queue label takes one of seven values, all created at startup: Server queue, Maintenance queue, Asynchronous task queue, Name lookup queue, IOCPQueue, SMTP delivery queue, External fetch queue. Cardinality is therefore fixed. The one to watch is Asynchronous task queue — it finalises received messages and sends the final 250, so depth there is mail waiting to be acknowledged.

The database

Metric	Type	Meaning, and the trap in it
hmailserver_database_connected	gauge	1 when the database answered a real round trip recently. It used to mean "the pool holds connection objects", which stayed true throughout an outage; an alert on == 0 could not fire. It can now
hmailserver_database_probe_success_timestamp_seconds	gauge	When the database last answered. The conventional Prometheus form, and what an alert expression wants
hmailserver_database_probe_age_seconds	gauge	The same fact measured on the server's own monotonic clock, so it survives the scraper and the mail server disagreeing about the time. /readyz reports 503 once this passes 20 s
hmailserver_db_connections{state}	gauge	Pool occupancy: busy and available. available pinned at zero with busy at the pool size is saturation, and every mail path is then waiting out DBConnectionAcquireTimeout
hmailserver_db_query_seconds	histogram	Statement execution time, all backends. Bucket bounds, in seconds: 0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 5.0, 10.0
hmailserver_db_slow_queries_total	counter	Statements slower than SlowQueryLogMilliseconds. 0 while that setting is 0

Both histograms emit the full _bucket, _sum and _count set, so histogram_quantile works from the first scrape whether or not any traffic has arrived yet. They were once declared as summaries carrying only _sum and _count, from which the only derivable figure was a mean — and a mean latency hides exactly the tail you care about. _sum and _count keep their old names and meaning, so anything already computing a mean still works.

16.5 Health probes

Three paths, no credential in any configuration, HTTP/1.0 responses that close the connection.

Path	200 when	503 when	Body	Use it for
/livez	Always, if the listener answered at all	Never	alive	Liveness restarts. It checks nothing on purpose: a database outage must not get a healthy process killed
/readyz	State is Running and the database has proved within the last 20 s that it can answer	Otherwise	ready, or not ready: <reason>	Load-balancer and VIP routing
/healthz	Same conditions as /readyz	Same	{"status":"ok|unavailable","state":"...","database":"up|down","uptime_seconds":N}	Dashboards and human debugging

Readiness is the server state plus three conditions on the database, and each of those three exists because of a way the previous answer was wrong:

flowchart TD
    A["GET /readyz"] --> B{"Server state is Running?"}
    B -->|"no"| B1["503 - not ready: server not in running state"]
    B -->|"yes"| C{"Connection pool holds connections?"}
    C -->|"no"| C1["503 - not ready: database not connected"]
    C -->|"yes"| D{"Did the last probe succeed?"}
    D -->|"no"| D1["503 - not ready: database did not answer the last readiness probe"]
    D -->|"yes"| E{"Was that success within 20 seconds?"}
    E -->|"no"| E1["503 - not ready: database readiness probe has not completed recently"]
    E -->|"yes"| F["200 - ready"]

Loading

The third condition is the one that is easy to leave out and expensive to omit. A probe against a database that has stopped answering does not necessarily fail — it can simply not come back, for as long as DBConnectionAcquireTimeout allows (60 s by default). Without a staleness ceiling, /readyz would keep a node in a load balancer's rotation for a full minute after the database went away, and every message routed to it in that minute is rejected or deferred. Twenty seconds is four probe intervals, so a single slow round trip cannot trip it.

Start-up is a refused connection, not a 503. The metrics listener is brought up only after the server state has already gone to Running — it is the first of the optional listeners, ahead of the REST API and the web services — so there is nothing listening until the server is ready. Shutdown is a 503 first, then a refusal: the state goes to Stopping before anything is torn down, so /readyz reports 503 while ShutdownDrainSeconds runs and in-flight sessions finish, and only then does the listener come down. Both read as unhealthy to a load balancer, which is all that matters — see High Availability Runbook.

Checking them by hand:

# Liveness and health, from the server itself
Invoke-RestMethod http://127.0.0.1:9100/livez
Invoke-RestMethod http://127.0.0.1:9100/healthz | ConvertTo-Json

# Readiness, keeping the status code when it is 503 (Invoke-RestMethod throws on 5xx)
try   { (Invoke-WebRequest http://127.0.0.1:9100/readyz -UseBasicParsing).Content }
catch { $_.Exception.Response.StatusCode.value__; $_.ErrorDetails.Message }

A healthy /healthz looks like this:

{"status":"ok","state":"running","database":"up","uptime_seconds":864317}

and an unhealthy one like this, with 503 as the status code:

{"status":"unavailable","state":"running","database":"down","uptime_seconds":41}

Session counts are deliberately not in that body. /healthz is open and unauthenticated in every configuration, including the one where /metrics is closed precisely so that session counts are not readable from the network; publishing them here would have handed out what the other rule refuses. They are hmailserver_sessions on /metrics, behind whatever protection that has.

16.6 Metric history inside the server

The server keeps its own short history, so the Control Panel's dashboard can show 24 hours, 7 days and 30 days without a Prometheus anywhere.

Setting	Default	Effect
MetricsHistoryDays	7	Days of history kept in hm_metricsamples. 0 switches the feature off, and the dashboard says so rather than showing an empty chart

One row per metric per minute, plus a sample at service start so the history shows a restart as the drop in every counter that it is. Pruning runs on the first pass and then hourly. Fourteen metrics are stored — the names are the exposition's without the hmailserver_ prefix, with the label folded into the name:

sessions_smtp, sessions_imap, sessions_pop3, processed_messages_total, messages_delivered_total, messages_deferred_total, messages_bounced_total, spam_messages_total, viruses_removed_total, auth_success_total, auth_failures_total, tls_handshakes_total, tls_handshake_failures_total, messagestore_missing_files.

Read it over COM as JSON — here, the last 24 hours in 15-minute buckets (0 for the bucket size returns every sample):

$app = New-Object -ComObject hMailServer.Application
$app.Authenticate("Administrator", "<password>") | Out-Null
$app.Utilities.GetMetricHistory("processed_messages_total", 1440, 15)

This is not a replacement for Prometheus — one row a minute, no labels, no histograms — but it survives a restart, needs nothing installed, and answers "was it like this yesterday?" on a server nobody has instrumented.

16.7 Pushing to an OpenTelemetry collector

If you would rather push than be scraped, three OTLP/HTTP signals can be configured independently. Each is a no-op until its own endpoint is set.

Setting	Default	Effect
OtelMetricsEndpoint	empty	OTLP metrics endpoint
OtelMetricsInterval	60	Seconds between exports, clamped to 5–3600
OtelEndpoint	empty	OTLP traces only — one span per delivery attempt, with milestone events and the terminal outcome, joined to the trace named by the message's own traceparent
OtelLogsEndpoint	empty	OTLP logs

The pushed metric names are the Prometheus names verbatim, so a dashboard built against a scrape works against the push and the other way round. Twelve counters, sessions as one gauge with a protocol attribute, hmailserver_messagestore_missing_files, and the per-domain counters when they are enabled. The histograms and the database and queue gauges are scrape-only.

16.8 The Diagnostics self-test

Monitoring & troubleshooting → Diagnostics runs ten checks against the running server and reports each in plain language. Run this first when something breaks.

Two things to take from a capture with nothing in its results pane. The two boxes above it are the ones the note under the table is about — three of the ten tests are skipped in silence when they are empty — and Copy puts the whole result on the clipboard, which is what to paste into an issue. The Message-store consistency card that 6.2.28 puts below the results did not exist in this build.

Test	What it actually does	Fails when
Server details	Reports version, logical processors, physical memory and database type	Never — it is information, not a check
Installation paths (6.3.0)	Lists hMailServer.ini, the registry InstallLocation (32-bit view), every [Directories] key, the directories derived from the program folder, and every path-shaped [Settings] and [Database] value, each marked exists, MISSING or not set	Any configured path does not exist, or ProgramFolder and the registry anchor name different directories — the exact failure of a moved installation (Relocating an Installation walks the repair)
Test IPv6	Asks whether the operating system offers IPv6	Never — it reports either way
Test outbound port	Connects to the configured smart host, or to the test domain on port 25 when no smart host is set, using the configured delivery bind address and connection security	Outbound 25 is blocked by your ISP, cloud firewall or local firewall; or the smart host's name, port or TLS mode is wrong
Test backup directory	Writes a small temporary file into the configured backup destination and deletes it again	No destination is configured, or the service account cannot write there. This is the check that catches a backup schedule which has been failing silently
Test MX records	Resolves MX for the local domain you named, through the server's own resolver	Your public DNS is wrong, or the resolver is not answering — see the DNSServer warning in Diagnosing Stalled Mail
Test local connect	Connects on port 25 to each host in your own MX records, in turn	Hairpin NAT does not work, port 25 is not forwarded, or the MX points somewhere else entirely
Test message file locations	Confirms that every message row's file lives under the data directory, and that paths are stored relative	Messages exist outside the data directory, or full paths are stored — either makes the built-in backup refuse to run
Test IP range configuration	Counts the expiring (auto-ban) IP ranges and looks for one that bans 127.0.0.1	Localhost has been auto-banned, which breaks local submission and anything that talks to the server through loopback
Error logs	Counts ERROR* files in the log directory	Any exist. Deliberately strict: an error log is a thing to read, not a thing to have

Three of these need a domain first: Test MX records and Test local connect use the local domain box, and Test outbound port uses the test domain box when no smart host is configured. Leave those boxes empty and the tests are skipped silently rather than failing.

The same tests over COM, which is what an unattended check would use:

$app = New-Object -ComObject hMailServer.Application
$app.Authenticate("Administrator", "<password>") | Out-Null
$d = $app.Diagnostics
$d.LocalDomainName = "example.com"
$d.TestDomainName  = "example.com"
$r = $d.PerformTests()
for ($i = 0; $i -lt $r.Count; $i++) {
    $t = $r.Item($i)
    "{0,-30} {1}" -f $t.Name, $(if ($t.Result) { "OK" } else { "FAILED" })
    if (-not $t.Result) { $t.ExecutionDetails }
}

16.9 A worked example: Prometheus, alerts and Grafana

Scrape configuration. On a non-loopback bind you must supply the credential:

scrape_configs:
  - job_name: hmailserver
    scrape_interval: 30s
    scrape_timeout: 10s
    metrics_path: /metrics
    authorization:
      credentials: "<the value of MetricsServerAuthToken>"
    static_configs:
      - targets: ["mail.example.com:9100"]

Use scheme: https when MetricsServerCertificateFile and MetricsServerPrivateKeyFile are set — the whole port is HTTPS then, probes included. Keep scrape_timeout well under scrape_interval: the listener serves one connection at a time.

Alerting rules. Each of these is written against a metric described above, and each has a reason to exist rather than a threshold picked for tidiness:

groups:
  - name: hmailserver
    rules:
      - alert: HMailServerDown
        expr: hmailserver_state{state="running"} == 0 or up{job="hmailserver"} == 0
        for: 2m
        annotations:
          summary: "hMailServer is not in the running state"

      - alert: HMailServerDatabaseUnreachable
        expr: hmailserver_database_connected == 0
        for: 2m
        annotations:
          summary: "The database has not answered a readiness probe recently"

      - alert: HMailServerQueueStalling
        expr: hmailserver_delivery_queue_oldest_message_age_seconds > 3600
        for: 15m
        annotations:
          summary: "The oldest queued message is more than an hour old"

      - alert: HMailServerDeferralsRising
        expr: rate(hmailserver_messages_deferred_total[15m]) > rate(hmailserver_messages_delivered_total[15m])
        for: 30m
        annotations:
          summary: "More delivery passes are being deferred than are succeeding"

      - alert: HMailServerCertificateExpiringSoon
        expr: hmailserver_tls_certificate_expiry_seconds - time() < 14 * 86400
        annotations:
          summary: "A configured TLS certificate expires in under 14 days"

      - alert: HMailServerAuthFailureSpike
        expr: rate(hmailserver_auth_failures_total[5m]) > 1
        for: 10m
        annotations:
          summary: "Sustained authentication failures - check Access and abuse protection, Auto-ban"

      - alert: HMailServerWorkQueueBacklog
        expr: hmailserver_workqueue_depth{queue="Asynchronous task queue"} > 20
        for: 5m
        annotations:
          summary: "Messages are queuing behind the accept pipeline"

      - alert: HMailServerMetricsProbed
        expr: rate(hmailserver_metrics_unauthorized_requests_total[5m]) > 0
        annotations:
          summary: "Something is presenting bad credentials to the metrics port"

      - alert: HMailServerDbPoolSaturated
        expr: hmailserver_db_connections{state="available"} == 0
        for: 5m
        annotations:
          summary: "Every pooled database connection is busy"

Grafana. The repository ships a dashboard — hmailserver/docs/grafana-dashboard.json — already wired to the metric names above. Import it and pick your Prometheus data source; the $instance variable selects the server.

Row	Panels
Health	Server state, Database, Uptime, Missing message files, TLS certificate expiry
Mail flow	Delivery outcomes (delivered, deferred and bounced rates), Delivery queue depth and oldest message, Messages processed / spam blocked / viruses removed, Active sessions
Authentication and TLS	Authentication outcomes, TLS handshakes
Latency and database	Command processing latency (p50/p95/p99), Database query latency (p50/p95/p99), Database connection pool, Slow queries

A five-minute smoke test after any change to this configuration:

# 1. Is the listener even there? A refused connection here is almost always
#    an invalid MetricsServerBindAddress - check the application log.
Test-NetConnection 127.0.0.1 -Port 9100

# 2. Do the probes answer without a credential? They must, in every configuration.
Invoke-RestMethod http://127.0.0.1:9100/livez
Invoke-RestMethod http://127.0.0.1:9100/healthz

# 3. Does the exposition answer, and how many lines is it?
$m = Invoke-RestMethod http://127.0.0.1:9100/metrics
($m -split "`n").Count

# 4. Spot-check the three signals that matter most.
($m -split "`n") |
  Select-String -Pattern '^hmailserver_state\{state="running"\}|^hmailserver_database_connected |^hmailserver_delivery_queue_oldest_message_age_seconds '

16.10 Verified against the code

Checked 8 September 2026 against hMailServer 6.2.28. The metric families, their help text, types and order are MetricsServer::BuildMetricsBody_; the histogram bucket bounds are CommandLatencyBounds and DatabaseLatencyBounds in ServerStatus.cpp; the state series are StateSeriesTable. The probe dispatch, the "probes first" ordering and the 503-rather-than-401 decision are MetricsServer::HandleClient_, BuildMetricsUnavailableResponse_ and IsLoopbackAddress_; the readiness conditions and their exact reason strings are IsDatabaseAnswering_ and IsReady_; the /healthz body, and the deliberate absence of session counts from it, are BuildHealthBody_. The refresher's cadences (5 s, 30 s after a failure, a 20 s staleness ceiling, 10 s for the queue, 5 minutes for certificates), the 8 KB request cap, the 5 s read deadline and the 15 s write deadline are the constants at the top of MetricsServer.cpp; the 128-series certificate ceiling and the ACME fallback are RefreshCertificateExpiryCache_. The counters' call sites are SMTPDeliveryManager::DoWork (processed, one per delivery pass) and SMTPDeliverer::DeliverMessage and SubmitErrorLog_ (delivered, deferred, bounced). The seven work-queue names are Application's constructor plus IOService, SMTPDeliveryManager and ExternalFetchManager. Every default in the tables is IniFileSettings::LoadSettings. The metric history is MetricsHistoryTask and the OTLP push is OtelMetricsExporter::BuildExportJson_. The nine self-tests are Diagnostic::PerformTests and the Test* classes beside it. The log file names are Logger::GetCurrentLogFileName, and what LogDeleteDays will and will not delete is LogRetentionTask::DoWork. The Dashboard's two-second refresh is DashboardView.xaml.cs. There is regression coverage for the parts that would otherwise fail silently: HealthProbes.cs, MetricsSecurity.cs, MetricsReadiness.cs, PrometheusConventions.cs, MetricsHistory.cs, PerDomainMetrics.cs, DatabaseMetrics.cs, DeliveryMetrics.cs and MessageStoreConsistency.cs, all under test/RegressionTests/Infrastructure.

---