Summary
When Symphony (or another fronting proxy) terminates TLS in front of Harper and routes by SNI using the UDS metadata Harper exports per listener socket, the MQTT network.securePort listener's exported certificate list can come up permanently empty — every 8883 connection then falls back to the instance's own node certificate instead of the SNI-matched one, regardless of which hostname the client requested.
HTTP-based listeners (operations API, plain https) are unaffected.
Root cause
createTLSSelector() (security/keys.ts) builds its secureContexts map from databases.system.hdb_certificate, and registers the map's only rebuild trigger (databases.system.hdb_certificate.subscribe(...)) inside the same synchronous pass that reads the table for the first time.
The raw TCP/TLS socket listener used by MQTT (onSocket() in server/threads/threadServer.js, backing the plugin API's server.socket()) creates its own independent createTLSSelector() per worker thread, separately from the HTTP listeners. If that selector initializes before databases.system has loaded on this thread — which selector creation doesn't control — the cert-table read throws (Cannot read properties of undefined), the .ready promise rejects before the rebuild subscription is ever registered, and the exported UDS metadata is left with an empty certificate list with no path to self-heal (only an unrelated private-key hot-reload elsewhere in the process would trigger a retry).
Secondary bug: the same raw-socket path always calls createTLSSelector('server', ...) regardless of caller, so even when certs are present, one tagged for a specific listener type (uses: mqtt) never gets its quality bonus over a generic node cert for the same hostname.
Fix
createTLSSelector now checks databases.system for readiness separately from databases itself, retries (via the same debounced rebuild used for cert-table changes) instead of permanently stranding the selector, and registers the cert-table subscription lazily/safely once the system database is actually available.- The generic raw-socket listener now accepts a
usageType option (mirroring the HTTP listener path), and the MQTT component passes usageType: 'mqtt'.
Verification
- Confirmed live: on 2 of 3 nodes of an affected production cluster, every worker's exported MQTT UDS metadata (
sockets/*-8883.yaml) had an empty certificates: list since the container's boot, while the HTTP-based listeners on the same workers were fully populated — unchanged for the container's entire uptime.
- Added a unit test (
unitTests/security/keys.test.js) that reproduces the exact race (deletes databases.system before calling createTLSSelector(...).initialize()) — fails against the pre-fix code with the predicted TypeError, passes after the fix.
Summary
When Symphony (or another fronting proxy) terminates TLS in front of Harper and routes by SNI using the UDS metadata Harper exports per listener socket, the MQTT
network.securePortlistener's exported certificate list can come up permanently empty — every 8883 connection then falls back to the instance's own node certificate instead of the SNI-matched one, regardless of which hostname the client requested.HTTP-based listeners (operations API, plain https) are unaffected.
Root cause
createTLSSelector()(security/keys.ts) builds itssecureContextsmap fromdatabases.system.hdb_certificate, and registers the map's only rebuild trigger (databases.system.hdb_certificate.subscribe(...)) inside the same synchronous pass that reads the table for the first time.The raw TCP/TLS socket listener used by MQTT (
onSocket()inserver/threads/threadServer.js, backing the plugin API'sserver.socket()) creates its own independentcreateTLSSelector()per worker thread, separately from the HTTP listeners. If that selector initializes beforedatabases.systemhas loaded on this thread — which selector creation doesn't control — the cert-table read throws (Cannot read properties of undefined), the.readypromise rejects before the rebuild subscription is ever registered, and the exported UDS metadata is left with an empty certificate list with no path to self-heal (only an unrelated private-key hot-reload elsewhere in the process would trigger a retry).Secondary bug: the same raw-socket path always calls
createTLSSelector('server', ...)regardless of caller, so even when certs are present, one tagged for a specific listener type (uses: mqtt) never gets its quality bonus over a generic node cert for the same hostname.Fix
createTLSSelectornow checksdatabases.systemfor readiness separately fromdatabasesitself, retries (via the same debounced rebuild used for cert-table changes) instead of permanently stranding the selector, and registers the cert-table subscription lazily/safely once the system database is actually available.usageTypeoption (mirroring the HTTP listener path), and the MQTT component passesusageType: 'mqtt'.Verification
sockets/*-8883.yaml) had an emptycertificates:list since the container's boot, while the HTTP-based listeners on the same workers were fully populated — unchanged for the container's entire uptime.unitTests/security/keys.test.js) that reproduces the exact race (deletesdatabases.systembefore callingcreateTLSSelector(...).initialize()) — fails against the pre-fix code with the predictedTypeError, passes after the fix.