Skip to content

Cert hot-reload misses Kubernetes projected-volume (..data symlink-swap) rotations #17

Description

@kriszyp

Background

@harperfast/symphony@0.5.0 (#14) added live cert-file hot-reload to the standalone symphony-server: it watches the parent directories of the cert/key files referenced by the config and, on a change event, filters by the referenced basename (certFilesByDir.get(dir)?.has(filename)) before triggering a debounced reconcile.

Limitation

Basename filtering handles in-place writes and renames of the real files — which is what host-manager does (writes keys/fullchain.pem / keys/privkey.pem directly). It does not handle rotation performed by swapping a symlinked directory, which is how several common environments rotate certs:

  • Kubernetes Secret / projected volumes (and cert-manager) update a mount atomically by writing a new timestamped dir (..2026_07_06_...) and re-pointing the ..data symlink. The watched directory sees rename events for ..data / ..2026_*, none of which match the configured basename (e.g. tls.crt), so they're filtered out and no reconcile fires.
  • Let's Encrypt live/ symlinks can hit the same shape depending on how the symlink is replaced.

Net: in a symlink-swap deployment the hot-reload silently no-ops, and symphony keeps serving the old cert until a restart or an unrelated config write. (Per-route KeyMismatch isolation and last-good carry-forward from #14 still hold — this is purely about noticing the rotation.)

Possible fixes

  1. Resolve symlinks (fs.realpathSync) and watch the real target directory, re-resolving on each reconcile.
  2. When the watched dir contains a ..data-style symlink, don't drop unmatched rename events — let them through to the debounced reconcile (it's already coalesced + cheap).
  3. A periodic re-stat fallback (compare mtime/inode of referenced files) as a backstop for watchers that miss events.

Option 2 is the smallest; option 1 is the most general. Both should stay bounded (dedup by dir, existing debounce).

Priority

Low for the current deployment model (host-manager writes real files in place, which works today). Relevant if symphony is ever run under Kubernetes projected volumes / cert-manager.

Filed as a follow-up to #14. Issue text generated by an LLM (Claude Opus 4.8) on Kris's behalf.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions