feat(worker): renew mTLS certificates automatically before they expire

[semics-tech]

What and why

Implements automatic certificate renewal for mTLS-authenticated workers. Previously, the 90-day client certificates issued at enrolment were the only issuance path, making certificate expiry an outage timer: every worker would lose connectivity at day 90 and require manual re-enrolment with a fresh token.

This change adds:

1. Worker-side renewal (CertificateRenewer): Monitors the client certificate's validity, generates a new CSR at half-life (45 days), and sends it to the control plane over the authenticated session. On success, installs the new certificate and reconnects to exercise it while the old one still works.

2. Server-side renewal (renewWorkerCertificate): Issues a fresh certificate over the CSR, stores it with label "Renewed", keeps the current credential valid (so the session carrying the response doesn't break), and revokes older spare credentials to prevent accumulation.

3. Protocol additions: New CertificateRenewalRequest and CertificateRenewalResponse message types in the worker↔hub stream.

4. Auth posture review: New startup check that warns operators if the deployment still has workers on token mode, which is a bearer secret and weaker than the now-automatic mtls.

5. Installer updates: Both bootstrap.sh and install.ps1 now accept --auth-mode / -AuthMode (defaulting to mtls), and enrol fails with a clear error if the configured mode does not match the mode the enrolment token was minted for.

The renewal basis is possession of a working credential (proven by the TLS handshake that opened the session), matching EST simplereenroll (RFC 7030) and kubelet client-certificate rotation. This means a stolen key can renew itself indefinitely, but that is addressed by the per-connection revocation check and audit trail, not by an expiry the legitimate worker would hit first.

Blast radius

• Touches authentication or authorisation
• Changes the wire contract (.proto)
• Changes the database schema — no schema change and no migration. Renewal reuses the existing worker_credentials table, which already supports multiple rows per worker with independent revoked_at. That is what makes the old and new certificates able to overlap without a migration.

What could go wrong:

• A worker configured for mtls but enrolled with a token-mode token would receive no certificate and fail to authenticate. Mitigation: enrol now validates the mode match and fails with a clear error instead of writing an empty certificate file; the dashboard's install command carries the token's own mode.
• A renewal that succeeds on the server but fails to install on the worker leaves the old certificate valid and retries hourly. Mitigation: renewal happens at half-life, leaving ~45 days of runway.
• Key and certificate are two files and cannot be replaced in one atomic step. A mismatched pair survives restarts and locks a worker out exactly as an expiry would. Mitigation: the pair is verified against each other before either is renamed into place, leaving only the gap between two renames, which contains no I/O.
• Stolen worker keys can renew indefinitely. Mitigation: per-connection revocation check and an audit row on every renewal. The hub also refuses renewal from any worker that did not authenticate with mTLS, so a token- or Entra-mode worker cannot obtain a certificate its enrolment never established it holds a key for.

How it was tested

• pnpm lint, pnpm typecheck, pnpm test:unit (670 tests), pnpm proto:check, pnpm audit --audit-level high
• pnpm build:sea, and the bundle started from an empty directory (node rsagent-worker.mjs --rsagent-selftest)
• Integration tests — not run locally (no Docker for the SQL Server container); green in CI on this branch

New test suites:

• packages/worker/test/cert-renewal.test.ts (12): state machine with faked timers — renewal at half-life, immediate renewal when already past it, jitter bounds, delays exceeding setTimeout's 24.9-day ceiling, refusal and timeout retries, mismatched-key rejection, and abandonment on disconnect.
• packages/server/test/worker-cert-renewal.test.ts (14): against a real Postgres — the certificate in use stays valid across its own renewal, older spares are revoked, other workers are untouched, the CSR subject is discarded in favour of the enrolled worker id, and the hub refuses renewal for token/entra workers.
• packages/server/test/auth-posture.test.ts (12): the startup review, including that it fails on rather than off.

Both installers' config templating was rendered for all three auth modes with and without --ca-cert and parsed back through the worker's own zod schema. That caught a set -e abort in the bash templating: a failing command substitution inside a variable assignment takes the assignment's exit status, which would have silently killed the installer for token/entra without --ca-cert. Safe in the heredoc the pattern came from; not in an assignment.

Checklist

• No SQL built by string concatenation or interpolation
• New behaviour has a test
• Error messages tell the reader what to do next
• Generated protobuf regenerated and committed (pnpm proto:gen)
• Docs updated — authentication.md (worker auth section rewritten around which mode to choose), security-architecture.md, security.md, deployment.md, README.md
• docs/migration.md updated — closes the "worker certificate auto-rotation" known gap and reverses the "worker authentication defaults to an API key" decision

Known gaps this leaves behind

Both are recorded in docs/migration.md and neither is in scope here:

• Nothing alerts on a worker that has stopped renewing. Renewal makes expiry unlikely, not impossible. certExpiresAt reaches the dashboard without being displayed; until it is, the signal is a worker.certificate.renewed audit row per worker per half-lifetime.
• RSAGENT_GRPC_TLS_CLIENT_CA (bring-your-own CA) is half-wired. TLS accepts the operator's certificate and then authentication rejects it, because identity is a fingerprint row that only enrolment and renewal create. Cheap to finish given identity is the fingerprint rather than the chain, but until then it should fail at startup rather than per connection.

Note on the base branch

This PR targets docs/security-architecture, not main — it is stacked on the commit it was branched from. Worth confirming that is intended before merging.

https://claude.ai/code/session_0178yxB15VZFq9XU5x1ViCEY