RFC: Online rotation for ENCRYPTION_KEY (MultiFernet)

[amasolov]

Follow-up from #861, where @ckakgun suggested taking this on separately because it helps Docker installs too. Revision 4, implemented in #871 · Code references are against 801ebb4, rechecked on c4d84bd.

Changes in revision 4

Following @mbakgun's review of revision 3, as implemented in #871. Where the sections below still say otherwise, this list wins.

• Retired keys, not the last target. The key check refuses a primary that a completed rotation moved away from: key_rotations records those keys. Refusing anything but the last completed target would have blocked phase 2 of the next rotation.
• Leadership uses the compatibility rule only, not dispatch eligibility, so enabled and the weight don't apply. A main kept at weight 0 can still lead, and so can a worker while main is down.
• A worker ahead of the schema. Without the new tables, the key check skips, and the worker starts as it does today.
• Installs where nothing decrypts get a documented way out: start once with HEYM_ACCEPT_CURRENT_ENCRYPTION_KEY=true.
• Unreadable rows for their owners. The Credentials list shows them as Unreadable, with delete, instead of failing the whole page.
• Fingerprint. It builds on the heym-keys-fingerprint HMAC already on main. The primary is its first half, and decrypt_fingerprints holds the same HMAC of each key.

Changes in revision 3

Following @mbakgun's second review (reply):

• The compatibility rule is anchored on main. Under the pairwise rule, one bad instance could push out good ones, main included (Instance compatibility).
• Leadership is covered. In cluster mode, a process must be eligible before it can win the leader lock, and the key check refuses a primary other than the last completed rotation's target (Leadership, Failing early).
• Rows no key can read get their own count and are listed, so "remaining" can reach 0. "0 remaining" comes from a final recount (Re-encryption).
• Canary seeding skips k8s: rows and tries up to 20 rows. The check is a module that the worker entrypoints run too, and the docs cover updating existing worker setups (Failing early, Docs).
• Delivery is a single PR with one migration, and no release tour entry (Delivery).

Changes in revision 2

Following @mbakgun's review:

• Compatibility is checked between every pair of live instances, not just against main. Superseded in revision 3 by a rule anchored on main.
• Re-encryption is an admin action. It runs only while every live instance reports the same primary key, and re-checks that before each batch (Re-encryption).
• Fingerprints use a domain-separated hash and no longer share a digest with the Fernet key; SECRET_KEY stays a strict match (Instance compatibility).
• The key check runs before uvicorn starts, after migrations. Its canary is created only once a stored value has decrypted, with ON CONFLICT DO NOTHING (Failing early).
• Extra keys drop empty entries, and the placeholder check applies to each one (Proposal).
• Rewrites keep updated_at, skip k8s: rows, and keep their progress in the database. audit() is called from the admin action (Re-encryption).
• Open questions 1–3 are settled, and the docs get a note on what rotation does and doesn't protect.

Problem

ENCRYPTION_KEY can't be rotated. services/encryption.py builds a single Fernet from sha256(ENCRYPTION_KEY), and everything Heym encrypts at rest goes through it:

• credentials.encrypted_config, for every credential
• the SSO client secret (services/sso_settings.py)
• Codex OAuth state tokens (api/codex_oauth.py), which are short-lived

Changing the value makes everything already stored unreadable. So a leaked key, a staff change or a compliance rule can't be acted on short of re-entering every credential. A key set wrong also fails late and quietly: on a single instance a run just loses $credentials.x, with no error that names the key.

Proposal: MultiFernet with online rotation

cryptography's MultiFernet takes an ordered list of keys. It encrypts with the first and decrypts with any of them, and rotate() re-encrypts a token under the first key while keeping its original timestamp.

Variable	Meaning
ENCRYPTION_KEY	Unchanged. The key used to encrypt.
ENCRYPTION_KEYS_EXTRA (new, optional)	Comma-separated keys accepted for decryption only

Each key goes through the same sha256 derivation as today, so existing data decrypts unchanged, and an install that never sets ENCRYPTION_KEYS_EXTRA behaves exactly as now.

ENCRYPTION_KEYS_EXTRA is split on commas and each entry is trimmed. Empty entries are dropped: otherwise a trailing comma would add sha256("") as a valid key. Each remaining entry goes through the same placeholder check as ENCRYPTION_KEY, and the settings validator refuses to start on a failure.

Rollout order for online rotation

With several processes, or several instances in a cluster, a rolling restart means old and new configuration run side by side for a while. If a process encrypts with the new key before every other process can decrypt it, those other processes fail. So rotation takes two configuration changes, not one:

Phase	Configuration	Safe because
1. Introduce	ENCRYPTION_KEY=old, ENCRYPTION_KEYS_EXTRA=new. Roll every instance.	Nothing is encrypted with new yet, so a mixed fleet is fine.
2. Promote	ENCRYPTION_KEY=new, ENCRYPTION_KEYS_EXTRA=old. Roll every instance.	Every process can already decrypt new.
3. Re-encrypt	An admin starts it once every instance reports new as its primary	Converts rows to new in batches; reads keep working throughout.
4. Retire	Remove old from ENCRYPTION_KEYS_EXTRA	Only once phase 3 has completed (its final recount found 0 remaining), and no database backup taken before it will ever need restoring.

A single Docker container restarts all its workers together, so it can go straight to phase 2. The phases matter for multi-instance clusters and for anyone doing rolling updates on Kubernetes.

Instance compatibility

Today cluster/identity.py:keys_fingerprint() concatenates sha256(ENCRYPTION_KEY)[:16] and sha256(SECRET_KEY)[:16], and cluster/registry.py:is_compatible_with() compares each instance only with main. Two problems with keeping that:

• During a rotation, comparing only with main isn't enough. A worker that skipped phase 1 still matches main in phase 2 if main hasn't rolled yet. But it can't read a row another worker has already written under new, for example during an OAuth refresh.
• sha256(ENCRYPTION_KEY) is also the Fernet key material. Publishing a prefix of it puts 64 bits of the actual key into the instances table.

Proposed:

• Fingerprints are domain-separated: HMAC-SHA256(key, "heym:fingerprint:v1"), truncated. That is unrelated to the Fernet key bytes.

• Each instance publishes its primary fingerprint, the set of fingerprints it can decrypt with, and a SECRET_KEY fingerprint.

• Eligibility is anchored on main, in two steps:

  1. An instance passes the first step if it can decrypt main's primary, main can decrypt its primary, and its version, schema and SECRET_KEY fingerprint match main's exactly.
  2. Among those, an instance stays eligible only if it can decrypt the primary of every other instance that passed the first step.

  A symmetric rule across all live instances would let one bad instance push out good ones: a worker joining with a typo in its key can't read main's primary, so main itself would drop out of the pool, which can't happen today. Anchoring on main means a bad instance only excludes itself.

  The second step can drop two instances that each fail on the other, for example with three different primaries live at once. That only costs capacity and main can't drop out, so the rule stays simple rather than looking for the largest consistent subset. The rollout order never produces that state.

• The admin Instances view shows each instance's primary and decrypt set, so a stuck rotation is visible.

Leadership

Dispatch eligibility doesn't apply to leader election today. lock_service.start() is the first call in lifespan, before heartbeat_service.start(), and _try_acquire_leader_lock checks only the advisory lock. So any process can win leadership, even before its instance has registered and whatever its version or keys. The leader runs cron, the Google and IMAP pollers and the RabbitMQ consumers, which read credentials and write them on OAuth refresh.

Two checks cover the two windows:

• During a rotation, before any rotation has completed, a worker that skipped phase 1 passes the canary with the old key but can't read rows written under the new one. So in cluster mode, leadership requires eligibility:
  • A process checks its own instance's row against the compatibility rule before calling pg_try_advisory_lock. The leader re-checks on each 5-second loop and releases the lock if it fails.
  • A process whose instance has no heartbeat row yet isn't eligible, so the order of calls in lifespan doesn't matter.
  • The check uses main's last published row even when main isn't live, so cron and the pollers keep running while main is down, as they do today.
  • Outside cluster mode nothing changes.
• After step 4, an instance that comes back still on the old primary passes the canary through its extra key. The key check refuses any primary other than the last completed rotation's target (see Failing early). This also catches a restored old .env on a single instance, where there's no cluster to exclude it from.

Re-encryption (phase 3)

• Started by an admin action, which calls audit() like every other admin route. It is refused unless every live instance reports the same primary fingerprint.
• Runs on the leader, in batches, and re-checks that condition before each batch. If an instance rolls back or a new one joins on the old primary, the job pauses and says why. Without this, a leader still on the old primary would rotate rows back to the old key.
• Idempotent. A row is rewritten with MultiFernet.rotate() only if the primary key alone can't decrypt it.
• Covers credentials and sso_settings. OAuth state tokens expire on their own and are skipped. Rows holding a k8s: pointer (RFC: Kubernetes Secret-backed credentials #861) are skipped; they store no ciphertext.
• Rows no key can read get their own count. Installs that changed the key the hard way before have rows that nothing decrypts. The job reports three counts: done, remaining and unreadable. The admin view lists unreadable rows by name, type, owner and last update, and they're never counted as remaining, so the job can finish.
• Completes on a final recount. Before marking a rotation complete, the job recounts every row, and completes only if that recount finds none remaining; otherwise it runs another pass. This catches rows rewritten mid-run by an instance that joined between heartbeats. The "0 remaining" that gates phase 4 comes from this recount, not from the running total.
• Keeps updated_at. The rewrite is a Core UPDATE … SET encrypted_config = :new, updated_at = credentials.updated_at, so onupdate doesn't fire. A re-encryption isn't an edit, and it shouldn't show as one or change anything keyed on updated_at. This is the one Core-level write to credentials, and RFC: Kubernetes Secret-backed credentials #861's guard test allow-lists it.
• Locks each row while rewriting it (SELECT … FOR UPDATE), so an OAuth token refresh at the same moment can't be overwritten with stale data.
• Keeps progress in the database (a key_rotations row: who started it, target primary, done, remaining, unreadable, state). Progress survives a leader change, every instance reports the same numbers, and the completed row's target is what the key check holds instances to.

Failing early on a wrong key

Store a small canary value in the database, encrypted with the primary key.

• A module, python -m app.key_check, run before uvicorn starts. release-entrypoint.sh runs it after alembic upgrade head. The worker definitions, which start uvicorn directly, run it first (… key_check && exec uvicorn …). With --workers 8, a failure inside the app's lifespan only makes uvicorn respawn the process, so the check has to run before uvicorn.
• Two refusals, each with an error that names the cause:
  • no configured key decrypts the canary: ENCRYPTION_KEY does not match the key this database was encrypted with;
  • the primary isn't the target of the last completed rotation (see Leadership).
• Created only once a stored value has decrypted. On first boot, the check tries up to 20 credentials, newest updated_at first and skipping k8s: rows, plus the SSO secret. On an install that changed the key before, the rows written since are the ones the current key can read. If one decrypts, or the database holds none, the canary is written. If values exist but none decrypt, startup fails without writing a canary, so a wrong key at first boot is never recorded as the right one.
• Concurrent first starts insert with ON CONFLICT DO NOTHING.
• Phase 3 re-encrypts the canary along with everything else.

Docs

• What rotation protects. Rotating protects backups taken afterwards. A leaked key together with a copy of the database still exposes every credential in that copy, so after a leak, rotate the credentials at their providers too.
• deploy/kubernetes and the Compose files gain the extra key as an optional entry in the existing secret (an optional secretKeyRef on Kubernetes). The community Helm chart will follow.
• Upgrade note for existing worker setups. The worker entrypoint is set inline in docker-compose.worker.yml, the worker service in docker-compose.cluster.yml and deploy/kubernetes/overlays/orbstack-cluster/worker.yaml. All three gain the key check, and the note tells operators to make the same change in their own copies. A worker left on the old entrypoint still runs, without the check, so it falls back on the leadership and dispatch rules. The community Helm chart's worker StatefulSet gets the same change.

Out of scope

• External key management (KMS, Vault Transit envelope encryption). Possible later, and the key list is the natural place to plug it in.
• Kubernetes Secret-backed credentials (RFC: Kubernetes Secret-backed credentials #861) store no ciphertext, so rotation doesn't touch them.

Decided

1. Name: ENCRYPTION_KEYS_EXTRA. In phase 1 it holds the new key, so PREVIOUS would mislead.
2. Start: an admin action with the gate above, not automatic.
3. Deployment: deploy/kubernetes and Compose, with the extra key optional.
4. Delivery: one PR, merged with its commits. See below.
5. No release tour entry for key rotation.

Delivery

One PR, not four. On its own, the key list lets an operator add a key but never drop the old one, so a leaked key stays required until the job lands, and dropping it is what a rotation is for. One PR also means one migration and one pass over the docs.

• One migration: the canary table, key_rotations, and the per-instance primary and decrypt-set fingerprint columns.
• Commits, in order: the key list (MultiFernet in encryption.py and ENCRYPTION_KEYS_EXTRA parsing and validation); the key check module and the entrypoints; compatibility and leadership; the re-encryption job and the admin action; docs and deployment entries.

The RabbitMQ config-hash fix from #861 lands with #861's own single PR, as its first commit, as agreed there.

[mbakgun]

Thanks for writing this up so carefully, @amasolov. We checked it and as expected changing ENCRYPTION_KEY makes every stored credential unreadable(its what it is - we keep credentials encrypted - key changes - credentials can't reverted), and on a single instance a run just loses $credentials.x without an error. MultiFernet with the same sha256 derivation reads existing rows unchanged, so the direction looks right to us.

A few points we'd like the design to cover:

• Compatibility across all instances. is_compatible_with only compares each instance with main. A worker that skipped phase 1 still passes against main, but can't read rows another worker writes under the new key during an OAuth refresh. Checking each instance's decrypt set against the primaries of all live instances would catch that.
• Gating the re-encryption job. The leader can be any instance, including a worker that hasn't been promoted yet, and with the "primary alone can't decrypt" rule it rotates rows back to the old key. Running the job only while every live instance reports the same primary also makes "0 remaining" safe to act on in step 4. So on question 2 we'd prefer an admin action with that gate.
• Fingerprint. It also covers SECRET_KEY, which should stay a strict match. While reworking it, a domain-separated hash would be better than a prefix of sha256(key), since that digest is also the Fernet key material.
• Canary. With --workers 8, uvicorn respawns a process whose lifespan refuses to start, so the check fits better before uvicorn starts, after migrations. It should be created only after an existing credential decrypts, so a wrong key at first boot isn't recorded as the right one, and with ON CONFLICT DO NOTHING for the concurrent first start.
• Extra keys. Empty entries need dropping (a trailing comma would add sha256("") as a valid key), and the placeholder check should apply to them too.
• Rewrites. An ORM rewrite bumps updated_at on every credential, which would reconnect every RabbitMQ consumer if that becomes the revision from RFC: Kubernetes Secret-backed credentials #861. The job should also skip k8s: rows and leave audit() to the admin action that starts it, keeping progress in the database.

On question 1, EXTRA reads better than PREVIOUS, since in phase 1 it holds the new key. On question 3, the Helm chart is community maintained(thank you once again), so here it would be the manifests under deploy/kubernetes and the Compose files, with the extra key optional. It may also be worth noting in the docs that rotation protects future backups, while a leaked key together with a database copy still means rotating the credentials at their providers.

Looking forward to proceed 🙏

[amasolov]

Thanks @mbakgun. I checked each point against main (8e60077), and all six hold. I've updated the RFC to revision 2. The changes are listed at the top, and a few needed choices:

• Compatibility. is_compatible_with only compares an instance with main. Eligibility is now pairwise: an instance gets work only if, for every live instance, each can decrypt the other's primary.
• Re-encryption gate. Confirmed: the leader is a Postgres advisory lock that any process on any instance can win. It's now an admin action, refused unless every live instance reports the same primary, and re-checked before every batch. If the fleet changes mid-run, the job pauses rather than rotating rows back.
• Fingerprint. sha256(ENCRYPTION_KEY) is also what becomes the Fernet key, so today's fingerprint publishes 64 bits of it. Revision 2 uses HMAC-SHA256(key, "heym:fingerprint:v1"), and SECRET_KEY stays a strict match against main.
• Canary. It moves to a check that both entrypoints run after migrations and before uvicorn. It's written only after an existing credential or the SSO secret decrypts (or when the database holds none), with ON CONFLICT DO NOTHING.
• Extra keys. Empty entries are dropped, and each one gets the placeholder check.
• Rewrites. The job keeps updated_at (SET updated_at = credentials.updated_at, so onupdate doesn't fire) and skips k8s: rows. It keeps its progress in a key_rotations row, which survives a leader change. audit() comes from the admin action.

On updated_at: in #861 revision 3, the RabbitMQ consumer key now uses a hash of the decrypted config rather than updated_at, so a re-encryption wouldn't reconnect consumers either way. Keeping updated_at is still right, because a re-encryption isn't an edit. This also means the rewrite is the one Core-level write to credentials, so #861's guard test will allow-list it.

I've taken your answers to the open questions: EXTRA, an admin action, and deploy/kubernetes plus Compose with the extra key optional. I've added the docs note about backups and rotating credentials at their providers.

I'd suggest four PRs, in order: the key list (MultiFernet plus the extra keys), the pre-uvicorn key check, compatibility, and the re-encryption job. The first one on its own allows an offline rotation on a single instance. Shall I start with it?

[mbakgun]

Thanks @amasolov, revision 2 covers everything we raised, and the #861 changes (config hash for the RabbitMQ key, the CHECK constraint) settle the updated_at point too. A few things left before you start:

• Compatibility rule. Making it symmetric lets one bad instance push out good ones. A worker that joins with a typo in its key can't read main's primary, so main itself drops out of the pool, which can't happen today. My earlier suggestion had the same hole. Anchoring on main fixes both: an instance first has to match main both ways, then it must decrypt the primaries of every instance that passed that check.
• The leader isn't covered. Dispatch eligibility doesn't apply to leader election. A Mismatch instance can already win the advisory lock and run cron, the Google and IMAP pollers and the RabbitMQ consumers, which read credentials and write them on OAuth refresh. After step 4, an instance that comes back still on the old primary passes the canary through its extra key and could do exactly that. Having the pre-uvicorn check refuse any primary other than the last completed rotation's target would close it, or leadership could require compatibility.
• Rows no key can read. Installs that changed the key the hard way before have rows nothing decrypts. Counting and listing those separately keeps "remaining" able to reach 0, and taking "0 remaining" from a final recount catches rows rewritten mid-run by an instance that joined between heartbeats.
• Canary details. Skip k8s: rows when seeding and try a few rows rather than one. docker-compose.worker.yml sets its entrypoint inline, so the docs should tell existing worker setups to update it.

On the split, we'd prefer to take this as a single PR, merged along with its commits. The key list alone lets you add a key but never drop the old one, so a leaked key stays required until the job lands. That's the part a rotation is for, and one PR also means one migration and one pass over docs. The RabbitMQ fix from #861 corrects today's behaviour, so that one is fine on its own.

And about release tour , we don't need to prepare release tour entry for key rotation capabilities.

Happy for you to go ahead on that basis 🙏

[amasolov]

Thanks @mbakgun. I checked all four against main (8e60077), and they hold. A single PR and no release tour entry are both fine. Here's how I'd handle each point; I'll fold them into revision 3.

Compatibility rule. Agreed. The pairwise rule lets one bad instance push out good ones, main included. Anchored on main, it becomes:

1. An instance passes the first check if it can decrypt main's primary, main can decrypt its primary, and version, schema and SECRET_KEY match.
2. Among those, an instance stays eligible only if it can decrypt the primary of every other instance that passed the first check.

The second check can drop two instances that each fail on the other, for example with three different primaries live at once. That only costs capacity, and main can't drop out, so I'd keep the rule simple rather than look for the largest consistent subset. The rollout order never produces that state.

Leader. Confirmed. lock_service.start() is the first call in lifespan, before heartbeat_service.start(), and _try_acquire_leader_lock checks only the advisory lock. So a process can lead before its instance has registered, whatever its version or keys. I'd do both of your options, because they cover different windows:

• The pre-uvicorn check refuses any primary other than the last completed rotation's target, with an error that names it. That covers the case after step 4. It also catches a restored old .env on a single instance, where there's no cluster to exclude it from.
• In cluster mode, leadership requires eligibility. A process checks its own instance's row against the rule above before pg_try_advisory_lock. The leader re-checks on each 5-second loop and releases the lock if it fails. This covers the window during a rotation, before any rotation has completed: a worker that skipped phase 1 passes the canary with the old key but can't read rows written under the new one.
  • A process whose instance has no heartbeat row yet isn't eligible, so the order of calls in lifespan stops mattering.
  • The check uses main's last published row even when main isn't live. Cron and the pollers keep running while main is down, as they do today.
  • Outside cluster mode nothing changes.

Rows no key can read. Agreed. The job reports three counts: done, remaining and unreadable. The admin view lists unreadable rows by name, type, owner and last update, and they're never counted as remaining, so the job can finish. Before marking a rotation complete, the job recounts every row and completes only if that recount finds none remaining. Otherwise it runs another pass. The "0 remaining" that gates step 4 comes from that final recount, not from the running total.

Canary. Agreed on both. Seeding skips k8s: rows and tries up to 20 rows, newest updated_at first, plus the SSO secret. On an install that changed the key before, the rows written since then are the ones the current key can read. One row decrypting is enough.

The inline worker entrypoint is in three places, not one. docker-compose.worker.yml, the worker service in docker-compose.cluster.yml, and deploy/kubernetes/overlays/orbstack-cluster/worker.yaml all start uvicorn directly. So I'd make the check a module (python -m app.key_check). release-entrypoint.sh runs it after alembic upgrade head, and the three worker definitions run it before uvicorn (… key_check && exec uvicorn …). The upgrade note will cover existing worker setups. It will also say that a worker still on the old entrypoint keeps running without the check, so it falls back on the leadership and dispatch rules. I'll update the community Helm chart's worker StatefulSet the same way.

One PR. Understood: one migration (the canary, key_rotations, and the per-instance fingerprint columns), one docs pass, and the commits kept. Edit: the RabbitMQ config-hash fix now goes in #861's single PR, as agreed there.

[mbakgun]

Thanks @amasolov, this all reads right, and anchoring on main plus gating leadership is what we had in mind. A few details on the new parts, which can go straight into revision 3:

• The rotation-target check would block the next rotation. In phase 2 of a second rotation, the new primary is never the last completed target, so each promoted instance would refuse to start. Refusing a primary that a completed rotation moved away from (keeping the source fingerprint in key_rotations) covers the same cases, including a restored old .env.
• Leadership should use the compatibility rule only. If it reuses dispatch eligibility, enabled and weight come with it, so a main kept at weight 0 for the UI could never lead, and with every worker down nobody would run cron.
• key_check on a worker ahead of the schema. Workers don't run migrations, so a worker restarted on the new image before main has migrated would find no canary table and crash-loop. Treating a missing table as "skip" keeps today's behaviour, where it starts and shows Mismatch.
• Installs where nothing decrypts today. If the key changed and no credential was re-entered since, the upgrade would refuse to start an instance that runs today. The error should point to a documented way out, for example a flag that accepts the current key.
• Unreadable rows for their owners. The credentials list decrypts every row without a guard, so one unreadable row fails the owner's whole Credentials page, and only the owner can re-enter it. Showing those rows as unreadable, with delete, makes the admin list actionable.
• Fingerprint. The HMAC version is already on main from the GHSA-hmfh merge, labelled heym-keys-fingerprint. Building on that avoids a second scheme.

Looking forward to the PR 🙏

[amasolov]

Thanks @mbakgun. All six hold, and they are in #871 as you described them. I have folded them into the RFC as revision 4.

I tested your first point live. On a two-instance cluster, a second rotation promoted a key no rotation had targeted yet, and every instance started. A restored old .env was refused, both by key_check and by the image's own entrypoint. The PR description has the full live run, including leadership moving off a worker that skipped phase 1.