RFC: Kubernetes Secret-backed credentials

[amasolov]

Status: revision 4, implemented in #867 · Code references are against heymrun/heym@801ebb4, rechecked on 8e60077

Changes in revision 4

Following @ckakgun's review of revision 3 (reply):

• Check constraint fixed. source holds the full pointer, and the check is encrypted_config = source. Revision 3's check compared k8s:ns/name with k8s:k8s:ns/name and refused the watcher's own rows (§3).
• No re-queue before the first sync. A process that hasn't synced doesn't claim runs, and its instance reports that in its heartbeat, so dispatch skips it. An in-process run fails only when a node uses a credential that isn't synced, with a clear error. The not_before column and the second migration are gone (§4).
• RabbitMQ revision is an HMAC under a per-process random key, over canonical JSON. The consumer key also covers the credential id and the queue name. Testing it live found two bugs on main that stopped any reconnect, and they're fixed in the same commit (§4).
• Deleted Secrets are never removed automatically. The row stays unavailable until an admin removes it (§5).
• Open questions settled (Decided), and delivery is a single PR (Delivery).

Changes in revision 3

Following @ckakgun's reply, @redbotster's comment and the #862 review:

• RabbitMQ revision is a hash of the decrypted config, for every credential, not resourceVersion or updated_at. It changes only when the value does, so a UI edit reconnects, and a key-rotation rewrite doesn't (§4). Revision 4: an HMAC, and it ships in the single PR.
• Delayed re-queue needs a not_before column on workflow_run_queue. Superseded in revision 4: unsynced processes don't claim runs.
• Write-back guard gains a database CHECK, which also covers raw SQL, migrations and admin scripts (§3, §7). Corrected in revision 4.
• Readiness: a Ready pod may not have synced yet; the admin view shows the sync state (§4).
• WebSocket trigger: nothing to change, as agreed in the thread (§4).

Changes in revision 2

Following @ckakgun's review:

• Write-back guard moved from encrypt_config(), which can't see the row, to an ORM set event on Credential.encrypted_config (§7).
• Startup no longer waits for the first sync. Credentials are unavailable until then (§4). Re-queueing runs that hit one was replaced in revision 4.
• Rotation of long-lived consumers (RabbitMQ) without a restart (§4).
• subprocess sandbox modes are refused while the feature is on (§2, Security).
• Audit needs no new work (Security).
• ENCRYPTION_KEY rotation moved to its own RFC, RFC: Online rotation for ENCRYPTION_KEY (MultiFernet) #862.

Summary

On Kubernetes, let a labelled Secret be a Heym credential. Heym watches Secrets in its namespace that carry heym.run/credential: "true". Each one appears as a read-only credential: annotations give its name, type, owner and sharing, and the Secret's data is its config. Values live in memory only and are never written to Heym's database. Updating the Secret rotates the credential immediately.

Heym only needs Kubernetes, not a particular secret store. Anything that writes a Secret works: External Secrets Operator, SealedSecrets, SOPS through Flux or Argo CD, or kubectl. That lets operators keep credentials in Vault, AWS Parameter Store, 1Password or similar, and declare them in git next to the rest of the deployment.

Motivation

Today a credential can only be created by a person in the UI (api/credentials.py:1223). Its config is encrypted with ENCRYPTION_KEY and stored in credentials.encrypted_config (db/models.py:1110). For teams that run Heym on Kubernetes from git:

• Secrets are copied in by hand, so they drift. Rotating at the source leaves Heym on the old value until someone edits it in the UI.
• Every database backup carries every secret, protected only by ENCRYPTION_KEY, which can't be rotated. Rotation itself is RFC: Online rotation for ENCRYPTION_KEY (MultiFernet) #862.
• An instance can't be rebuilt from its manifests. Credentials, and who they're shared with, have to be re-entered by hand.

Heym already treats Kubernetes as a first-class platform. It has an in-cluster API client (services/sandbox/kubernetes/client.py), runs sandbox pods through it, and ships a Kubernetes deployment guide. This feature is a small module next to that code.

Goals

• A Kubernetes Secret declares a complete credential: name, type, owner, sharing and value.
• Secret values never enter Heym's database or its backups.
• Changing a Secret takes effect without a restart.
• These credentials behave like UI credentials for nodes, $credentials, sharing and output masking.
• Off by default; nothing changes for Docker installs or for Kubernetes installs that don't turn it on.

Non-goals (v1)

• Docker and Podman installs. A file-based source could come later behind the same mechanism (see Future work).
• Clients for particular secret stores, or reading ExternalSecret objects. Heym reads Secrets and nothing else; how a Secret gets made is the operator's choice.
• OAuth credential types (codex, google_sheets, google_drive, bigquery, and the OAuth modes of linear and notion). Heym writes refreshed tokens back to these, so they can't be read-only.
• SSO settings, MCP connections, workflows. The same mechanism can extend to them later.

Design

1. The Secret

apiVersion: v1
kind: Secret
metadata:
  name: heym-cred-llm-gateway
  labels:
    heym.run/credential: "true"
  annotations:
    heym.run/name: llm-gateway              # credential name shown in Heym
    heym.run/type: custom                   # a CredentialType value
    heym.run/owner: admin@example.com       # an existing user, by email
    heym.run/share-users: alice@example.com # optional, comma-separated
    heym.run/share-teams: automation        # optional, comma-separated team names
stringData:
  base_url: http://llm-gateway.internal/v1
  api_key: sk-...

Config comes from the Secret's data, in one of two forms:

• Flat keys. Each data key is a config field, and values are strings. Non-string fields such as port: "5432" or secure: "true" are converted by the credential type's existing validation (validate_credential_config).
• A single config.json key, for types with nested config. It's parsed as the whole config.

Non-secret fields (base_url, host) can sit in the Secret too. Keeping the whole credential in one object is the point.

With External Secrets Operator, the ExternalSecret's target template carries the label and annotations:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: heym-cred-llm-gateway
spec:
  secretStoreRef: { kind: ClusterSecretStore, name: parameter-store }
  target:
    template:
      metadata:
        labels: { heym.run/credential: "true" }
        annotations:
          heym.run/name: llm-gateway
          heym.run/type: custom
          heym.run/owner: admin@example.com
      data:
        base_url: http://llm-gateway.internal/v1
        api_key: "{{ .key }}"
  data:
    - secretKey: key
      remoteRef: { key: /heym/llm-gateway-key }

2. Enabling it

Variable	Default	Meaning
HEYM_K8S_CREDENTIALS	false	Turn the feature on
HEYM_K8S_CREDENTIALS_NAMESPACE	Heym's own namespace	Where to watch for labelled Secrets

Startup refuses to run with the feature on in two cases. Both are configuration errors that won't fix themselves:

• Missing permissions. Heym checks with the existing KubeClient.can_i (client.py:122). If the API server answers that list or watch on secrets is denied, startup fails with a message naming the missing permission. If the API server can't be reached, that's not a configuration error: startup continues, and the check runs again with the first sync (§4).
• A subprocess sandbox mode. HEYM_PYTHON_TOOL_SANDBOX (also used by skills), HEYM_MCP_STDIO_SANDBOX and HEYM_PLAYWRIGHT_SANDBOX each accept subprocess, which runs user code inside the backend container, next to the ServiceAccount token. Scrubbing the environment, as those paths do, doesn't help, because the token is a file. Startup names whichever switch is set.

deploy/kubernetes gains a Role granting get, list and watch on secrets in that namespace, and the variables. The community Helm chart follows with kubernetesCredentials.enabled and .namespace.

3. Storage: a row without a value

Credentials still need a database row: nodes reference credentials by ID, and sharing is stored as rows. For a Secret-backed credential:

• id is deterministic: uuid5(HEYM_NS, "<namespace>/<secret-name>"). If the Secret is deleted and recreated (a GitOps prune and re-apply, or ESO recreating its target), the credential keeps its ID, so workflows that reference it keep working.
• encrypted_config holds a pointer, not a ciphertext: k8s:<namespace>/<secret-name>. It contains nothing secret.
• A new nullable column, source, holds the same full pointer (k8s:<namespace>/<secret-name>). The API and UI use it to decide that a row is read-only. The migration adds it, together with a check constraint: CHECK (source IS NULL OR encrypted_config = source). Postgres then refuses a real value on a Secret-backed row, whatever writes it (§7). Keeping the prefix in source lets a later file source (file:…) use the same constraint unchanged. The constraint is tested against a real Postgres, so a mismatch like revision 3's can't pass unnoticed.

4. Resolution

Every process (each API worker and each cluster worker) runs a watch on secrets?labelSelector=heym.run/credential=true and keeps decoded configs in memory. The cache behaves like a client-go informer:

• Before the first sync, Secret-backed credentials are unavailable, and the process serves normally. Startup doesn't wait for the API server. The release image runs eight API processes per instance (--workers 8), so blocking each one on the API server would keep the UI down whenever the API server is unreachable.
• A process that hasn't synced doesn't claim runs. Re-queueing a run that hits an unsynced credential would repeat the side effects of the nodes that already ran: get_credentials_context drops credentials it can't resolve, so a run only finds out at the node. With clustering off, runs execute in-process and never reach the queue anyway. Instead:
  • A process that hasn't synced doesn't call claim_next.
  • A process whose watch is healthy sets credentials_synced_at = now() on each heartbeat. Sync state belongs to each process, but cluster_instances has one row per instance that all its processes upsert, so the upsert keeps the newer value (GREATEST), and an unsynced sibling can't clear it.
  • Dispatch treats an instance as synced while that timestamp is fresh, the same way liveness reads heartbeat_at, and skips it otherwise, as it does a mismatched instance. Queued rows then go to an instance that has synced.
  • An in-process run fails only when a node uses a credential that isn't synced, with credential "x" is not synced from Kubernetes yet. Until the first sync, Secret-backed credentials enter the credentials context as a marker that raises that error, rather than being missing.
  • A credential that is still missing once the process has synced fails normally (§5).
• Ready is not the same as synced. The pod's readiness doesn't wait for the first sync, for the reason above, so a Ready pod can briefly have no Secret-backed credentials. The admin view (§6) shows each process's sync state and last sync time, and the guide says so, so nobody builds a check that assumes otherwise.
• After the first sync, a broken watch keeps the last-known values while it re-lists. An API-server blip doesn't drop credentials that were working.

Every read of a credential goes through decrypt_config(): 111 calls in 64 files. Resolution goes inside it, so none of those call sites change:

def decrypt_config(stored: str) -> dict:
    if stored.startswith("k8s:"):
        return k8s_credentials.get(stored[4:])   # raises CredentialUnavailable: not synced yet, or Secret missing
    return _decrypt(stored)

Rotation. A watch event replaces the cached config, and the next use gets the new value. Most readers already pick it up:

Reader	Behaviour
Node execution, $credentials	Reads per run. Nothing to change.
Connection pools (Redis, Qdrant, Grist, ClickHouse)	Each pool's key includes a hash of the secret (redis_pool.py:15), so a rotated secret gets a new pool. Nothing to change.
IMAP trigger	Reloads the credential on every poll. Nothing to change.
RabbitMQ consumer	Decrypts once when the consumer starts (rabbitmq_consumer.py:195). The consumer key becomes {workflow_id}_{node_id}_{credential_id}_{queue_name}_{revision}, where the revision is a short HMAC of the decrypted config as canonical JSON, under a random key generated per process, for every credential. An HMAC keeps a weak password from being checkable if the key ever reaches a log, and only the leader holds consumers, so revisions never need to match across processes. _sync_consumers stops consumers whose key disappears before starting new ones, so the leader reconnects with the new value on its next 30-second sync. The revision changes only when the value does: not on an annotation edit, and not when #862's job re-encrypts every row. It also fixes today's gap where editing a RabbitMQ credential, switching to another one or renaming the queue keeps the old connection. Two bugs on main are fixed with it, because they stopped any reconnect from working: _stop_consumer called channel.cancel(), which aio-pika channels don't have, so a removed or deactivated trigger kept consuming (and a deactivated one redelivered each message in a tight loop); and the start task was created without a reference, so it could be garbage-collected mid-connect.
WebSocket trigger	Nothing to change. Its connection headers come from the node's websocketHeaders rather than from credentials, and the credentials context is built per event (websocket_trigger_service.py:419).

5. Syncing rows

Rows (not values) are created, updated and soft-deleted from watch events by one process at a time, under DistributedLockService:

Event	Effect on the row
Secret added	Create the row with its deterministic ID; set owner and shares from annotations
Annotations changed	Update name, type, owner and shares to match
Data changed	Nothing. The value is in memory only.
Secret deleted	Mark the row unavailable, keeping its ID and shares. Using it fails with credential "x": Secret ns/name not found. Rows are never deleted automatically: an admin removes them from the admin view (§6). If the Secret comes back, the row becomes available again with the same ID.
Owner email isn't a user yet	Skip it and log a warning. Retry on the next full re-list (every 5 min), so a first boot before anyone has signed in recovers by itself.
An owner already has a UI credential with that name	Leave both untouched, and log a conflict and show it in the admin view. Never take over a UI credential.

6. Read-only in the API and UI

For rows with source set:

• PUT, DELETE and the share endpoints return 409 "Managed by Kubernetes Secret ns/name".
• The masked value shows the source (k8s:ns/name), not mask_api_key() of the real value, which reveals its first 7 characters (api/credentials.py:398).
• The UI shows a "Kubernetes" badge and disables edit, delete and share. An admin view lists every Secret-backed credential with its status: available, unavailable, owner missing, or name conflict. An admin can remove an unavailable row there. It also shows each instance's sync state and last sync time.

7. Write-back guard

OAuth token refresh, codex_node and the PUT handler decrypt a credential, change it and encrypt it again. On a Secret-backed row, that would write the real value into the database.

• Types: OAuth types are rejected as heym.run/type.
• Guard on the model, not the function. encrypt_config() only sees the config, not the row, so it can't tell. Instead, a set event on Credential.encrypted_config raises for any value other than the k8s: pointer when the row's source is set. It follows the listens_for pattern already in models.py, and a later write-back site fails loudly in tests instead of leaking quietly.
• Every write site goes through it today. All 14 writes to encrypted_config are ORM attribute assignments or Credential(...) constructors, and attribute events fire for constructor arguments too. There are no Core-level update()/insert() statements on credentials and no raw SQL.
• The database enforces it too. An ORM event can't see raw SQL, a data migration or an admin script. The check constraint from §3 can: Postgres rejects any write that puts something other than the pointer on a row with source set. The ORM event stays, because it fails earlier, with a message that names the credential. A test runs a raw UPDATE against a Secret-backed row and expects the constraint to refuse it.

Security

• Only the watcher produces k8s: rows. The API always stores encrypt(config), whose Fernet token starts with gAAAAA. No request body can produce a row that resolves from a Secret, and there's no reference syntax a user could aim at files or environment variables.
• Unlabelled Secrets are ignored, but RBAC can't express that. list and watch on secrets can't be narrowed by label, so the backend's token can read every Secret in the watched namespace. Heym's own keys are in its environment anyway, but other apps' Secrets shouldn't share that namespace. The guide should say: run Heym in its own namespace, or point HEYM_K8S_CREDENTIALS_NAMESPACE at a namespace that holds only Heym credentials.
• The feature depends on no user-controlled code running in the backend container, because anything there can read the ServiceAccount token. Sandbox pods set automountServiceAccountToken: false (sandbox/kubernetes/pod.py:207), and startup refuses the subprocess sandbox modes (§2).
• Creating a Secret means granting a credential. Whoever can create Secrets in the watched namespace can give any user a credential and share it with any team. On Kubernetes that permission is already administrative.
• Values are absent from the database and its backups. Losing ENCRYPTION_KEY doesn't affect these credentials.
• Output masking doesn't change. mask_sensitive_output masks every value in the credentials context, whatever its source. Secret-backed values reach a run, and from there a prompt or a tool, exactly as UI credentials do today; this RFC changes where a value is stored, not how a run uses it.
• Audit needs no new work. Heym's audit lines come from its own API, and these credentials change outside Heym. The record of who changed what already lives where the Secret is managed: git history for GitOps, the secret store's audit log, and the Kubernetes audit log when it's enabled.

Alternatives considered

Alternative	Why not
Mounted files with $__file{} references (Grafana-style provisioning)	Works on Docker too, but needs a file format, a reference syntax, path and env allowlists to stop $__file{/proc/self/environ}, a mount on every workload (with a subPath trap that turns rotation off), and a polling reconciler. Worth doing later as a second source, but it shouldn't block the Kubernetes case.
Reading ExternalSecret objects directly	Ties Heym to one operator and its API versions, and Heym would still need the resulting Secret.
Secrets Store CSI driver	Produces files, so it brings back every problem of the file-based design.
Clients for each secret store inside Heym	Store credentials in a process that runs user workflows, and one integration to maintain per store.

Future work

• A file source for Docker and Podman, behind the same source column, read-only path and write-back guard. Only the loader is new.
• SSO settings from a Secret (heym.run/kind: sso). That would make the security-critical toggles (account creation on first sign-in, password sign-in) declarative.
• MCP connections and their headers.
• OAuth client ID and secret from a Secret, with refresh tokens kept in the database.
• Kubernetes Events on the Secret for conflicts and missing owners, so kubectl describe shows them. This needs create on events.

Decided

1. Owner: an existing user named by email. It keeps today's access rules, with no system user to add.
2. Teams: by name, with an error on ambiguity.
3. Prefix: heym.run/, which the sandbox pods already use.
4. Deleted Secrets: no automatic deletion. Rows stay unavailable until an admin removes them from the admin view.
5. Values reaching a model are outside this RFC, since it doesn't change how a run uses a credential.

Delivery

One PR, merged with its commits, and one migration: the source column and its check constraint, and credentials_synced_at on cluster_instances. Commits, in order:

1. RabbitMQ revision: the HMAC revision in the consumer key, and the two stop fixes. Written and tested against a real RabbitMQ.
2. Core: the source column and its check constraint, the k8s: pointer in decrypt_config, deterministic IDs, read-only API and UI, and the write-back guard (the model event, plus tests for it and for the constraint against a real Postgres).
3. Watcher: the Secret watch and informer-style cache, claim gating and the heartbeat's sync state, the not-synced marker for in-process runs, row sync under the distributed lock, the startup checks (permissions and subprocess modes), and the admin status view.
4. Deployment and docs: the Role in deploy/kubernetes, plus guides for ESO, plain Secrets and SOPS. The community Helm chart follows separately.

[ckakgun]

Thanks for the detailed RFC, @amasolov, and for grounding it in the code. We'll take it forward here as a discussion.

We checked the motivation and it holds: credentials only come in through the API, ENCRYPTION_KEY can't be rotated today, and shared users see the first 7 characters of the masked value. A few points we'd like the design to cover:

• Write-back guard: encrypt_config() only sees the config, not the row. A guard on the Credential model, for example a flush hook, would catch every write site.
• Rotation: RabbitMQ and WebSocket triggers read their credential once when they connect, so they'd need a restart when the Secret changes.
• Startup: the release image runs 8 API processes per instance, so waiting for the first list in each would keep the UI down while the API server is unreachable. Marking these credentials unavailable until the first sync seems safer.
• Sandbox modes: once the ServiceAccount can read Secrets, the subprocess sandbox modes should be refused while this is on.

On audit, we don't see a need for extra work. Heym's audit lines come from the API on the main instance, and these credentials are changed outside Heym, so the record of who changed what already lives where the Secret is managed: git history for GitOps, the secret store's own audit log, and the Kubernetes audit log when it's enabled. Teams usually stream to Splunk, VictoriaLogs or similar.

Separately, we think rotation support for ENCRYPTION_KEY deserves its own development, since it would help Docker installs too. Lets follow this one on a separate discussion. Thanks again 🙏

[amasolov]

Thanks @ckakgun. I checked each point against main (801ebb4). Three hold as stated, and one I couldn't reproduce:

Write-back guard. Agreed, encrypt_config() has no row to check. Every write to credentials.encrypted_config today is an ORM attribute assignment or a Credential(...) constructor (14 sites, mostly OAuth refresh). There are no Core-level update()/insert() statements on credentials and no raw SQL. So a set event on Credential.encrypted_config, following the listens_for pattern already in models.py, sees every one of them, constructors included. I'd add a test that fails if a Core-level write to credentials ever appears, since that would bypass the hook.

Rotation. RabbitMQ: confirmed. _start_consumer decrypts the credential once (rabbitmq_consumer.py:195). Rather than requiring a restart, the Secret's resourceVersion could go into the consumer key, which today is {workflow_id}_{node_id}. _sync_consumers already stops consumers whose key disappears and starts new ones, so a changed key reconnects on the leader's next 30-second sync.

WebSocket: I couldn't find the problem. The trigger's connection headers come straight from the node's websocketHeaders and aren't resolved against credentials (build_websocket_connect_kwargs). The credentials context is built per incoming event (websocket_trigger_service.py:419), so runs already pick up a new value. Is there a path I'm missing?

Startup. Agreed: mark them unavailable until the first sync rather than blocking the eight processes. Two additions:

• A run that hits an unsynced credential, such as a cron firing during an API-server outage, should be re-queued with a delay rather than failed.
• Once a process has synced, a broken watch should keep the last-known values while it re-lists, as client-go informers do, rather than dropping them.

Sandbox modes. Agreed. Three switches accept subprocess: HEYM_PYTHON_TOOL_SANDBOX (also used by skills), HEYM_MCP_STDIO_SANDBOX and HEYM_PLAYWRIGHT_SANDBOX. Startup should refuse if any of them is subprocess while this is on. The subprocess paths scrub the environment, but that doesn't help here: the ServiceAccount token is a file, not a variable.

Audit: agreed.

ENCRYPTION_KEY rotation: agreed, it belongs in its own discussion, which I'll open. Since the key feeds a single Fernet, the natural route is MultiFernet. Rotating online with several instances needs a careful rollout order, and keys_fingerprint() will need to change with it, because today a fingerprint mismatch excludes an instance from dispatch.

I'll update the RFC with these points.

[ckakgun]

Thanks for checking each point, @amasolov.

WebSocket: you're right, we got that one wrong. The headers come straight from websocketHeaders and the credentials context is built per event, so there's nothing to fix there.

RabbitMQ: putting a revision into the consumer key sounds good. If the revision comes from the credential itself, the Secret's resourceVersion here and updated_at for UI credentials, it would also close a gap that exists today: editing a RabbitMQ credential in the UI keeps the old connection, because connect_robust reconnects with the URL it started with.

Startup: agreed on both. For the delayed re-queue, a queue row has no "not before" time today, so that part needs a small addition to the run queue.

The write-back guard, the sandbox switches and the ENCRYPTION_KEY discussion all sound good. Looking forward to the updated RFC 🙏

[redbotster]

the memory-only, never-written-to-db constraint is the right call. one thing to double-check: the ORM set event write-back guard is good, but bulk updates or raw SQL that bypass the ORM (migrations, admin scripts) could still write stale data back silently. worth adding an integration test that intentionally tries a raw update and verifies it fails or no-ops. the re-queue approach for startup is cleaner than blocking, but document that the window can be nonzero even after the pod is "ready" so operators don't wire up health checks that miss it.

separately, since heym is an agent runner, it's worth thinking about whether the credential values themselves can end up in model context (prompt, tool output, logs) during a run, which is a different surface than the db. that's the problem we built 1Claw to address, path references to vault tokens that never surface the raw value. full disclosure, i work on 1Claw. not saying you need it, just flagging it as a layer this RFC doesn't cover.

[amasolov]

I've updated the RFC to revision 3. The changes are listed at the top; here's the reasoning.

RabbitMQ revision. @ckakgun, your point that a revision would also fix UI edits holds: _get_consumer_key is just {workflow_id}_{node_id}, so an edited credential keeps its old connection. But your #862 review found that a rewrite bumps updated_at on every credential, and resourceVersion changes when only an annotation does. So I've used a short SHA-256 of the decrypted config for every credential. It changes only when the value changes, works the same for both sources, and is the same idea the Redis pool key already uses. The UI fix doesn't depend on the rest of the RFC, so I've listed it as a separate first PR. Happy to open it if you're fine with that.

Delayed re-queue. Agreed. WorkflowRunQueue has not_after but nothing like "not before", so the watcher PR adds a nullable not_before column, and the claim query skips rows that aren't due yet. Since that's a migration, I've called it out in §4.

@redbotster: raw SQL and migrations. Good catch. An ORM event can't see them. The migration that adds source now also adds CHECK (source IS NULL OR encrypted_config = 'k8s:' || source), so Postgres refuses a real value on a Secret-backed row whatever writes it. The ORM event stays, because it fails sooner and names the credential. A test runs a raw UPDATE and expects the constraint to refuse it. On readiness: a Ready pod may not have synced yet. That's deliberate, so the UI stays up without the API server. The admin view now shows each process's sync state, and the guide will say so. Credential values reaching a run or a model are unchanged by this RFC: Secret-backed credentials enter a run exactly as UI ones do, under the same output masking.

Still open (§ Open questions): the owner model, teams by name, the annotation prefix, and the grace period on delete. Your call on those would let me start on the core PR.

[ckakgun]

Thanks for revision 3, @amasolov, and thanks @redbotster for the review.

Check constraint: as written, it refuses the watcher's own rows. source holds the same pointer as encrypted_config, so encrypted_config = 'k8s:' || source compares k8s:ns/name with k8s:k8s:ns/name. We tried it on Postgres 16. Either source holds only ns/name, or the check becomes encrypted_config = source.

Re-queue before the first sync: get_credentials_context skips credentials it can't resolve, so a run only notices when a node reaches one. Re-queueing at that point repeats the side effects of the nodes that already ran. Also, with clustering off, runs execute in-process and never enter the queue, which covers most single-instance installs. A simpler shape: a process that hasn't synced doesn't claim runs and says so in its heartbeat, so dispatch skips it like a mismatched instance and queued rows move to one that has synced. In-process runs fail with a clear "not synced yet" error until then. That avoids not_before and the second migration.

Config hash: sounds good. Since the key would hold a hash of a secret, an HMAC with a per-process random key over canonical JSON (sort_keys) would keep a weak password from being checkable if the key ever reaches a log.

Open questions:

1. Owner: an existing user named by email, as proposed. It keeps today's access rules, with no system user to add.
2. Teams: by name, with an error on ambiguity.
3. Prefix: heym.run/, which the sandbox pods already use.
4. Grace period: we'd rather not delete rows automatically. Keep them unavailable and let an admin remove them from the admin view.

On values reaching a model, we agree it's outside this RFC, since it doesn't change how a run uses a credential.

We're happy to take all of it in a single PR, the RabbitMQ fix included. Thanks again 🙏

[amasolov]

Thanks @ckakgun. I checked each point against main (8e60077), and they hold.

Check constraint. You're right, and it's my mistake: revision 3 says source holds the same pointer, so the check compares k8s:ns/name with k8s:k8s:ns/name. I'd keep the full pointer in source and make the check encrypted_config = source. A later file source (file:…) then fits the same constraint unchanged. The constraint test will run against a real Postgres, like test_cluster_run_queue_retarget_postgres.py, so this can't slip through again.

Re-queue before the first sync. Agreed. get_credentials_context drops any credential it can't decrypt (except Exception: continue), so a run only finds out at the node, after earlier nodes have had their side effects. With clustering off, the run never reaches the queue at all. I'll drop the re-queue, the not_before column and the second migration. One detail on the heartbeat: sync state belongs to each process, since each of the 8 runs its own watch, but cluster_instances has one row per instance that all of them upsert. A plain boolean would flip with whichever process wrote last. So:

• A process that hasn't synced doesn't call claim_next.
• A process whose watch is healthy sets credentials_synced_at = now() on each heartbeat. The upsert keeps the newer of the stored and incoming values (GREATEST), so an unsynced sibling can't clear it.
• Dispatch treats an instance as synced while that timestamp is fresh, the same way liveness reads heartbeat_at. It skips the instance otherwise, as it does a mismatched one.
• An in-process run fails only when a node uses a credential that isn't synced, with credential "x" is not synced from Kubernetes yet. Until the first sync, Secret-backed credentials enter the context as a marker that raises that error, rather than being missing.

Config hash. Done, as an HMAC over json.dumps(config, sort_keys=True) with a 32-byte key generated per process. Revisions don't need to match across processes, because only the leader holds consumers, and a new leader starts them from scratch.

Open questions. Agreed on all four. On the grace period: with deterministic IDs, a Secret that comes back makes its row available again with the same ID, so workflows that reference it keep working without edits.

Single PR. Fine, with the RabbitMQ fix included. It's written and tested, and will be the first commit:

• The consumer key now covers the credential id, the queue name and the revision. A different credential or a renamed queue reconnects too; both kept the old connection before. Stale consumers are stopped before new ones start, so two consumers don't share a queue during a reconnect.
• _start_consumer connects with the config the key was computed from, rather than reading the credential again. A missing or unreadable credential is still logged once, not on every sync.
• Tested against a real RabbitMQ: editing the credential to another user reconnects as that user on the next sync. Writing the same value back with a new ciphertext, as RFC: Online rotation for ENCRYPTION_KEY (MultiFernet) #862's job does, keeps the connection.

Testing it live turned up two bugs on main that stop the reconnect from working. They affect every RabbitMQ trigger today, not just rotation:

• A stopped consumer keeps consuming. _stop_consumer calls channel.cancel(tag), but aio-pika's RobustChannel has no cancel. The AttributeError skips both closes, and the sync loop logs "Unregistered" anyway. So a deleted or deactivated trigger keeps its connection until the process restarts. For a deactivated trigger it's worse. _handle_message nacks with requeue=True, so the broker redelivers each message at once. On main one message was rejected about 4,100 times in 5 seconds, each pass opening a DB session. The fix cancels through the queue (queue.cancel(tag)), and each teardown step runs even if an earlier one fails.
• The start task can be garbage-collected. asyncio.create_task(self._start_consumer(...)) keeps no reference, and the loop holds tasks only weakly. I hit it once on Python 3.14: "Task was destroyed but it is pending" mid-connect, and the consumer stayed registered without consuming. The consumer now holds its start task, the same pattern as _DETACHED_TASKS in workflows.py. Stopping a consumer also cancels a start that's still connecting.

Both fixes are in the same commit, with tests that use RobustChannel-specced mocks. A plain MagicMock accepted .cancel(), which is how the first bug went unnoticed. The redelivery loop is a live problem on main. If you'd rather not wait for the #861 PR, I can send just that commit now.

I'll update the RFC to revision 4 with these changes and start on the PR.

[amasolov]

The implementation is up as #867: one PR, with the RabbitMQ fix as its first commit. Its description covers the live tests on a kind cluster and the three places it departs from revision 4: how a run fails before the first sync, making room under the executor's line limit, and when the write guard checks a pointer without a source.