Secret Rotation

Secret Rotation

This runbook covers the scheduled rotation of secrets in the tmi-secrets Kubernetes Secret (tmi#965), available from server 1.17.0. A CronJob named tmi-rotator runs daily and rotates the Redis password and the settings encryption key. Each run does at most one idempotent step per rotation, so a rotation that needs a server rollout finishes over one or more runs and resumes after any failure.

All commands use namespace tmi-platform. On AWS, add --context <your-context> to every kubectl call. Never print Secret data; read annotations only.

Overview

• The rotator writes tmi-secrets and waits for tmi-server to roll (Stakater Reloader rolls it when the Secret data changes; annotation-only changes do not roll it).
• The CronJob schedule is 17 3 * * * (daily). concurrencyPolicy: Forbid, backoffLimit: 0, so a failed phase is resumed by the next day's run, not retried in a loop.
• The rotator runs settings-key first, then redis-password. A failure in one does not stop the other; the Job exits 1 if any failed.
• Redis is not restarted for a password rotation. The rotator changes the password with ACL SETUSER and the new value is written to the Secret. A restarted Redis takes its password from tmi-secrets.

What rotates and how often

Rotation name	Secret key(s)	Default interval
redis-password	TMI_REDIS_PASSWORD	90d
settings-key	TMI_SECRET_SETTINGS_ENCRYPTION_KEY (+ _CONTEXT_ID)	90d

A rotation is started when it is never-rotated (no rotated-at annotation), when rotated-at is unreadable, or when rotated-at is older than its interval. A rotation whose phase annotation is set is always resumed, whether or not it is due.

Overriding the interval

Set the annotation tmi.dev/rotate-every.<name> on tmi-secrets. The value is either <n>d (days, 1 to 36500) or any Go duration such as 720h. Zero, negative, or unparsable values are logged as a warning and the 90d default is used (never "rotate now").

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotate-every.redis-password=30d --overwrite

Phases

Phase is stored in tmi.dev/rotation-phase.<name>. No annotation means idle.

redis-password

Phase	Meaning
(idle)	Nothing in progress.
swapped	Redis already accepts the new password and tmi-secrets holds it. Waiting for tmi-server to roll onto it; then the rotator re-validates the Secret and retires the old password with ACL SETUSER default resetpass >NEW, which leaves the new password as the only one Redis accepts, and the phase clears.

settings-key

Phase	Meaning
(idle)	Nothing in progress.
staged	A new key is in _PREVIOUS_KEY with the next context id. Waiting for the server to roll so every replica can decrypt with it before it becomes current.
promoted	The new key is current and the old key is now _PREVIOUS_KEY. Waiting for the roll, then all settings rows are re-encrypted under the new key.
reencrypted	Rows are re-encrypted. The old key is kept for the grace period (see Grace period), then dropped once no row still uses its id.

Annotations the rotator writes (on tmi-secrets)

Annotation (suffix is the rotation name)	Purpose
tmi.dev/rotation-phase.<name>	Current phase; absent when idle
tmi.dev/rotated-at.<name>	RFC 3339 time of last completed rotation (settings-key: the promotion time)
tmi.dev/rotate-every.<name>	Operator-set interval override (never written by the rotator)
tmi.dev/rotation-generation.<name>	tmi-server Deployment generation recorded at the last phase write; the rotator waits for a rollout beyond it
tmi.dev/rotation-promoted-at.settings-key	Promotion time, present in reencrypted; the grace period is measured from it

Forcing a rotation

make rotate-secret CLUSTER=docker-desktop name=redis-password
make rotate-secret CLUSTER=k3s name=settings-key
make rotate-secret CLUSTER=aws CONTEXT=<your-context> name=settings-key

name is redis-password or settings-key. CONTEXT is required for aws and is never taken from your current context. The command creates a Job from the CronJob, sets ROTATE=<name>, waits up to 40 minutes, and prints the Job logs. Forced Jobs expire 24 hours after they finish (ttlSecondsAfterFinished), so read their logs before then.

A forced run starts the named rotation regardless of age and also performs any other rotation that is due or in progress. A secret that has never been rotated is due on its first run, so the first forced or scheduled run rotates both.

Exit code	Meaning
0	Job succeeded
1	The Job failed or did not complete (read the logs printed above)
2	Bad arguments or context (for example CLUSTER=aws without CONTEXT)
3	A tmi-rotator Job is already running; wait for it

Note: the rotator binary itself uses exit 1 for a failed rotation and exit 2 for configuration or connection errors (Kubernetes, Redis, database).

Reading state

Annotations only:

kubectl -n tmi-platform get secret tmi-secrets -o json | jq '.metadata.annotations | with_entries(select(.key | startswith("tmi.dev/")))'

Recent Jobs and logs:

kubectl -n tmi-platform get cronjob tmi-rotator
kubectl -n tmi-platform get jobs -l app=tmi-rotator
kubectl -n tmi-platform logs job/<job-name>

Every run logs one rotation status line per rotation before deciding what to do, with rotation (the rotation name), age_days (whole days since rotated-at, or -1 if never rotated) and phase. Other useful lines: Rotation due, Rotation forced, Rotation resuming, Rotation phase recorded, Rotation failed, Rotation run complete.

Recovering

Phases are resumable: fix the cause, then re-run (make rotate-secret ... or wait for the next CronJob run). Do not clear a phase annotation unless a case below says so.

Stuck phase. Read the Rotation failed line in the last Job log. Common causes: tmi-server did not finish rolling within TMI_ROTATOR_ROLLOUT_TIMEOUT (default 10m; fix the rollout, then re-run), or the database or Redis was unreachable.

"another rotator run is active?" (conflict). The phase changed between the rotator's read and write, meaning a second run overlapped or someone edited the annotation. Overlapping forced runs behave this way: one exits with the conflict (also reported as "secret changed since it was read") and writes nothing, and the other completes. Check kubectl get jobs -l app=tmi-rotator for a running Job and re-run once it ends.

Abandoning a redis-password rotation in swapped. Nothing needs undoing: the Secret already holds the working password and Redis accepts it. Remove the phase annotation to leave the rotation idle:

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotation-phase.redis-password-

The old password stays valid in Redis until the next Redis restart (Redis rebuilds the default user from the Secret at every start) or the next completed Redis rotation, whose retire step drops it.

Redis rejected the password in tmi-secrets. The rotator logs "Redis rejected the password in tmi-secrets" and exits 2. This happens if Redis restarted between the Secret write and the roll. Redis takes its password from the Secret at start, so:

kubectl -n tmi-platform rollout restart deploy/redis

The append-only file keeps data across the restart (sessions survive). Then re-run the rotator. The same command fixes a tmi-server that logs WRONGPASS: the restart reloads the password from the Secret.

Redis restart during a Redis rotation. A Redis restart while a redis-password rotation is in progress can fail that run: the rotator's own Redis connection still uses the pre-swap password. The next forced run or the nightly job completes the rotation.

Orphaned Redis passwords. If a run adds the new password to Redis and then loses the compare-and-swap on the Secret (see the conflict case), that password stays accepted by Redis but is not in tmi-secrets. No manual ACL cleanup is needed: when a rotation completes, the retire step (ACL SETUSER default resetpass >NEW) drops every password except the one in the Secret. A Redis restart also discards them.

Settings key refused to promote. staged phase but the staged key pair is missing or inconsistent; refusing to promote means _PREVIOUS_KEY is empty, or _PREVIOUS_CONTEXT_ID is not the current context id plus one, while the phase is staged. In staged the rotator has only added the previous pair; the current key and id are unchanged and nothing is encrypted under the staged key. Two options, both supported by the code:

Untested. These recovery steps were derived from reading the rotator code and have not been exercised on a real cluster. Read the annotations first and proceed carefully.

1. Restore the pair: put a 64-hex-character key in TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY and set TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_CONTEXT_ID to the current TMI_SECRET_SETTINGS_ENCRYPTION_CONTEXT_ID plus one (if the current context id is absent, it counts as 1, so use 2). Re-run; the rotator promotes it.
2. Start over: remove the phase annotation and both previous entries. With no phase and no previous id, the next due or forced run stages a fresh key. Removing the phase alone does not work: a run refuses to start while a previous key id is present.

kubectl -n tmi-platform annotate secret tmi-secrets tmi.dev/rotation-phase.settings-key-

Remove the two previous entries with kubectl patch --patch-file (a umask 077 file setting each to null). Use either option only in staged, and only if the current key and id were never edited. In promoted or reencrypted, or if the current key or id changed, the new key may already encrypt rows: contact the maintainers.

_PREVIOUS_KEY without _PREVIOUS_CONTEXT_ID. An existing deployment with TMI_SECRET_SETTINGS_ENCRYPTION_PREVIOUS_KEY but no ..._PREVIOUS_CONTEXT_ID makes the CronJob fail every day by design: the rotator refuses to guess the id. Add the id (a positive integer, the id the previous key was current under, normally 1), or remove the previous key if nothing is encrypted under it. Also: a run refuses to start a new settings-key rotation while any previous key id is present.

Undecryptable settings rows. Rows the rotator cannot re-encrypt are logged as Setting could not be re-encrypted key=<setting key> (key only, never values) and skipped. They do not fail the run, but the previous key is not dropped while any row still uses its id. Investigate or delete those settings, then the next run drops the key.

Grace period

After re-encryption, the old settings key is kept for TMI_ROTATOR_SETTINGS_PREVIOUS_GRACE (Go duration, default 192h, 8 days) because encrypted session and refresh-token entries in Redis may still be encrypted under the old key id until their TTLs expire. The default assumes a 7-day maximum lifetime. If auth.jwt.refresh_token_days or auth.jwt.session_lifetime_days exceed 7, raise the grace to at least that many days plus one day. Set it as a container env var on the tmi-rotator CronJob. Other rotator settings are TMI_ROTATOR_NAMESPACE, TMI_ROTATOR_SECRET, TMI_ROTATOR_SERVER_DEPLOYMENT, TMI_ROTATOR_ROLLOUT_TIMEOUT (see Configuration Reference).

CloudWatch alarm (AWS)

The rotator's rotation status log line feeds a metric (TMI/Rotator SecretAgeDays, taken from age_days). The alarm tmi-secret-rotation-stale fires when the daily maximum exceeds 100 days for two consecutive days, and notifies the security alerts topic.

• Threshold: 100 days (the default interval is 90d, leaving a 10 day margin).
• Missing data counts as breaching. Two consecutive days with no datapoint means the CronJob is not running (suspended, image pull failure, cluster down); check kubectl get cronjob tmi-rotator and recent Jobs.
• A never-rotated secret reports age -1, which never breaches. A fresh cluster still alarms after two days until the rotator's first run has produced a datapoint.
• An alarm with a healthy CronJob means a rotation is failing repeatedly: read the newest Job log.

tmi-secrets ownership on AWS

Terraform seeds tmi-secrets once, then ignores its data and annotations (ignore_changes). The rotator is the only writer of the rotated keys.

• To change a value Terraform seeded but the rotator does not rotate (database URL, JWT secret), patch the Secret manually with a umask 077 patch file so the value never reaches argv or history: kubectl -n tmi-platform patch secret tmi-secrets --patch-file <file>, then delete the file.
• Never terraform apply -replace, taint, or delete tmi-secrets or its namespace while RDS holds encrypted settings. Terraform would reseed the original settings key, and rows encrypted under a rotated key become unreadable.
• There is currently no off-cluster copy of the rotated settings key. This is an open decision; until it is made, treat the in-cluster Secret as the only copy.

Redis persistence

Redis stores an append-only file on the redis-data PVC, so sessions and other encrypted entries survive Redis restarts and rolls. Sessions are encrypted with the settings key held in tmi-secrets, so the two must stay together.

• make dev-down keeps both the PVC and tmi-secrets.
• make dev-nuke deletes the namespace, removing tmi-secrets and the PVC together, and the next start reseeds a fresh password.
• Redis runs with --maxmemory 160mb --maxmemory-policy allkeys-lru, which leaves headroom for the append-only-file rewrite and prevents an out-of-memory crash loop. Trade-off: under memory pressure Redis may evict keys, including token-blacklist entries.
• redis-data survives make dev-down and make dev-up.
• Never delete one without the other: a PVC without its Secret holds entries no key can read, and a Secret without its PVC loses sessions.

Running redis-cli in the pod

Only when needed; authentication comes from the pod's own environment, so no password reaches your terminal:

kubectl -n tmi-platform exec deploy/redis -- sh -c 'REDISCLI_AUTH="$REDIS_PASSWORD" redis-cli --tls --cacert /tls/ca.crt <cmd>'

Coming later

These are planned and not yet operational; this page will be extended as each lands.

• JWT signing keyring (ES256).
• Database user passwords.
• Internal CA rotation (trust old and new CA, reissue leaf certificates, drop the old CA).