Skip to content

v0.0.12

Choose a tag to compare

@AndrewGrachov AndrewGrachov released this 28 Oct 21:35
· 91 commits to main since this release

Highlights

  • Shared secret store. The board now has one encrypted store for server-side secrets. Three backends are available, chosen with SECRETS_BACKEND:

    • database: AES-256-GCM envelopes in PostgreSQL, with a key ring, rekey rotation (npm run secrets:rotate) and a collector;
    • aws: AWS Secrets Manager;
    • vault: HashiCorp Vault KV v2.

    Every board boot wires it in. (BEL-64 #1079, BEL-136 #1080 #1081, BEL-137 #1082 #1083 #1086)

  • Environment secrets: groundwork for encryption at rest (step 1 of 3). The env_var table can now hold a secret store reference. Claim-time resolution reads secrets stored either way: plaintext rows as before, or rows that point into the store. A secret that cannot be read fails the claim closed with a 503; the job stays queued and the attempt is not used up. Writes are still plaintext in this release. Encrypted writes and a one-shot conversion come in the next release (BEL-149), so a mixed-version rollout never hands a runner an empty secret. (BEL-54 #1088)

  • Dashboard replicas. The chart value dashboard.replicas can go above 1, but only with an S3 artifacts backend. Above one replica, a stopping pod drains for dashboard.preStopSeconds before SIGTERM. The repo allowlist is read on every request. (BEL-90 #1075, BEL-124, #1078)

  • Older builds tolerate a newer schema. A build whose database is ahead of it skips the repeatable views instead of re-creating them. (BEL-125 #1077)

Fixes

  • BEL-127: the k8s driver read a killed runner pod's null container times as real values, and leases looped until the task died. (#1076)
  • BEL-151: the dashboard e2e seed produced a week with no token data only when it ran Friday to Monday, so CI's e2e job failed Tuesday to Thursday. (#1090)
  • BEL-152: the secret store conformance harness's 200 ms per-call deadline raced cases that hold a call. (#1091)
  • BEL-153: on hosts with uutils coreutils (ubuntu-26.04), the tool broker recreated the shared TMPDIR without setgid. Shipped runner images use GNU coreutils and were not affected. (#1092)
  • BEL-154: server/test/secrets.boot.test.ts read the ambient environment, so the repository's test gate failed inside every runner that had GITHUB_TOKEN or BASE_BRANCH set. (#1095)
  • BEL-156: bellows batch wait could exit 0 ("every stage verified") before the board had written the stage's verification. It now succeeds only once the board marks the batch completed. (#1098)
  • Runner images: npm's bundled undici 6.29.0 had two HIGH findings (a certificate-validation bypass and a TLS-renegotiation DoS). No npm release ships the fix yet, so both images replace it with 6.29.1 in place. (#1099)
  • The takeover gremlin bug proof can no longer pass without catching the planted lapse; the cap on excused seeds drops from 50% to 25%. (#1089)

Upgrade notes

  • The secret store is required at boot. The board refuses to start without it.
    • Helm: set secretStore.backend (database, aws or vault).

      • Every backend needs secretStore.operationKey and secretStore.fingerprintKey (each openssl rand -base64 32), or a secretStore.existingSecret that holds them.
      • database also needs secretStore.dataKeys (k1:<32 random bytes, base64>) and secretStore.activeKeyId.
      • aws needs secretStore.aws.region and secretStore.aws.prefix.
      • vault needs secretStore.vault.addr, mount, prefix, role and audience (default vault).

      The chart's validate.yaml refuses a render that is missing any of these.

    • Compose and local runs: run npm run secrets:init once. It writes a gitignored secrets.env (see secrets.env.example).

    • Local kind clusters: make build only restarts pods and does not re-apply chart values. After secrets:init, re-apply them once: node scripts/k8s-local-values.mjs | helm upgrade dev charts/factory -f charts/factory/values-local.yaml -f -, or run make start.

  • Back up the store's keys apart from the database. A database backup can't be decrypted without them. After a rotation, keep a retired key until secrets:rotate verifies it is unused.
  • Four new migrations, applied automatically at board start:
    • 090_views_<hash> (a view marker)
    • 091_secret_store (new tables)
    • 092_secret_external (new tables)
    • 093_env_var_secret_ref: adds two nullable columns to env_var and drops NOT NULL from env_var.value. Additions only; an older pod keeps working against it.
  • Deploy this release before the next one. The next release (BEL-149) moves environment secrets into the store. Every deployment must run this version on all its pods first. Do not upgrade from v0.0.11 straight past it.
  • New chart values: dashboard.replicas (default 1) and dashboard.preStopSeconds (default 15).

Validation

  • On the release candidate v0.0.12-rc-2 (57ce6d6, the code this release ships; 65153d4 only bumps VERSION): it adds only the runner-image undici swap (#1099) to 82d0a26. Both runner images passed the release scan locally, and the test:jobs phases that build and exercise them passed (73 passed).
  • On 82d0a26: typecheck, lint, npm test (10376 passed) and the full test:jobs lane (379 passed) are green, and CI on main is green, e2e included. On 32e8e4d, which differs only by a test file and the CLI's batch verdict, test:db (1271 passed), the helm assertions (347 passed) and verify:ui (102 passed) were also green.
  • Staging: v0.0.12-rc-2 ran on a two-replica staging deployment with the database secret store; migrations 091–093 applied on boot and a task ran end to end.
  • Upgrade from v0.0.11. That release's schema was seeded with rows, then this release's migrations ran on top. 090–093 applied, every table kept its rows, and the migration ledger reached 093.
  • Live batch on kind, with this release's board, driver and runner images and the chart upgraded in place:
    • It ran one automerge stage of two parallel tasks, both writing the same scratch file on a sandbox branch.
    • The first run found BEL-154: the test gate failed in every runner. The agents reported themselves blocked instead of widening their changes.
    • In the rerun:
      • both tasks passed their gates;
      • the second PR conflicted after the first merged, and the board repaired it with a merge that kept both lines;
      • main was untouched.
    • Both PRs were merged by hand and stage verification stayed pending, because the sandbox branch gets no CI (BEL-155). The probe also found BEL-156.

Known limits

  • Environment secret values are still stored in plaintext; that changes with BEL-149, and BEL-150 then removes the plaintext path.
  • Live probes against a sandbox branch can't exercise automerge or stage verification, because CI runs only for main, so a sandbox PR reports no checks and the board waits on a person (BEL-155).
  • driver/src/review/review-step.ts also relies on mkdir -m for special bits. It runs only inside the bookworm runner image (GNU coreutils), so it is noted in BEL-153 rather than changed.