[Bug] sandbox-runner rejects all manifests with not_yet_valid until worker + sandbox-runner restart

[Rodesc]

Hello team!

Grateful for the amazing work you're doing. I stumbled upon an issue that might be nice to fix:

What happened

With hardened sandbox mode enabled, every /v1/exec job failed. The sandbox-runner rejected signed execution manifests with not_yet_valid. Clients only saw a generic 500.

Environment

• Kubernetes deployment via helm/codeapi
• CODEAPI_HARDENED_SANDBOX_MODE=true
• Split execution-manifest keys (worker private key, sandbox-runner public key)
• MicroVM / hardened sandbox mode

Reproduction steps

We have not found a reliable local repro yet. In production:

1. Run CodeAPI with hardened sandbox mode and signed execution manifests for several days.

2. Send a normal exec request, e.g. POST /v1/exec with a trivial Python job (print(1)).

3. Observe sandbox-runner logs:

   Rejected sandbox request by execution manifest
   reason: "not_yet_valid"
   

4. Observe service-worker logs:

   Error from sandbox: Execution manifest is not valid yet
   

5. Observe API response:

   500 {"error":"Internal server error"}
   

6. Rolling restart of service-worker and sandbox-runner restores service once the new sandbox-runner pod is Ready.

Expected behavior

Freshly signed manifests should be accepted and code should execute.

Actual behavior

All executions fail with not_yet_valid until both worker and sandbox-runner are restarted.

Notes

• Outage window: ~06:28–08:27 UTC on 2026-07-30.
• Affected pods had been running ~9–15 days.
• Host/pod clocks looked in sync; key pair was unchanged (restart alone fixed it).
• Manifest iat is set at sign time in service/src/sandbox-dispatch.ts; not_yet_valid is raised in assertManifestTimeWindow() when claims.iat > now + tolerance (default 30s).

Workaround

kubectl rollout restart deployment/<release>-service-worker -n <namespace>
kubectl rollout restart deployment/<release>-sandbox-runner -n <namespace>

Wait for the new sandbox-runner pod to be Ready before testing again.