Decide running-pod refresh policy (layer 2)

[Kenny-Heitritter]

Part of #16

Question

imagePullPolicy: Always + :latest only refreshes a pod at spawn, so already-running sessions keep a stale binary until they restart. What is the policy for pushing an update to running sessions?

Decide between (and spec): accept-and-document (rely on idle-cull + new logins), a "restart your server to update" in-Lab banner/notice, idle-cull window tuning during rollouts, or a forced drain (and if so, with what user-warning + safeguards so in-progress work isn't destroyed). Consider urgency tiers (routine vs security-critical codeq update).

HITL. Output: chosen policy (possibly tiered), execution-ready.

[Kenny-Heitritter]

Resolution — running-pod refresh policy (layer 2)

Decision: passive convergence for all routine updates; documented manual break-glass for security-critical; no in-Lab banner; no automated drain.

Mechanism (locked first)

A running singleuser pod is pinned to the codeq binary in the image it spawned with. The only way a running session picks up an update is a full pod restart (stop → start singleuser server → re-pull the :latest image). There is no in-place binary refresh — the binary ships inside the Lab image, so nothing short of a pod restart re-pulls it. Every policy option therefore reduces to when/how a pod restarts and who triggers it.

Policy

Routine updates (features, bugfixes — no security urgency): passive convergence. Accept-and-document.

• No action on rollout. Running sessions converge on their own via idle-cull (idle singleuser servers are stopped; the user's next login spawns a fresh pod on the new image) plus natural re-logins. New pods already get the update immediately at spawn.
• Rationale: force-draining live sessions to ship a routine change is disproportionate and destroys in-progress user work (agent runs, unsaved notebook state).
• Document the expected convergence window in CODEQ_DEPLOYMENT.md: window ≈ the idle-cull timeout (+ absolute max-server-lifetime cap, if one is configured). (Concrete cull-window values feed the doc-completion ticket Validate & complete the CodeQ deployment documentation #17.)

Security-critical updates (e.g. a CVE in codeq): passive convergence remains the default, plus a documented manual break-glass.

• No new automation is built now. We document that an operator can force convergence when a genuine security-critical update warrants it: force-stop stale singleuser pods (JupyterHub admin "stop server", or kubectl delete pod of the singleuser pods) → each user re-logs into a fresh pod on the patched image.
• Document who pulls this lever and the one-line runbook step. This is a blunt instrument (it interrupts active sessions) reserved for the security tier precisely because the trade-off is only justified there.

Rejected

• In-Lab "restart to update" banner/notice — not worth the Lab UI work for a passive baseline; users who care restart themselves, idle-cull converges the rest.
• Automated forced drain on every rollout — disproportionate; destroys in-progress work; unnecessary given passive handles routine and break-glass handles urgent.
• Idle-cull window tuning during rollouts — no separate mechanism; the standing idle-cull window is the convergence SLA, documented as-is.

Execution-ready output

Layer-2 policy = passive convergence (idle-cull + re-login) for routine; documented manual break-glass drain for security-critical; no banner; no automated drain. The only execution artifact is documentation — capture the convergence window + the break-glass runbook line in CODEQ_DEPLOYMENT.md (feeds #17). No Terraform/script changes are required for layer 2.