[codex] fix post-restart missing Job slot stalls (BLO-17239)

[kkroo]

Thinking Path

• Paperclip coordinates agent work and persists each heartbeat run's control-plane state.
• Kubernetes adapters additionally persist an external-runtime reservation and exact Job identity so one live execution owns each slot.
• The orphan reaper protects fresh runs from partial Kubernetes inventory and allows an in-process adapter time to parse terminal output.
• During a worker rollout, four Jobs disappeared after the old worker stopped, leaving no in-process adapter capable of finalizing their still-running rows.
• Exact-name Kubernetes lookups returned NotFound, but the reaper still applied the generic 15-minute freshness grace and held those agents' only slots.
• This pull request bypasses that grace only for a confirmed exact-name 404 with no local adapter owner, while preserving both existing false-positive protections.
• The benefit is immediate post-restart convergence without making partial-list or same-process finalization races destructive.

Linked Issues or Issue Description

Bug report (no matching GitHub issue exists):

• Observed behavior: after a worker rollout, a running K8s heartbeat row and active external-runtime reservation can remain after the exact persisted Job name is already absent. The reservation blocks the agent's only slot until the 15-minute silence floor expires.
• Expected behavior: if Kubernetes positively returns NotFound for the persisted exact Job name and this worker has no adapter invocation finalizing the run, finalize it as job_missing and release the reservation immediately.
• Reproduction: start an external-lifecycle run, persist its launched Job name/UID, restart the worker after the Job disappears, and invoke reapOrphanedRuns while the run has fresh output.
• Production impact: four OpenCode agents were held by missing Jobs during the 2026-07-21 rollout. They were recovered through the supported cancel endpoint and resumed on fresh runs.
• Internal incident: BLO-17239.
• Related PRs reviewed for overlap: fix(heartbeat): require silence floor before reaping when Job missing from kube snapshot (BLO-6843) #143, fix agents reaping missing k8s jobs #360, fix(heartbeat): retry job_missing pr_review runs from the vanished-Job reconciler (BLO-10448) #425, fix(heartbeat): reconcile exact external runtime Jobs #680, fix(heartbeat): hold process_lost during cold-boot kube-blind window so live Jobs reattach across restart (BLO-12564) #686, and [codex] reap terminal prelaunch reservations #740. They cover inventory false negatives, missing-Job recovery, exact identity, cold-boot blindness, and terminal prelaunch reservations, but not this owner-aware exact-404 restart race.

What Changed

• Finalize a fresh missing-Job run immediately when the persisted exact-name lookup confirms NotFound and activeRunExecutions has no local owner.
• Retain the 15-minute grace when a local adapter still owns finalization.
• Retain the grace when the exact lookup is inconclusive, preserving partial-inventory protection.
• Add embedded-PostgreSQL tests for all three branches and reservation release.

Verification

• pnpm --filter @paperclipai/server typecheck
• pnpm --filter @paperclipai/server exec vitest run src/__tests__/heartbeat-process-recovery.test.ts --testNamePattern='exact-missing Job|exact Job lookup' — 3 passed
• Full heartbeat-process-recovery.test.ts — all 144 test cases passed; the local process subsequently hit the suite's existing 120-second embedded-PostgreSQL afterAll cleanup timeout
• git diff --check

Risks

• Low and bounded: behavior changes only after a positive exact-name NotFound and only when there is no in-process adapter owner.
• A same-process adapter parsing terminal output keeps the existing grace.
• An unavailable or inconclusive Kubernetes lookup keeps the existing grace.
• No schema, API, adapter, plugin, dependency, or UI changes.

Model Used

• OpenAI Codex based on GPT-5, with reasoning, shell/tool use, code execution, and live production/GitHub inspection. Exact context-window size is not exposed by this runtime.

Checklist

• I have included a thinking path that traces from project context to this change
• I have specified the model used (with version and capability details)
• I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
• I have searched GitHub for duplicate or related PRs and linked them above
• I have either (a) linked existing issues with Fixes: # / Closes # / Refs # OR (b) described the issue in-PR following the relevant issue template
• I have run tests locally and they pass
• I have added or updated tests where applicable
• If this change affects the UI, I have included before/after screenshots
• I have updated relevant documentation to reflect my changes
• I have considered and documented any risks above
• All Paperclip CI gates are green
• Greptile is 5/5 with no open P2s, recommendations, or follow-ups
• I will address all Greptile and reviewer comments before requesting merge

[allyblockcast]

🔗 Paperclip issue: BLO-17239

[allyblockcast]

🔗 Paperclip issue: BLO-17239

[cursor]

Bugbot is not enabled for your account, so this pull request was not reviewed.

Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs.

[kkroo]

@ally review

[allyblockcast]

Hey @kkroo! Before this PR can be reviewed, a few things need attention:

Missing or incomplete:

• Missing section: ## Thinking Path
• Missing section: ## What Changed
• Missing section: ## Verification
• Missing section: ## Risks
• Missing section: ## Model Used
• No linked issue or inline issue description found — either tag an existing issue with Fixes #NNN / Closes #NNN / Refs #NNN, or describe the underlying issue inline in the PR body following one of our issue templates (https://github.com/paperclipai/paperclip/tree/master/.github/ISSUE_TEMPLATE). See CONTRIBUTING.md → "Link Issues or Describe Them In-PR".
• Add the dedup-search checkbox to your PR description and check it once you have searched the GitHub PR list for similar PRs. See the PR template at .github/PULL_REQUEST_TEMPLATE.md and CONTRIBUTING.md → "Before You Start: Search First".

Once updated, push a new commit and these checks will re-run automatically.

— commitperclip

[kkroo]

@ally review