Skip to content

fix agents reaping missing k8s jobs - #360

Merged
kkroo merged 1 commit into
masterfrom
codex/agent-k8s-job-missing-reaper
Jun 11, 2026
Merged

fix agents reaping missing k8s jobs#360
kkroo merged 1 commit into
masterfrom
codex/agent-k8s-job-missing-reaper

Conversation

@kkroo

@kkroo kkroo commented Jun 11, 2026

Copy link
Copy Markdown

Thinking Path

  • Paperclip orchestrates AI agents as durable company workers.
  • The heartbeat service owns agent run lifecycle recovery when a worker process or external job disappears.
  • Kubernetes-backed agents persist a backing Job name in heartbeat_runs.external_run_id, but the reaper still treated a missing namespace-list entry as potentially false negative.
  • During the production rollout for PR [codex] fix agent workspace submodule preflight #359, a live opencode_k8s Job was deleted while its run still had fresh output, leaving BLO-9818 marked running with no backing Job.
  • This pull request uses the persisted Job name for an exact Kubernetes lookup when the namespace-wide list misses a run.
  • The benefit is faster, more precise recovery for deployment-killed agent jobs without weakening the older false-positive protections for list-only misses.

What Changed

  • Added readAgentJobRunStatusByName() to classify one persisted Kubernetes Job by exact name, including a first-class missing result for 404s.
  • Updated reapOrphanedRuns() to immediately fail/release an external-lifecycle run when its persisted Job name is confirmed missing, bypassing the generic silence floor only for that exact 404 case.
  • Added regression coverage for a fresh-output opencode_k8s run whose persisted Job is gone after a rollout.

Verification

  • pnpm exec vitest run server/src/__tests__/heartbeat-process-recovery.test.ts — file collected successfully, but embedded Postgres tests were skipped on this host because the Postgres init script exited with code 1.
  • pnpm --filter @paperclipai/server typecheck
  • git diff --check

Risks

  • Low risk. The accelerated reap only applies when external_run_id contains a concrete Job name and an exact Kubernetes read returns 404.
  • If the Kubernetes API cannot answer the exact lookup, behavior falls back to the existing silence/staleness heuristic.
  • A run whose Job was deleted after actually completing but before Paperclip recorded completion will still be classified as job_missing; that matches existing behavior for missing external-lifecycle jobs with no terminal status.

For core feature work, check ROADMAP.md first and discuss it in #dev before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See CONTRIBUTING.md.

Model Used

  • OpenAI Codex, GPT-5 coding agent with tool use and repository-local command execution.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • I will address all Greptile and reviewer comments before requesting merge

@kkroo
kkroo merged commit 9e2b0b7 into master Jun 11, 2026
11 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant