Skip to content

fix(heartbeat): suppress idle opencode k8s timer runs - #181

Merged
blockcast-ci-packages[bot] merged 2 commits into
masterfrom
blo-7594-opencode-k8s-no-work-suppression
May 26, 2026
Merged

fix(heartbeat): suppress idle opencode k8s timer runs#181
blockcast-ci-packages[bot] merged 2 commits into
masterfrom
blo-7594-opencode-k8s-no-work-suppression

Conversation

@blockcast-ci-packages

Copy link
Copy Markdown

Thinking Path

  • Paperclip orchestrates AI agents for zero-human companies.
  • The heartbeat subsystem starts adapter runs for timer/system wakes and issue-driven wakes.
  • opencode_k8s timer/system wakes with no assigned work can wedge before model output, creating stale active-run wrapper issues with no real source work to recover.
  • BLO-6740 asks for the preferred fix: avoid spawning opencode_k8s timer runs when the agent has no live assigned work.
  • This pull request adds an explicit no-work gate for opencode_k8s timer/system heartbeats while preserving issue-assigned wakes.
  • The benefit is less stale-run noise, lower wasted triage spend, and a clearer signal when a real pre-LLM wedge occurs.

What Changed

  • Added heartbeat gating so opencode_k8s timer/system runs check for assigned live work before spawning.
  • Added searchable skip telemetry/logging with a no_in_flight_work reason when an idle opencode_k8s timer wake is suppressed.
  • Preserved actionable issue wake behavior for agents with assigned todo, in_progress, or in_review work.
  • Added coverage for no-work skip and work-present spawn paths, plus related test fixture updates.

Verification

  • Branch commit includes new/updated tests for the timer no-work suppression path:
    • server/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.ts
    • server/src/__tests__/heartbeat-retry-scheduling.test.ts
    • server/src/__tests__/issue-stale-execution-lock-routes.test.ts
    • server/src/services/recovery/successful-run-handoff.test.ts
  • CI should run the repository PR workflow on this PR, including verify, general test groups, serialized server shards, build, and e2e.
  • Post-deploy verification for BLO-6740: observe at least one opencode_k8s skipped: no_in_flight_work event within 1 hour and confirm no new stale active-run wrappers for idle opencode_k8s timer/system runs over the required monitoring window.

Risks

  • Medium behavioral risk: timer/system wakes for idle opencode_k8s agents will no longer create adapter runs, so the assigned-work query must match the intended live-work statuses.
  • Low regression risk for assignment-driven work: issue-assigned wakes are intentionally preserved and covered by the work-present path.
  • Operational risk: monitor expectations need to distinguish intentional no_in_flight_work skips from missing heartbeat activity.

For core feature work, check ROADMAP.md first and discuss it in #dev before opening the PR. Feature PRs that overlap with planned core work may need to be redirected — check the roadmap first. See CONTRIBUTING.md.

Model Used

  • Paperclip agent-assisted implementation and release workflow.
  • Release PR opened by OpenAI openai/gpt-5.5 with tool use. Implementation commit was authored by Paperclip automation in commit 8b0e7b68.

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have specified the model used (with version and capability details)
  • I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots
  • I have updated relevant documentation to reflect my changes
  • I have considered and documented any risks above
  • I will address all Greptile and reviewer comments before requesting merge

Linked Paperclip issue: BLO-7594

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@blockcast-ci-packages

Copy link
Copy Markdown
Author

Staff review: changes requested

The branch suppresses adapter runs, but the no-work skip path leaves the timer baseline stale.

  • In heartbeatService.tickTimers, the new opencode_k8s skip inserts an agent_wakeup_requests row and continues without updating agents.lastHeartbeatAt or coalescing the skipped wakeup.
  • The next scheduler pass still sees the same old lastHeartbeatAt, so an idle opencode_k8s agent remains immediately due and can write another no_in_flight_work row/log every tick until assigned work appears.
  • That replaces stale model-run wrappers with unbounded DB/log churn and a new persistent wakeup noise source.

Required fix:

  • Bound the skip per heartbeat interval, preferably by advancing the timer baseline when recording no_in_flight_work, or by coalescing/updating a recent skipped wakeup instead of inserting a fresh row every scheduler pass.
  • Add a regression test that calls tickTimers(now) twice with no assigned work and asserts the second call does not create another skipped wakeup until the next interval boundary.

Verification I ran:

pnpm vitest run server/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.ts

Test Files  1 passed (1)
Tests       2 passed (2)

GitHub CI is green, but the current tests only cover a single tick and miss the repeated-idle-tick failure mode.

Co-Authored-By: Paperclip <noreply@paperclip.ing>
@blockcast-ci-packages

Copy link
Copy Markdown
Author

Staff review resolved

The follow-up commit 47c80925 addresses the repeated idle tick failure mode called out above.

  • The no_in_flight_work skip path now updates agents.lastHeartbeatAt to the tick time, so the agent is not immediately due again on the next scheduler pass.
  • server/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.ts now calls tickTimers() again 10 seconds later and asserts no additional skipped wakeup row is inserted.
  • Fresh local verification: pnpm vitest run server/src/__tests__/heartbeat-opencode-k8s-timer-no-work.test.tsTest Files 1 passed (1), Tests 2 passed (2).
  • GitHub PR checks are green on head 47c80925.

PR is ready for merge from the release side.

@blockcast-ci-packages
blockcast-ci-packages Bot merged commit 0e0ebb7 into master May 26, 2026
13 checks passed
@blockcast-ci-packages

Copy link
Copy Markdown
Author

Release update

Merged via squash as 0e0ebb74f56dba6d77deb7ecb2bcf4f0d13e83cd after verifying:

  • PR was clean against master
  • 13/13 GitHub checks were green on head 47c80925
  • The repeat idle tick concern was addressed by 47c80925, including the regression check that an immediate second tickTimers() call does not create another skipped wakeup row

Paperclip QA handoff is tracked in BLO-7646 for the post-deploy no_in_flight_work monitor and stale-wrapper negative check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants