Skip to content

Don't reap executions that are waiting on a child - #867

Merged
AbirAbbas merged 1 commit into
mainfrom
fix/dont-reap-executions-awaiting-children
Aug 4, 2026
Merged

Don't reap executions that are waiting on a child#867
AbirAbbas merged 1 commit into
mainfrom
fix/dont-reap-executions-awaiting-children

Conversation

@AbirAbbas

Copy link
Copy Markdown
Contributor

An execution's updated_at stops moving while it waits for a child to return, so the staleness sweep read "blocked on a child" as "stuck". Any agent that does many minutes of work inside a single request had its whole ancestor chain marked timed out mid-flight — the caller got execution timed out (no activity) while the work was still running, and the agent went on to finish normally against a run already reported as failed.

Observed on a real build: the root execution was marked failed at 20:15:46, and the agent doing the work kept emitting progress until 20:46 and beyond. A comparable build whose work was split across many short reasoner calls succeeded in the same window — each call bumped updated_at, so it never looked idle.

This is the invariant the code already documents ("callers must ensure updated_at is bumped on every meaningful execution activity") meeting a caller that legitimately cannot bump it, because it is blocked.

Change

Rows with a non-terminal child are skipped by both sweeps (MarkStaleExecutions and MarkStaleWorkflowExecutions had the same shape). parent_execution_id already exists and is indexed on both tables.

There is deliberately no recency test on the child, which is what makes the rule safe for arbitrary nesting depth: the chain drains bottom-up. If work genuinely stops, the leaf goes stale and is reaped first, which makes its parent childless and eligible on the next sweep, and so on up to the root. One sweep per level, and nothing is stranded in running. Requiring the child to be recently active instead would only protect one level and would still reap grandparents.

Tests

internal/storage/stale_execution_parent_test.go covers:

  • a three-level chain where only the leaf is working — nothing is reaped
  • the same chain fully idle — drains one level per sweep, leaf first
  • a terminal child does not shield a genuinely stuck parent
  • an unrelated stuck execution is still reaped in the same sweep as a protected chain

Both new behavioural tests were confirmed to fail with the fix reverted, so they pin the behaviour rather than the implementation.

go build ./... and the full ./internal/storage/... suite pass, existing reaper tests included.

🤖 Generated with Claude Code

An execution's updated_at stops moving while it waits for a child to
return, so the staleness sweep read 'blocked on a child' as 'stuck'. An
agent doing many minutes of work inside a single request had its whole
ancestor chain marked timed out mid-flight: the caller was told
'execution timed out (no activity)' while the work was still running and
went on to finish.

Rows with a non-terminal child are now skipped. There is deliberately no
recency test on the child, so the chain still drains when work genuinely
stops — the leaf goes stale first, which makes its parent childless and
eligible on the next sweep, and so on up. One sweep per level, and nothing
is stranded in running.

Applies to both the execution and workflow-execution sweeps, which had the
same shape.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AbirAbbas
AbirAbbas requested a review from a team as a code owner August 3, 2026 21:31
@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

📊 Coverage gate

Thresholds from .coverage-gate.toml: per-surface ≥ 84%, aggregate ≥ 85%, max per-surface regression ≤ 1.0 pp, max aggregate regression ≤ 0.50 pp.

Surface Current Baseline Δ
control-plane 87.00% 87.40% ↓ -0.40 pp 🟡
sdk-go 92.70% 92.00% ↑ +0.70 pp 🟢
sdk-python 93.82% 93.73% ↑ +0.09 pp 🟢
sdk-typescript 91.05% 90.42% ↑ +0.63 pp 🟢
web-ui 84.76% 84.79% ↓ -0.03 pp 🟡
aggregate 85.59% 85.75% ↓ -0.16 pp 🟡

✅ Gate passed

No surface regressed past the allowed threshold and the aggregate stayed above the floor.

@github-actions

github-actions Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

📐 Patch coverage gate

Threshold: 80% on lines this PR touches vs origin/main (from .coverage-gate.toml:thresholds.min_patch).

Surface Touched lines Patch coverage Status
control-plane 12 100.00%
sdk-go 0 ➖ no changes
sdk-python 0 ➖ no changes
sdk-typescript 0 ➖ no changes
web-ui 0 ➖ no changes

✅ Patch gate passed

Every surface whose lines were touched by this PR has patch coverage at or above the threshold.

@AbirAbbas
AbirAbbas merged commit 22d65d1 into main Aug 4, 2026
30 checks passed
@AbirAbbas
AbirAbbas deleted the fix/dont-reap-executions-awaiting-children branch August 4, 2026 01:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant