fix(ci): raise shard watchdog floor to 15min to stop Full Suite false-kills - #1695
Merged
Conversation
Align workflow switcher status count badges with their board column colors. - Map Todo, In Progress, and Done workflow count badge text to the matching board column CSS tokens. - Cover the badge color contract with a CSS fixture regression test. - Document the color-token behavior for board and list workflow dropdown counts. Files changed: docs/dashboard-guide.md | 2 +- .../dashboard/app/components/WorkflowSwitcher.css | 10 +++++++--- .../components/__tests__/WorkflowSwitcher.test.tsx | 22 ++++++++++++++++++++++ 3 files changed, 30 insertions(+), 4 deletions(-) Fusion-Task-Id: FN-6805 Fusion-Task-Lineage: aab7140c-ad37-4bd8-b73d-199534285b37
Keep the experimental left sidebar Settings control clear of the fixed executor footer while preserving readable project selector fallbacks. - Share the executor footer visibility state with the left sidebar and project content. - Reserve executor-footer height on the sidebar when the status footer is visible so Settings stays clickable in expanded and collapsed modes. - Add project-selector translation fallbacks for incomplete locale fixtures. - Cover footer-clearance behavior in left sidebar tests and add a patch changeset. Files changed: .changeset/fn-6819-sidebar-footer-clearance.md | 5 +++ packages/dashboard/app/App.tsx | 8 +++-- .../dashboard/app/components/LeftSidebarNav.css | 9 ++++++ .../dashboard/app/components/LeftSidebarNav.tsx | 4 ++- .../dashboard/app/components/ProjectSelector.css | 2 +- .../dashboard/app/components/ProjectSelector.tsx | 11 +++++-- .../components/__tests__/LeftSidebarNav.test.tsx | 36 ++++++++++++++++++++++ 7 files changed, 67 insertions(+), 8 deletions(-) Fusion-Task-Id: FN-6819 Fusion-Task-Lineage: 950da852-8dcf-4379-a91b-0188b12d1a6c
Resolve synchronous AI CLI probe launch failures as unavailable auth states. - Catch synchronous spawn failures in Claude and Droid probe helpers so fire-and-forget validation paths do not reject. - Return unavailable/unauthenticated sentinels for Droid model discovery and CLI presence/auth checks. - Add regression coverage for Claude and Droid presence/auth probes plus a patch changeset. Files changed: .../fn-6808-cli-probe-unhandled-rejection.md | 5 ++++ .../src/__tests__/process-manager.test.ts | 25 ++++++++++++++++++++ .../src/__tests__/process-manager.test.ts | 25 ++++++++++++++++++++ packages/pi-claude-cli/src/process-manager.ts | 12 +++++++++- .../src/process-manager.ts | 27 +++++++++++++++++++--- 5 files changed, 90 insertions(+), 4 deletions(-) Fusion-Task-Id: FN-6808 Fusion-Task-Lineage: 79702ef2-e116-4313-a557-da17a6888c30
The Full Suite (non-blocking) workflow has been red for 30+ runs on main. Diagnosis: the @fusion/engine [1/2], [2/2] and @fusion/core [2/2] shard slices were SIGKILLed at their watchdog budgets (405s/405s/338s), not because they hang but because those budgets are too tight for current wall-clock. Local baselines (this machine, all pass, exit 0): - engine [2/2]: 145s wall / 309 files - core [2/2]: 283s wall / 172 files (old budget was only 338s!) deriveBudgetMs tightens the budget to expected*3.5 whenever the committed scripts/test-timings.json is <30d old. The snapshot (2026-06-03) undercounts the import- and real-git-subprocess overhead of these heavy slices, so the 'fresh' snapshot produced a too-tight, false-kill budget on slower CI runners -- the exact failure mode the floor/ceiling band exists to prevent. Fix (plan KTD-2): raise the shard band floor 5min -> 15min so the heaviest slices can't be tightened into a false-kill, while a true hang is still bounded far under the job's 60min ceiling. Mirrors the dashboard-lane heavy-lane floor. Follow-up: refresh scripts/test-timings.json from a default-branch CI run.
- Reformat shard-floor justification as an FNXC:TestInfrastructure comment (project-standards: AGENTS.md FNXC_LOG convention). - Clarify that the shard and dashboard-lane 15min floors are not coupled and may diverge (maintainability: avoid implying an unenforced contract). - Add a regression-guard test pinning shard.floor=15min and asserting a 525s derived budget clamps up to the floor, so an accidental revert to the old 5min floor fails loudly (correctness + testing + project-standards).
Contributor
📝 WalkthroughWalkthroughThree independent patches: CLI probes for Droid and Claude now wrap ChangesCLI Probe Spawn Error Handling
Dashboard UI Fixes
CI Watchdog Shard Floor and Hang Plan
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Ready to review this PR? Stage has broken it down into 5 individual chapters for you: 
Chapters generated by Stage for commit 9af9bf4 on Jun 21, 2026 4:54am UTC. |
Contributor
Greptile SummaryThis PR raises the CI shard watchdog floor from 5 minutes to 15 minutes in
Confidence Score: 4/5Safe to merge — the watchdog floor change is clearly justified, well-tested with a concrete regression guard, and the spawn-guard fixes are minimal and well-scoped. All changed code paths are exercised by updated or new tests. The core watchdog change is a single numeric constant guarded by a pinned regression test. The spawn-guard additions are mechanical and match the existing error-sentinel convention. The only loose end is the plan document whose 'active' status may mislead future contributors, but this does not affect runtime behavior. docs/plans/2026-06-21-001-fix-full-suite-engine-core-hang-plan.md — status field and KTD-1 language are in tension with what was actually implemented; worth updating before the follow-up work begins. Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A["deriveBudgetMs(klass, expectedDurationMs, timingsFresh)"] --> B{timingsFresh AND\nexpectedDurationMs > 0?}
B -- No --> C["return band.ceiling\n(generous fallback)"]
B -- Yes --> D["derived = round(expectedDurationMs × 3.5)"]
D --> E{derived < band.floor?}
E -- Yes --> F["return band.floor\n(shard: 15min ← was 5min)"]
E -- No --> G{derived > band.ceiling?}
G -- Yes --> H["return band.ceiling\n(shard: 30min)"]
G -- No --> I["return derived\n(tightened value)"]
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
flowchart TD
A["deriveBudgetMs(klass, expectedDurationMs, timingsFresh)"] --> B{timingsFresh AND\nexpectedDurationMs > 0?}
B -- No --> C["return band.ceiling\n(generous fallback)"]
B -- Yes --> D["derived = round(expectedDurationMs × 3.5)"]
D --> E{derived < band.floor?}
E -- Yes --> F["return band.floor\n(shard: 15min ← was 5min)"]
E -- No --> G{derived > band.ceiling?}
G -- Yes --> H["return band.ceiling\n(shard: 30min)"]
G -- No --> I["return derived\n(tightened value)"]
Reviews (1): Last reviewed commit: "docs(plans): add Full Suite engine/core ..." | Re-trigger Greptile |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| </div> | ||
| </div> | ||
| {viewMode === "project" && currentProject && ( | ||
| {executorFooterVisible && currentProject && ( |
Contributor
There was a problem hiding this comment.
&& currentProject guard is redundant here — executorFooterVisible is defined as viewMode === "project" && !!currentProject, so it can only be true when currentProject is already truthy. The double-check adds no protection and may mislead readers into thinking currentProject could be falsy at this point.
Suggested change
| {executorFooterVisible && currentProject && ( | |
| {executorFooterVisible && ( |
Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!
Comment on lines
+120
to
+126
| --- | ||
|
|
||
| ## Implementation Units | ||
|
|
||
| ### U1. Reproduce the hang locally and isolate the leaking test(s) | ||
|
|
||
| **Goal:** Turn the CI-only failure into a deterministic local repro and name the |
Contributor
There was a problem hiding this comment.
The document's front-matter sets status: active and KTD-1 explicitly says "raising the watchdog budget would mask a real open-handle leak." This PR implements only the KTD-2 budget-floor raise (the stated last resort), with U1–U3 left for follow-up work. Future contributors reading this plan may be confused about whether the open-handle investigation is still expected or was superseded. Consider setting status: deferred or adding a short note at the top clarifying that KTD-2 was applied as the immediate fix and the hang-diagnosis work is tracked separately.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
plugins/fusion-plugin-droid-runtime/src/process-manager.ts (1)
294-303: ⚡ Quick winAdd an explicit regression test for
discoverDroidModelssynchronousspawnthrows.This branch was added here, but the updated tests only cover presence/auth sync-throw behavior. Add a focused test that makes one discovery attempt throw synchronously and asserts fallback/empty behavior remains non-rejecting.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/fusion-plugin-droid-runtime/src/process-manager.ts` around lines 294 - 303, Add a focused regression test for the discoverDroidModels function that verifies when the spawn call for the "droid" process throws synchronously, the function properly handles the error in the try-catch block and continues execution without rejecting. The test should mock or configure spawn to throw synchronously during one discovery attempt, then assert that the function still completes successfully with appropriate fallback behavior (returning null or an empty result), ensuring the error handling in the loop at the specified location does not cause unexpected rejections.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/plans/2026-06-21-001-fix-full-suite-engine-core-hang-plan.md`:
- Around line 103-115: The fenced code block in the document is missing a
language specifier after the opening triple backticks, which violates
markdownlint rule MD040. Add a language specifier such as "text" immediately
after the opening ``` on the code block that contains the ASCII diagram showing
the CI shard flow and process hang issue. This applies to the block that starts
with "CI shard → pnpm --filter..." and should be changed from ``` to ```text.
---
Nitpick comments:
In `@plugins/fusion-plugin-droid-runtime/src/process-manager.ts`:
- Around line 294-303: Add a focused regression test for the discoverDroidModels
function that verifies when the spawn call for the "droid" process throws
synchronously, the function properly handles the error in the try-catch block
and continues execution without rejecting. The test should mock or configure
spawn to throw synchronously during one discovery attempt, then assert that the
function still completes successfully with appropriate fallback behavior
(returning null or an empty result), ensuring the error handling in the loop at
the specified location does not cause unexpected rejections.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 179369be-3baa-4b5a-9ec1-9c6b5a07908a
📒 Files selected for processing (18)
.changeset/fn-6808-cli-probe-unhandled-rejection.md.changeset/fn-6819-sidebar-footer-clearance.mddocs/dashboard-guide.mddocs/plans/2026-06-21-001-fix-full-suite-engine-core-hang-plan.mdpackages/dashboard/app/App.tsxpackages/dashboard/app/components/LeftSidebarNav.csspackages/dashboard/app/components/LeftSidebarNav.tsxpackages/dashboard/app/components/ProjectSelector.csspackages/dashboard/app/components/ProjectSelector.tsxpackages/dashboard/app/components/WorkflowSwitcher.csspackages/dashboard/app/components/__tests__/LeftSidebarNav.test.tsxpackages/dashboard/app/components/__tests__/WorkflowSwitcher.test.tsxpackages/droid-cli/src/__tests__/process-manager.test.tspackages/pi-claude-cli/src/__tests__/process-manager.test.tspackages/pi-claude-cli/src/process-manager.tsplugins/fusion-plugin-droid-runtime/src/process-manager.tsscripts/__tests__/run-vitest-watchdog.test.mjsscripts/lib/run-vitest-watchdog.mjs
Comment on lines
+103
to
+115
| ``` | ||
| CI shard → pnpm --filter @fusion/engine exec vitest run --project=… | ||
| │ | ||
| ├── all test bodies pass ✅ (Test Files summary never printed) | ||
| │ | ||
| └── process does NOT exit ❌ | ||
| │ leaked handle keeps event loop alive: | ||
| │ • spawned git/worktree child process not awaited/killed | ||
| │ • setInterval / heartbeat timer not cleared in teardown | ||
| │ • worktree-pool / db handle not closed | ||
| ▼ | ||
| watchdog budget reached → SIGTERM/SIGKILL → exit≠0 → shard FAIL | ||
| ``` |
Contributor
There was a problem hiding this comment.
Add a language to the fenced block to satisfy markdownlint MD040.
The fenced block is missing a language specifier; this can fail markdown lint checks.
Suggested fix
-```
+```text
CI shard → pnpm --filter `@fusion/engine` exec vitest run --project=…
│
├── all test bodies pass ✅ (Test Files summary never printed)
@@
watchdog budget reached → SIGTERM/SIGKILL → exit≠0 → shard FAIL
-```
+```🧰 Tools
🪛 markdownlint-cli2 (0.22.1)
[warning] 103-103: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/plans/2026-06-21-001-fix-full-suite-engine-core-hang-plan.md` around
lines 103 - 115, The fenced code block in the document is missing a language
specifier after the opening triple backticks, which violates markdownlint rule
MD040. Add a language specifier such as "text" immediately after the opening ```
on the code block that contains the ASCII diagram showing the CI shard flow and
process hang issue. This applies to the block that starts with "CI shard → pnpm
--filter..." and should be changed from ``` to ```text.
Source: Linters/SAST tools
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
The Full Suite (non-blocking) workflow has been red on
mainfor 30+ consecutive runs. The failure is not a test assertion — every test that reports a result passes. Three sharded vitest groups never finish within their wall-clock watchdog budget and get SIGKILLed:@fusion/engine [1/2]@fusion/engine [2/2]@fusion/core [2/2]@fusion/core [1/2]It is not a hang. Both failing groups run to completion and exit cleanly when measured locally:
@fusion/engine [2/2]→ 145s,Test Files 309 passed, exit 0@fusion/core [2/2]→ 283s,Test Files 172 passed, exit 0A clean exit rules out the open-handle/leak hypothesis — these slices are genuinely slow, not wedged.
Root cause
deriveBudgetMstightens the watchdog budget toexpected × 3.5wheneverscripts/test-timings.jsonis younger than the 30-day staleness window. The committed snapshot (2026-06-03, 18 days old → still "fresh") undercounts these import- and real-git-subprocess-heavy slices — 368 of 520 engine files sit at the 100ms bucket floor, so the snapshot total is far below current wall-clock. The result was a too-tight, false-kill budget (340–405s) that the floor/ceiling band exists precisely to prevent.core[2/2]needs 283s locally against a 338s budget — only 1.2× headroom even on a fast machine, guaranteeing failure on slower CI runners.The fix
Raise the
shardbudget-band floor from 5min to 15min (scripts/lib/run-vitest-watchdog.mjs). Heavy slices can no longer be tightened into a false-kill, while a genuine hang is still bounded far under the job's 60min ceiling. This follows the watchdog's own KTD-2 ("adjust the budget only once a group is proven slow-not-hung, with measured numbers") and the predecessor plandocs/plans/2026-06-13-001-fix-test-timeout-failures-plan.md, which flagged the 5min floor as provisional.New budgets: engine/core slices → 900s (was 405s/338s); a genuinely huge slice still caps at the 30min ceiling; a true infinite hang is still killed.
Tests
deriveBudgetMs"tightens within band" test input so it still exercises the un-clamped path under the new floor.shard.floor = 15minand asserting a 525s derived budget clamps up to the floor — an accidental revert to 5min now fails loudly.node --test scripts/__tests__/run-vitest-watchdog.test.mjs→ 14/14 pass.check-no-test-timeout-appeasementandcheck-no-kill-4040guards both pass — this raises an outer wall-clock watchdog, not a per-test assertion timeout, so it is not appeasement.Follow-up (not in this PR)
Refresh
scripts/test-timings.jsonfrom a default-branch CI run so per-slice budgets reflect current durations; consider a dedicated heavy-shard band so light shards keep tighter hang protection.🤖 Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Documentation
Tests