fix(ci): comment router and spam scanner fail every run after the hydrator gained new scripts

[masatohoshino]

Closes #914

What Problem This Solves

Fixes an issue where every repair comment router and spam scanner run fails at
setup-state, before any routing or scanning happens, so @clawsweeper comment commands,
waiting repair-dispatch retries, and spam triage are not processed at all.

Both jobs sparse-check-out this repository and name the hydrator scripts one by one:

scripts/hydrate-state.ts
scripts/worker-records.ts

./.github/actions/setup-state now needs two more scripts from that directory, and neither
is checked out, so the action cannot run:

• scripts/worker-blobs.ts — a static import at the top of scripts/hydrate-state.ts
  since feat(state): R2-backed ledger/v1 + assets blob transport, migration workflow, and opt-in dual-read (phase 3 foundation) #899. The blob transport is dormant by default (CLAWSWEEPER_LEDGER_SOURCE
  defaults to git), but a static import is resolved before that default is consulted, so
  the fail-safe does not cover it.
• scripts/prepare-worker-record-cache.ts — run by the Resolve Worker record snapshot cache key step whenever records-source is worker, which these two call sites became
  in feat(state): make the records-source cutover flippable via repo variable #908.

The last successful router run was
run 30297600459 at
2026-07-27T19:17:31Z on 9b4c227. c32e826 landed at 19:20:52Z and every run since
has failed: 483 failures and no successes for the router, 13 and no successes for the spam
scanner, still failing on aad59aa as of
run 30352119688
(2026-07-28T10:46:47Z).

Why This Change Was Made

The two missing paths are added to both sparse lists. Because a hand-maintained list is
exactly what drifted here, the accompanying test does not restate today's list: it reads
the scripts setup-state executes, follows their relative imports transitively
(from "./x.ts" and import("./x.ts"), either quote style), and asserts that every sparse
hydrating checkout contains that closure. It also pins the affected call sites to those two
jobs, so a third one has to be acknowledged deliberately. It is a drift guard for the
import style this repository actually uses, not a general module resolver — an
extensionless, package-relative, or re-exported dependency would still need a manual entry.

Two alternatives are deliberately not taken here, as both are product decisions: making the
blob import lazy so a dormant transport is not an import-time dependency, and dropping the
sparse checkout from these two jobs.

User Impact

Both lanes hydrate generated state and run again. Operators also stop seeing the misleading
secondary failure: when setup-state fails, setup-pnpm is skipped but the next step
still runs, reporting pnpm: command not found (exit 127) and
Could not access .../clawsweeper-state/jobs/jobs instead of the real cause.

Evidence

Production failure, first run after c32e826
(run 30298180330,
2026-07-27T19:25:32Z, head c32e826):

##[group]Run node "./scripts/hydrate-state.ts" --state-dir "clawsweeper-state" --worktree "."
    throw new ERR_MODULE_NOT_FOUND(
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '/home/runner/work/clawsweeper/clawsweeper/scripts/worker-blobs.ts' imported from /home/runner/work/clawsweeper/clawsweeper/scripts/
  code: 'ERR_MODULE_NOT_FOUND',
##[error]Process completed with exit code 1.

Production failure after b9edc42
(run 30328480253,
2026-07-28T04:20:35Z, head b9edc42). The cache step now fails first; it only runs when
records-source == 'worker', so its presence also shows the repository variable is set:

##[group]Run node "./scripts/prepare-worker-record-cache.ts"
  CLAWSWEEPER_RECORDS_URL: https://clawsweeper.openclaw.ai
  CLAWSWEEPER_RECORDS_SECRET: ***
Error: Cannot find module '/home/runner/work/clawsweeper/clawsweeper/scripts/prepare-worker-record-cache.ts'
  code: 'MODULE_NOT_FOUND',
##[error]Process completed with exit code 1.

Actions logs expire, so those lines are quoted rather than only linked.

The new test is red on the unfixed workflows and green with this patch:

# workflows reverted to b9edc42, test applied
$ node --test --test-name-pattern='sparse hydrating checkouts' test/state-writer-workflow.test.ts
ℹ pass 0
ℹ fail 1
  AssertionError [ERR_ASSERTION]: .github/workflows/repair-comment-router.yml:route-comments must check out scripts/prepare-worker-record-cache.ts

# with this patch, Node v24.18.0
$ node --test test/state-writer-workflow.test.ts
✔ sparse hydrating checkouts carry the whole setup-state script closure (56.326254ms)
✔ the rollout scans 50 and grants four concurrent size-8 preparations (0.108987ms)
ℹ fail 0

(The aggregate count that follows differs by runner aggregation mode, so only the per-test
lines and fail 0 are quoted here.)

A local reproduction materializes exactly the sparse list of repair-comment-router.yml
at b9edc42 and runs the two commands setup-state runs (Node v24.18.0):

# commit under test: b9edc42
# scripts/ in the simulated sparse checkout:
hydrate-state.ts
worker-records.ts
# A) hydrate-state.ts (runs on every setup-state call, any records-source)
Error [ERR_MODULE_NOT_FOUND]: Cannot find module '<work>/cs/scripts/worker-blobs.ts' imported from <work>/cs/scripts/hydrate-state.ts
# B) prepare-worker-record-cache.ts (runs when records-source is worker)
node:internal/modules/cjs/loader:1520
  throw err;
# C) same two commands after adding the missing closure members
{"hydrated":["records","jobs","results","ledger","notifications","assets","apply-report.json","repair-apply-report.json"],"recordsSource":"git","ledgerSource":"git",...}
if (!webhookSecret) throw new Error("CLAWSWEEPER_RECORDS_SECRET is required")

A reproduces the production error exactly. In C hydration completes and the cache
script resolves, stopping at its own missing-secret precondition.

Scope check: parsing every workflow shows these are the only two jobs that sparse-check-out
this repository and call setup-state. The closure is four scripts — hydrate-state.ts
and prepare-worker-record-cache.ts are the entry points, they reach worker-blobs.ts and
worker-records.ts, and worker-records.ts imports only Node builtins.

pnpm run check on Node v24.18.0 ran 2,899 tests with 10 failures, all of which this
host causes and none of which this patch touches. Reverting the three changed files back to
b9edc42 and re-running only the affected files reproduces the same ten by name:

# base b9edc42, same host
node --test test/repair/git-publish.test.ts test/repair/state-publication-batch.test.ts test/repair/target-validation.test.ts
✖ publishMainCommit continues after a bounded fetch restores an unavailable object
✖ publishMainCommit fails fast when unavailable ledger objects remain missing after recovery
✖ publishMainCommit escapes sustained immutable ledger races through a server merge
✖ deletion index records use the SHA-256 repository object width
✖ bun dependency setup rejects and reaps detached descendants
✖ npm dependency setup rejects and reaps detached descendants
✖ replacement branch plumbing bypasses checkout and reference hooks
✖ replacement branch plumbing restores overwritten refs when HEAD cannot switch
✖ replacement branch plumbing rejects branches attached to another worktree
✖ validation rejects and reaps an immediate detached double fork
ℹ tests 257  ℹ pass 247  ℹ fail 10

The causes are local: this host has Git 2.34.1, so Git must support --no-lazy-fetch for bounded missing-object recovery fails the ledger recovery paths, and the bun/npm
containment fixtures are absent. Everything else, including the whole workflow suite,
passed. CI is authoritative for these files.

Proof summary

• Behavior addressed: setup-state aborts in the two sparse hydrating jobs because
  scripts/worker-blobs.ts and scripts/prepare-worker-record-cache.ts are not checked
  out, failing every repair comment router and spam scanner run.

• Real environment tested: production GitHub Actions runs on c32e826 and b9edc42
  for the failure; a local Node v24.18.0 reproduction of the exact sparse file set for
  before/after.

• Exact steps or command run after this patch:

  # 1. materialize exactly the sparse list of repair-comment-router.yml at b9edc42
  #    into an empty directory, then run the two commands setup-state runs:
  node scripts/hydrate-state.ts --state-dir ../state --worktree .
  node scripts/prepare-worker-record-cache.ts
  #    repeat after copying scripts/worker-blobs.ts and
  #    scripts/prepare-worker-record-cache.ts into scripts/
  # 2. the derived-closure regression test
  node --test test/state-writer-workflow.test.ts

• Evidence after fix: the reproduction block above — hydration returns its
  {"hydrated":[...],"recordsSource":"git",...} result instead of ERR_MODULE_NOT_FOUND,
  and the cache script resolves and stops at its own missing-secret precondition.

• Observed result after fix: the test block above —
  sparse hydrating checkouts carry the whole setup-state script closure passes with
  fail 0, and fails by name when the workflows are reverted.

• What was not tested: no secret was exported and no request reached the Worker, so a
  real records-source=worker hydration is not demonstrated end to end. What is
  demonstrated is module resolution plus the git-source hydration path; the outage itself
  rests on the two production run logs above. The ten unrelated local pnpm run check
  failures are host-caused and reproduce on the base commit, as shown above.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed July 29, 2026, 6:08 PM ET / 22:08 UTC.

ClawSweeper review

What this changes

The PR adds setup-state’s two missing script dependencies to the sparse checkouts used by comment routing and spam scanning, then tests that these checkouts include the action’s relative-import closure.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

This remains necessary: it is the open candidate fix for the comment-router and spam-scanner outage tracked in #914. The patch and its supplied before/after production and sparse-checkout proof are strong, but GitHub reports the branch as conflicted with current main; resolve that conflict and refresh the focused proof/review on the rebased head before merge.

Priority: P1
Reviewed head: 43a4a72854df2f7dfe98dc5cd67f51412eb7be58

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	The repair is narrowly scoped and strongly evidenced, but cannot land until its conflicted branch is rebased and the proof is refreshed for that exact head.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The PR provides detailed after-fix terminal evidence for the exact sparse checkout plus focused test evidence, anchored by production Actions failure logs; fresh equivalent proof is still required after the conflict is resolved. Any new logs should continue to redact secrets and non-public endpoints.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The PR provides detailed after-fix terminal evidence for the exact sparse checkout plus focused test evidence, anchored by production Actions failure logs; fresh equivalent proof is still required after the conflict is resolved. Any new logs should continue to redact secrets and non-public endpoints.
Evidence reviewed	5 items	Affected workflow sparse lists: The proposed diff adds scripts/prepare-worker-record-cache.ts and scripts/worker-blobs.ts to both sparse checkouts, matching the two scripts the shared state setup path needs beyond the already-listed hydrator and records helper. Regression coverage: The added workflow test derives the required script closure from .github/actions/setup-state/action.yml, follows the repository’s relative script imports, and pins the two sparse hydrating jobs so future call sites require an explicit update. Feature provenance: The reported dependency boundary is consistent with the merged state-blob transport change that introduced scripts/worker-blobs.ts, followed by the records-source change that made the cache script run for these worker-backed call sites.
Findings	None	None.
Security	None	None.

How this fits together

The comment router and spam scanner are scheduled GitHub Actions lanes that sparse-check out the ClawSweeper repository, hydrate operational state through the shared setup-state action, then route commands or scan reports. The shared action invokes state scripts, so each sparse checkout must include the scripts and local imports needed before either lane can process work.

flowchart LR
  A[Scheduled or comment dispatch] --> B[Comment router or spam scanner]
  B --> C[Sparse repository checkout]
  C --> D[Shared state setup]
  D --> E[Hydrator and cache scripts]
  E --> F[State hydration]
  F --> G[Routing or spam triage output]

Loading

Before merge

• Resolve merge risk (P1) - GitHub currently reports a merge conflict with main; resolving it could alter the sparse lists or the closure test, so the supplied proof and prior review are not sufficient for the eventual merged head.
• Resolve merge risk (P1) - The change repairs production automation inputs, so an incorrect conflict resolution could leave either the comment router or spam scanner without a required state-setup script.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Patch surface	3 files affected: 4 workflow entries added and 54 test lines added	The operational change is narrow, while the regression test covers both known sparse hydrating jobs.

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #914
Summary: This PR is the candidate fix for the open sparse-checkout outage report.

Members:

• canonical: repair comment router and spam scanner fail every run because their sparse checkouts miss the hydrator's new script dependencies #914 - The issue and PR identify the same two workflows, the same missing setup-state scripts, and the same proposed closure-based regression guard.
• same_root_cause: feat(state): R2-backed ledger/v1 + assets blob transport, migration workflow, and opt-in dual-read (phase 3 foundation) #899 - Its merged state-blob work introduced the statically imported blob helper that the sparse lists must now include.
• same_root_cause: feat(state): make the records-source cutover flippable via repo variable #908 - Its merged records-source rollout made the worker cache preparation step relevant to these two workflow call sites.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge-risk options

Maintainer options:

1. Rebase and re-prove the repaired workflows (recommended)
   Resolve the current conflict against main, rerun the focused sparse-checkout test, and attach a fresh successful router or scanner run for the rebased head.
2. Pause the PR if the workflow contract changed
   If conflict resolution shows that current main uses a different state-setup or checkout contract, pause this branch and create a narrow follow-up from the current workflow definitions.

Technical review

Best possible solution:

Rebase onto current main, retain the two required script entries in both workflow sparse lists, resolve any closure-test conflict without weakening its coverage, then rerun the focused workflow test and capture a successful real workflow run before merge.

Do we have a high-confidence way to reproduce the issue?

Yes, from source and supplied runtime evidence: materializing the old sparse lists omits statically imported state scripts and produces the documented module-resolution failure before routing or scanning begins. The PR body also shows the focused regression test failing when the workflow is reverted and passing with the added closure members.

Is this the best way to solve the issue?

Yes, conditionally: adding the missing scripts to the two affected sparse lists and testing the shared action’s current relative-import closure is the narrowest maintainable repair. The branch must first be rebased because its conflicted merge result has not been proven.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 8365a79af804.

Labels

Label justifications:

• P1: Both the scheduled spam scanner and the comment-command router are reported to fail before processing any work.
• merge-risk: 🚨 automation: The patch changes the sparse checkout inputs required by production GitHub Actions state setup.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR provides detailed after-fix terminal evidence for the exact sparse checkout plus focused test evidence, anchored by production Actions failure logs; fresh equivalent proof is still required after the conflict is resolved. Any new logs should continue to redact secrets and non-public endpoints.
• proof: sufficient: Contributor real behavior proof is sufficient. The PR provides detailed after-fix terminal evidence for the exact sparse checkout plus focused test evidence, anchored by production Actions failure logs; fresh equivalent proof is still required after the conflict is resolved. Any new logs should continue to redact secrets and non-public endpoints.

Evidence

What I checked:

• Affected workflow sparse lists: The proposed diff adds scripts/prepare-worker-record-cache.ts and scripts/worker-blobs.ts to both sparse checkouts, matching the two scripts the shared state setup path needs beyond the already-listed hydrator and records helper. (.github/workflows/repair-comment-router.yml:103, 43a4a72854df)
• Regression coverage: The added workflow test derives the required script closure from .github/actions/setup-state/action.yml, follows the repository’s relative script imports, and pins the two sparse hydrating jobs so future call sites require an explicit update. (test/state-writer-workflow.test.ts:91, 43a4a72854df)
• Feature provenance: The reported dependency boundary is consistent with the merged state-blob transport change that introduced scripts/worker-blobs.ts, followed by the records-source change that made the cache script run for these worker-backed call sites. (scripts/hydrate-state.ts:1, b9edc427bbcd)
• Current landing state: GitHub reports this PR as mergeable: false with mergeableState: dirty against current main at 8365a79, so the existing proof cannot establish the resolved merge result. (.github/workflows/repair-comment-router.yml:103, 8365a79af804)
• Real behavior proof: The PR body supplies production failure evidence plus a before/after local reproduction of the exact sparse script set, where hydration changes from ERR_MODULE_NOT_FOUND to a completed hydration result after the missing scripts are included; it also supplies a focused regression-test failure on the reverted workflow and pass on the patch. (test/state-writer-workflow.test.ts:91, 43a4a72854df)

Likely related people:

• steipete: The merged state-blob and records-source work introduced the two state-setup dependencies and worker-backed call-site behavior that this repair reconciles with sparse checkout. (role: related feature introducer; confidence: high; commits: c32e826494fd, b9edc427bbcd; files: scripts/hydrate-state.ts, scripts/worker-blobs.ts, .github/actions/setup-state/action.yml)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)

• reviewed 2026-07-28T11:16:37.564Z sha 43a4a72 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T19:09:37.583Z sha 43a4a72 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T20:11:53.872Z sha 43a4a72 :: needs maintainer review before merge. :: none
• reviewed 2026-07-29T21:08:40.357Z sha 43a4a72 :: needs maintainer review before merge. :: none

[masatohoshino]

Superseded by b3ed1bf, which added the same two sparse-checkout entries to both workflows, so the diff here is a no-op against current main. The hydrating checkout now completes; the failures still on this lane come from a later step and are reported separately in #946.