Skip to content

dev server: survive watch-mode rebuild failures - #8

Merged
lolipopshock merged 2 commits into
mainfrom
fix/dev-server-watch
Jul 14, 2026
Merged

dev server: survive watch-mode rebuild failures#8
lolipopshock merged 2 commits into
mainfrom
fix/dev-server-watch

Conversation

@lolipopshock

Copy link
Copy Markdown
Contributor

Summary

Fifth PR in the migration-fix series (independent of the others). The dev server died whenever a watch-mode rebuild hit a validation error: the content watcher fires on transient states (half-swapped chapter sets, a _meta.json entry whose file isn't written yet), strict validation throws, and rebuild()'s voided promise turned that into an unhandled rejection that killed the process — reproduced during a multi-file chapter swap on a real project. The held build mutex would also have deadlocked later rebuilds.

rebuild() now wraps its body in try/catch/finally: report, keep serving the last good build, consume the batch's change events (paths stay in changesSinceLastBuild, so the next change event retries), always release the mutex.

Verification

  • npm run check passes.
  • Live reproduction: with the patched server running, moving a demo chapter file away logged Rebuild failed — still serving the last good build, the site kept answering HTTP 200, and restoring the file produced a clean recovery rebuild — process alive throughout. Same scenario killed the unpatched server.

🤖 Generated with Claude Code

The content watcher fires on transient states — a half-swapped chapter set, a
_meta.json entry whose file isn't on disk yet — and the strict validators
(navigation completeness, frontmatter titles) throw on them. rebuild() had no
error handling: the voided promise turned the throw into an unhandled
rejection and killed the dev server mid-session (reproduced during a
multi-file chapter swap on a real project), and the held mutex would have
deadlocked any later rebuild even if the process had survived.

rebuild() now wraps its body in try/catch/finally: on failure it reports the
error, keeps serving the last good build, consumes this batch's change events
(the paths stay recorded in changesSinceLastBuild, so the next change event
retries them), and always releases the build mutex.

Verified live: broke the demo nav mid-serve (moved a chapter file away) — the
server logged 'Rebuild failed — still serving the last good build', kept
answering with HTTP 200, and recovered with a clean rebuild when the file
came back.

Co-Authored-By: Shannon's Claude <257597027+shannonshen49@users.noreply.github.com>
@sepo-agent-app

Copy link
Copy Markdown

Do you want Sepo to review this PR?

Add the agent/review label or comment @sepo-agent /review to launch an agent review.
For a preview deployment, add the sepo-preview label.

Comment thread quartz/build.ts
`Emitted ${emittedFiles} files to \`${argv.output}\` in ${perf.timeSince("rebuild")}`,
)
console.log(styleText("green", `Done rebuilding in ${perf.timeSince()}`))
changes.splice(0, numChangesInBuild)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes.splice(0, numChangesInBuild) also runs in the catch below. If clientRefresh() throws after this success-path splice, control enters the catch and splices another batch from changes, which can drop queued change events. Consider moving the shared splice into finally next to release() so the queue is consumed exactly once on both success and failure.

@sepo-agent-app

This comment has been minimized.

@lolipopshock

Copy link
Copy Markdown
Contributor Author

Re regression coverage for the recovery path: agreed as follow-up — it needs a small integration harness (spawn the dev server, mutate content, assert liveness), which doesn't fit the unit suite. The manual reproduction procedure is documented in the PR body; happy to add the harness in a separate PR.

@sepo-agent-app

Copy link
Copy Markdown

AI Review Synthesis

Dual-agent review by Claude and Codex.
Requested by @lolipopshock.

Summary of PR/Issue

PR #8 makes the dev server survive watch-mode rebuild failures instead of crashing on transient invalid content states. The change wraps rebuild() in try/catch/finally, logs failures, keeps serving the last successful output, and always releases the build mutex. The PR changes only quartz/build.ts. The reported manual verification shows the dev server staying alive through a broken chapter state and recovering after the files are restored.

Review

The core fix is correct and narrowly scoped. Both reviewer artifacts agree on one concrete remaining warning: the watcher queue can be over-consumed if clientRefresh() throws after a successful rebuild.

Issue Severity Description
Queue events can be over-consumed if clientRefresh throws WARNING changes.splice(0, numChangesInBuild) runs before clientRefresh() and again in catch, so a refresh throw can drop queued watcher events.
Recovery guarantee has non-blocking limits INFO Parser-stage process.exit(1) paths and mid-emitter writes can still bypass or weaken the “last good build” guarantee, but these are outside the reported docs-nav validation failure path.
Failed-rebuild recovery lacks automated regression coverage INFO The recovery path is manually verified, and the author acknowledged the needed integration harness as follow-up rather than in-scope for this PR.

Progress

  • No new inline comments were posted in this synthesis pass.
  • An existing same-agent inline comment already covers the queue-consumption issue on quartz/build.ts:393; the PR head is still 28eebb95755308b95c8b1500c9c45607b1eae9a9, so the comment remains current.
  • The existing inline thread is unresolved and viewerCanResolve is false, so no thread resolution was attempted.
  • The author already acknowledged automated regression coverage as a separate follow-up.

Issue Details

Queue events can be over-consumed if clientRefresh throws

Cause: The success path splices the current batch at quartz/build.ts#L393, then calls clientRefresh(). If clientRefresh() throws, execution enters catch, which splices the same batch size again.

Candidate solutions: Move the shared changes.splice(0, numChangesInBuild) into finally next to release(), or guard queue consumption so each rebuild attempt consumes its batch exactly once.

Comments: This is low probability and still better than the previous crash, but it is concrete branch-change work and already has inline feedback.

Recovery guarantee has non-blocking limits

Cause: One reviewer noted that parser worker errors can still exit the process, and another noted that emitter failures may happen after partial output writes. The new catch covers the reported docs-navigation validation path, but not every possible rebuild failure mode.

Candidate solutions: Treat these as follow-up hardening only if maintainers want the guarantee to cover arbitrary parser and emitter failures.

Comments: This is informational and should not drive automated fix-pr work for this PR.

Failed-rebuild recovery lacks automated regression coverage

Cause: The behavior spans watcher batching, retained changesSinceLastBuild, skipped refresh on failure, mutex release, and later recovery, which is not covered by the current tests.

Candidate solutions: Add a future integration harness that spawns the dev server, mutates content into an invalid state, asserts liveness, then restores content and checks recovery.

Comments: The author already agreed this belongs in a separate follow-up PR.

Recommended Next Step

FIX_PR: the duplicated queue splice is a concrete, safe branch change suitable for an automated fix-pr pass.

Final Verdict

MINOR_ISSUES

Action Items

  • Ensure changes.splice(0, numChangesInBuild) runs exactly once per rebuild attempt, including when clientRefresh() throws.

codex | gpt-5.5/xhigh | Macmini-runner-4

…hrows

Review finding: changes.splice(0, numChangesInBuild) ran on the success path
and again in catch, so a clientRefresh() throw after a successful rebuild
consumed the batch twice — dropping unrelated queued watcher events. The
catch-path splice is now guarded by a batchConsumed flag.

Co-Authored-By: Shannon's Claude <257597027+shannonshen49@users.noreply.github.com>
@lolipopshock

Copy link
Copy Markdown
Contributor Author

Round-2 finding addressed in 4632f6b: the catch-path splice is guarded by a batchConsumed flag, so a clientRefresh() throw after a successful rebuild no longer consumes the queue twice. The parser-stage process.exit(1) paths and the integration harness remain follow-ups tracked in #15.

@lolipopshock
lolipopshock merged commit 2be738a into main Jul 14, 2026
3 checks passed
@sepo-agent-app

Copy link
Copy Markdown

Rubrics Update

No changes were committed to agent/rubrics from PR #8.

no rubric changes

I inspected PR #8’s body, issue comments, review comments, reviews, linked follow-up issue #15, collaborator/owner signals, and existing rubric locations. There is no existing rubrics/ directory, and the PR conversation did not contain trusted maintainer/user feedback that warrants a durable rubric. The substantive review guidance came from the bot, and the human comments were PR-specific follow-up tracking rather than reusable team preference.


codex | gpt-5.5/xhigh | Macmini-runner-2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant