fix(#595): report stalled checkpoint flushes and trace-tag stream errors

[zgeoff]

Description

Closes #595. Closes #596.

Checkpoint-flush faults now reach the error backend with a trace id the service side shares. The submitter mints a W3C traceparent per flush, and a streak of three unanswered flushes reports a stall without stopping the stream.

• A stall fires onFlushStalled once per streak; any answered request (success or defined error) resets it, and the queue keeps retrying untouched.
• CHECKPOINT_INVALID rejections now carry the rejecting request's trace id through the worker message, store, and the game mount's Sentry capture (traceID tag, matching the service-side tag).
• The worker's RPC link forwards a per-call traceparent, so the service continues the worker's trace instead of minting its own.

Testing

• bun run typecheck passes
• bun run test passes
• bun run lint passes
• New tests added for new functionality

[coderabbitai]

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: aebeb4a4-82f3-45f0-ae72-b1b5f4c10a38

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The idle client now generates per-flush trace identifiers, detects repeated unanswered checkpoint flushes, broadcasts stall telemetry through the worker, stores it in idle state, and reports trace-tagged checkpoint errors from the web game mount.

Changes

Checkpoint flush telemetry

Layer / File(s)	Summary
Trace and telemetry contracts libs/game/idle-client/src/types.ts, libs/game/idle-client/src/submission/*	Adds flush-stall message and telemetry types, traceparent generation, a failure threshold, context-aware RPC types, and traceparent header forwarding.
Flush failure detection and tracing libs/game/idle-client/src/submission/create-checkpoint-submitter.ts, libs/game/idle-client/src/submission/*test.ts	Tracks consecutive unanswered flush failures, resets the streak for answered responses, and passes trace IDs to stall and invalid callbacks with coverage for retry and header behavior.
Worker stall message propagation libs/game/idle-client/src/worker/*	Creates and broadcasts flush-stalled messages, includes trace IDs on invalid messages, and records incoming stall telemetry in the simulation worker.
Idle state and web error reporting libs/game/idle-client/src/state/*, apps/web/src/routes/-game/*, apps/web/src/lib/idle/*	Adds store state and hooks for flush stalls, exposes the state through the worker handle, and reports trace-tagged checkpoint errors to Sentry.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

• zgeoff/vers#578: Both changes update checkpoint submitter callback wiring and onInvalid handling.
• zgeoff/vers#580: Both changes modify checkpoint flush handling, including retry and telemetry behavior.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title is concise and accurately summarizes the main checkpoint-flush and trace-tagging changes.
Description check	✅ Passed	The description is clearly related to the PR and matches the implemented telemetry and trace-correlation work.
Linked Issues check	✅ Passed	The changes satisfy #595 and #596 by surfacing repeated flush stalls and preserving trace IDs through the error path.
Out of Scope Changes check	✅ Passed	No clearly unrelated code changes were introduced beyond the flush-stall and trace-correlation objectives.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch 595-checkpoint-flush-telemetry

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 8

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@apps/web/src/routes/-game/game-simulation-mount.test.tsx`:
- Around line 73-77: Replace the inline checkpointFlushStall object in the test
with the package’s faker-defaulted create-mock-* factory, overriding only
activityID, reason, and traceID at the call site. Preserve the existing values
for those three fields and remove the manually constructed domain state object.
- Around line 69-88: Update the test around GameSimulationMount to assert the
Sentry capture side effect when checkpointFlushStall is present. Verify the
captured exception contains the activity/reason message and that its telemetry
tags include traceID set to 'trace_1', while preserving the existing
empty-render assertion.

In `@libs/game/idle-client/src/state/set-checkpoint-flush-stall.test.ts`:
- Around line 5-23: Restore the process-global Zustand state after the tests
that mutate it: in
libs/game/idle-client/src/state/set-checkpoint-flush-stall.test.ts lines 5-23,
clean up checkpointFlushStall and rewardSlotLedger after each test; in
libs/game/idle-client/src/worker/use-simulation-worker.test.ts lines 147-171,
clean up checkpointFlushStall after the worker-message test. Use the existing
onTestFinished() cleanup mechanism and restore each field to its prior baseline.

In `@libs/game/idle-client/src/submission/constants.ts`:
- Around line 13-18: Update the comments describing the flush-stall threshold
and streak in libs/game/idle-client/src/submission/constants.ts:13-18 to define
the streak as consecutive non-defined failures, with success or defined contract
responses resetting it; in
libs/game/idle-client/src/submission/create-checkpoint-submitter.ts:73-79,
replace “answered request” with “success or defined contract response”; and at
:103-105, replace “unanswered flushes” with “non-defined flush failures.”

In `@libs/game/idle-client/src/submission/create-activity-service-client.ts`:
- Around line 13-17: Add a Bun regression test around the RPCLink fetch boundary
used by the activity service client, covering both cases: a call context with a
per-call traceparent must send that value as the outbound traceparent header,
while a call without traceparent must omit the header. Preserve the existing
headers behavior in the RPCLink configuration.

In `@libs/game/idle-client/src/submission/create-checkpoint-submitter.test.ts`:
- Around line 553-558: Replace the inline ActivitySubmissionContext object
passed to registerActivity in each affected test site with the package’s
faker-defaulted create-mock-* factory, overriding only activityID, appendedHead,
lastHash, and startChainIndex as asserted. Apply the same change to the
additional occurrences around the other registerActivity calls, preserving each
test’s existing field values.
- Around line 618-650: Extend the test around submit and traceparent capture to
perform two flushes: make the first request fail non-terminally, then make the
second fail with CHECKPOINT_INVALID. Assert two traceparent headers are captured
and differ, and verify onInvalid is called with the second request’s trace ID
while preserving the existing format and activity assertions.

In
`@libs/game/idle-client/src/worker/create-checkpoint-flush-stalled-message.test.ts`:
- Around line 8-12: Update the expected message in the checkpoint flush stalled
contract test to assert the literal discriminator value
'checkpoint_flush_stalled' instead of WorkerMessageType.CheckpointFlushStalled,
and remove the now-unused WorkerMessageType import.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 68bb36d9-e1e7-4ad8-a253-03d0ea8c0870

📥 Commits

Reviewing files that changed from the base of the PR and between 2858fa9 and 6a46772.

📒 Files selected for processing (25)

• apps/web/src/lib/idle/use-idle-worker-handle.ts
• apps/web/src/routes/-game/game-simulation-mount.test.tsx
• apps/web/src/routes/-game/game-simulation-mount.tsx
• apps/web/src/test-utils/idle-worker-handle-holder.ts
• libs/game/idle-client/src/index.ts
• libs/game/idle-client/src/state/create-sync-slice.test.ts
• libs/game/idle-client/src/state/create-sync-slice.ts
• libs/game/idle-client/src/state/set-checkpoint-flush-stall.test.ts
• libs/game/idle-client/src/state/set-checkpoint-flush-stall.ts
• libs/game/idle-client/src/state/use-checkpoint-flush-stall.ts
• libs/game/idle-client/src/submission/constants.ts
• libs/game/idle-client/src/submission/create-activity-service-client.ts
• libs/game/idle-client/src/submission/create-checkpoint-submitter.test.ts
• libs/game/idle-client/src/submission/create-checkpoint-submitter.ts
• libs/game/idle-client/src/submission/create-traceparent.test.ts
• libs/game/idle-client/src/submission/create-traceparent.ts
• libs/game/idle-client/src/submission/types.ts
• libs/game/idle-client/src/types.ts
• libs/game/idle-client/src/worker/create-checkpoint-flush-stalled-message.test.ts
• libs/game/idle-client/src/worker/create-checkpoint-flush-stalled-message.ts
• libs/game/idle-client/src/worker/create-checkpoint-stream-invalid-message.test.ts
• libs/game/idle-client/src/worker/create-checkpoint-stream-invalid-message.ts
• libs/game/idle-client/src/worker/create-worker-runtime.ts
• libs/game/idle-client/src/worker/use-simulation-worker.test.ts
• libs/game/idle-client/src/worker/use-simulation-worker.ts

[cubic-dev-ai]

All reported issues were addressed across 25 files

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>