Skip to content

fix(mobile): stream and event durability - #4987

Merged
iscekic merged 27 commits into
mainfrom
audit-w2c-stream-durability-75e6
Aug 5, 2026
Merged

fix(mobile): stream and event durability#4987
iscekic merged 27 commits into
mainfrom
audit-w2c-stream-durability-75e6

Conversation

@iscekic

@iscekic iscekic commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Close the session snapshot-to-stream gap with a Durable Object event watermark.
  • Ignore duplicate and out-of-order stream replay events.
  • Persist remote CLI command outcomes through Durable Object hibernation and dedupe retries by mutation ID.
  • Fence stale Event Service connection attempts and resync client state after sequence gaps.
  • Tolerate unknown bot capabilities and load bot status through HTTP without a WebSocket.
  • Release the Event Service socket and bot polling while mobile is inactive.
  • Bound mobile control requests and sends with Hermes-safe deadlines.

Why

Mobile streams, remote CLI commands, and Event Service state could lose updates, duplicate updates, or hang during lifecycle changes.

How

  • Read a session-local event watermark before the first history page, then open stream replay from that watermark.
  • Add monotonic replay dedupe and durable command correlation terminal outcomes.
  • Add optional Event Service seq, ack, gap detection, and consumer resync invalidation without an Event Service event log.
  • Use a shared AbortController plus setTimeout deadline helper because Hermes does not reliably provide static AbortSignal helpers.

Evidence and decisions

  • Durable Object identity: /stream derives CLOUD_AGENT_SESSION from ${userId}:${cloudAgentSessionId} in services/cloud-agent-next/src/server.ts:316-318, so the event table and max(events.id) are session-local.
  • Watermark design: getSessionMessagesPage reads getSession(...).latestEventId before the history page. The stream-ticket route remains rejected because it reads no Durable Object state and the client fetches ticket and page concurrently.
  • Extension pass-through: out of scope. The field is optional and this mobile audit changes only the mobile adapter.
  • CLI transport gate: historical and live CLI transports contain no Cloud Agent stream first-connect replay=false rule, so they remain unchanged.
  • Event identity: event IDs are Durable Object-local SQLite autoincrement IDs; execution_id is non-unique, so IDs remain monotonic across execution changes.
  • Re-issuing call sites: remote create-session retries retain their mutation ID; bare old-CLI fallback remains a separate intent. No other re-issuing caller exists.
  • Hibernation test: Worker test reconstruction injects persisted correlations and rehydrates attachments. The harness cannot force production eviction directly.
  • Listener isolation: unchanged. activeContexts is refcounted and on() returns per-handler unsubscribe.
  • Bot status surface: absent status renders Unknown; aged lastAt renders Idle or Offline in bot-send-state.ts.
  • useBotStatus callers: both mobile conversation screens were updated for active-state polling. hasAttachmentsCapability still checks the filtered list.
  • Resync mapping: mobile conversation events, conversation lists, and bot status use onResync; no Event Service consumer needs background delivery. Screen subscriptions release on blur, and push plus AppState invalidation handles badges.
  • Connection indicator: no Event Service indicator exists. The visible connectivity banner uses device network state, so debounce is not applicable.
  • Retention: CloudAgentSession.cleanupOldEvents retains events for Limits.SESSION_TTL_MS; this section does not change it.
  • Existing implementation retained: event cleanup in CloudAgentSession.cleanupOldEvents; listener isolation in EventServiceClient.activeContexts; device-network banner in ConnectivityBanner.
  • Agent-viewer stale and unknown behavior is out of scope and owned by W6-C.
  • Decisions D1–D9: session-TTL event retention; no Event Service log; Hermes-safe deadlines; no timeout resend; optional old-peer fields; server-side watermark carrier; optional seq with ack opt-in; durable CLI terminal TTL; reversible acquire/release lifecycle.
  • EV-03: E2E uses simulators and emulators because the harness has no physical-device path.

Verification

  • pnpm typecheck
  • pnpm lint
  • pnpm format:check
  • Changed package tests, worker tests, mobile tests, and targeted web tests pass.
  • E2E: passed on real docker-local KiloClaw with kilo-auto/efficient: iOS stream reconnect, remote CLI reconstruction, and iOS plus Android background/foreground lifecycle. The live harness has no force-hibernate control, so EV-03 used mid-flight background reconstruction.
  • E2E recovery: a foreign section held kilo_pixel7; the workflow now fails fast on AVD contention. This run used free kilo_pixel9_api35 and did not stop the foreign emulator.

Visual Changes

N/A

@iscekic iscekic self-assigned this Aug 4, 2026
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts Outdated
Comment thread packages/kilo-chat/src/schemas.ts
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts Outdated
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts Outdated
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts
Comment thread services/session-ingest/src/types/user-connection-protocol.ts Outdated
Comment thread packages/cloud-agent-sdk/src/cloud-agent-transport.ts Outdated
Comment thread services/event-service/src/do/user-session-do.ts
Comment thread packages/event-service/src/client.ts Outdated
@kilo-code-bot

kilo-code-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Code Review Summary

Status: 2 Issues Found | Recommendation: Merge — both remaining findings are carried items with recorded author dispositions; the new tmux keep-alive test fix and both main-merge conflict resolutions introduce no new issues.

Executive Summary

The incremental scope — a tmux keep-alive fix for the stale-environment test plus two merges of main — is clean: the merges preserved every PR-attributable hunk verbatim, and the two carried findings remain unchanged with recorded author dispositions.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 1439 Carried (unchanged this increment, re-verified at HEAD): countDurablePendingCommands() counts 'done' entries (kept for the full 35 s TTL) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.test.ts 6022 Carried (unchanged this increment, re-verified at HEAD): Typo in comment — ractests should be race-tests (author declined: comment-only)
Files Reviewed (7 files, incremental)
  • dev/local/tmux.test.ts - no issues; keep-alive session correctly held until after global-env restore and killed in finally
  • apps/mobile/src/lib/trpc.ts - PR hunks semantically identical after merge (deadline fetch wiring preserved under main's restructured links)
  • apps/mobile/src/lib/trpc.test.ts - PR hunks identical after merge (deadline tests rebased verbatim onto main's new mocks)
  • apps/web/src/routers/cli-sessions-v2-router.ts - PR hunks identical after merge
  • apps/web/src/routers/cli-sessions-v2-router.test.ts - PR hunks identical after merge
  • packages/cloud-agent-sdk/src/cli-live-transport.ts - PR hunks identical after merge
  • packages/cloud-agent-sdk/src/cli-live-transport.test.ts - PR hunks identical after merge

Fix these issues in Kilo Cloud

Previous Review Summaries (7 snapshots, latest commit 3f12c37)

Current summary above is authoritative. Previous snapshots are kept for context only.

Previous review (commit 3f12c37)

Status: 2 Issues Found | Recommendation: Merge — both remaining findings are carried items with recorded author dispositions; every previously flagged new issue is resolved in the latest commits.

Executive Summary

The incremental changes correctly close the completedCorrelationIds sweep leak (marker released at the done-past-TTL delete) and strengthen the terminal-stash test precondition to toBe(1); only two previously reported, author-acknowledged findings remain open.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 1439 Carried: countDurablePendingCommands() counts 'done' entries (kept for the full 35 s TTL) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.test.ts 6022 Carried: Typo in comment — ractests should be race-tests (author declined: comment-only)
Resolved in new commits
  • completedCorrelationIds markers never released on sweep success paths (previous line 2494 comment) — fixed by releasing the marker at the done-past-TTL durable delete (UserConnectionDO.ts line 2495); delete-then-release ordering is safe and no stale 'pending' snapshot can survive across sweep cycles to exploit the release
  • Vacuous terminalStash precondition assertion (previous test comment) — strengthened to expect(terminalStash.size).toBe(1) at UserConnectionDO.test.ts line 7468, so the rejection-cleanup path is now genuinely exercised
Files Reviewed (2 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 1 carried issue
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 1 carried issue

Fix these issues in Kilo Cloud

Previous review (commit fc4c6c8)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

The new cross-sweep claim gates close the double-delivery window and the initial-write rejection path now cleans its stash, but the same change removed the only completedCorrelationIds garbage collection, so sweep-claimed markers now accumulate in the Durable Object until eviction.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 2491 NEW: Sweep success paths never release completedCorrelationIds claims (lines 2348/2424/2460) and the old release at the done-entry delete was removed — every expired/terminal-swept command leaks a UUID marker in memory until DO eviction; releasing at the past-TTL delete is safe since no stale 'pending' snapshot can survive that long
services/session-ingest/src/dos/UserConnectionDO.ts 1439 Carried: countDurablePendingCommands() counts 'done' entries (kept for the full 35 s TTL) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.test.ts 7468 NEW: expect(terminalStash.size).toBeGreaterThanOrEqual(0) is vacuous — the stated precondition (stash exists before rejection) is never verified, so the test can pass without exercising the .catch cleanup; should be toBe(1)
services/session-ingest/src/dos/UserConnectionDO.test.ts 6022 Carried: Typo in comment — ractests should be race-tests (author declined: comment-only)
Resolved in new commits
  • Concurrent durable terminal sweeps double-delivery (previous line 2481 comment) — fixed by the new completedCorrelationIds claim gates in finishDurablePendingCommands and both expirePendingCommands sweeps
  • Stashed terminal entry and fence marker leak on initial-write rejection (previous line 1643 comment) — fixed by the new .catch cleanup in dispatchWebCommandSync
Files Reviewed (2 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 2 issues (1 new, 1 carried)
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 2 issues (1 new, 1 carried)

Fix these issues in Kilo Cloud

Previous review (commit 57e4530)

Status: 4 Issues Found | Recommendation: Address before merge

Executive Summary

The new terminal-write fencing closes the live-send-vs-durable-write window, but the concurrent durable-sweep double-delivery gap remains open, and the new initial-write rejection path leaks the stashed terminal entry and fence marker until Durable Object eviction.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 2
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 2423 Carried: Concurrent durable terminal sweeps (expirePendingCommands vs finishDurablePendingCommands) can both pass the state === 'pending' check on a stale list() snapshot and double-deliver terminal responses for the same originalId — the new awaited terminal writes and retry arms add delivery paths but no cross-sweep dedupe
services/session-ingest/src/dos/UserConnectionDO.ts 1439 Carried: countDurablePendingCommands() counts 'done' entries (kept for the full 35 s TTL) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 1638 NEW: If the initial storage.put rejects after a terminal outcome was stashed in terminalDuringInitialWrite, the only stash cleanup (in .then, line 1617) is skipped and the .finally keeps the fence marker — the Map entry and marker leak in memory until DO eviction
services/session-ingest/src/dos/UserConnectionDO.test.ts 6022 Carried: Typo in comment — ractests should be race-tests (author declined: comment-only)
Files Reviewed (2 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 3 issues (1 new, 2 carried)
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 1 issue (carried)

Fix these issues in Kilo Cloud

Previous review (commit fe32137)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The new durable terminal-delivery paths close the memory-vs-durable duplicate window but leave a sweep-vs-sweep gap: two concurrent durable terminal sweeps (expiry on every web command/alarm vs CLI-disconnect/owner-change) read the same stale storage snapshot and can both deliver terminal responses for one correlationId; the author-acknowledged done-entry cap counting and a rejected comment typo carry forward unchanged.

Overview

Severity Count
CRITICAL 0
WARNING 2
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 2284 NEW: Concurrent durable terminal sweeps (expirePendingCommands vs finishDurablePendingCommands) can both pass the state === 'pending' check on a stale list() snapshot and double-deliver terminal responses for the same originalIddeliveredInMemory/skipIds only dedupe memory-vs-durable within one sweep
services/session-ingest/src/dos/UserConnectionDO.ts 1421 Carried: countDurablePendingCommands() counts 'done' entries (kept 35 s) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.test.ts 6003 Carried: Typo in comment — ractests should be race-tests (author declined: comment-only)
Files Reviewed (2 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 2 issues (1 new, 1 carried)
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 1 issue (carried)

Fix these issues in Kilo Cloud

Previous review (commit b06820c)

Status: 2 Issues Found | Recommendation: Address before merge

Executive Summary

The latest commit verifiably fixes the two open UserConnectionDO warnings — the rehydrated-path duplicate window is now fenced by a synchronous completedCorrelationIds reservation taken before the async durable read, and the completion-marker leak plus unhandled waitUntil rejection are resolved via try/finally cleanup, catch-and-rethrow around the durable puts, and a .catch at the waitUntil boundary — while the author-confirmed countDurablePendingCommands() done-entry cap counting remains open on unchanged lines and one comment typo was newly introduced in the test file.

Overview

Severity Count
CRITICAL 0
WARNING 1
SUGGESTION 1
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 1399 countDurablePendingCommands() still counts 'done' entries (kept 35 s) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible

SUGGESTION

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.test.ts 5989 Typo in new test comment — ractests should be race-tests
Files Reviewed (2 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 1 issue
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 1 issue

Fix these issues in Kilo Cloud

Previous review (commit 3c3a5f7)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The latest commits verifiably fix 7 prior findings (durable result bounding below the 128 KiB DO limit, kiloclaw synced-schema regeneration, rehydration/live expiry guards, bounded mutationId, fromId=0 replay closing the materialization gap, raised zombie-close threshold, resync state ordering), while the three previously reported UserConnectionDO warnings — rehydrated-path duplicate delivery, completion-marker leak on a failed durable put, and the recorded D8 done-entry cap counting — remain open on current HEAD.

Overview

Severity Count
CRITICAL 0
WARNING 3
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 830 Rehydrated-path duplicate window remains: completedCorrelationIds is only populated in the live path (lines 918/1022), so after a DO restart two racing CLI responses for one correlation id can both read 'pending' — both storage gets issued before the first put — and be delivered twice
services/session-ingest/src/dos/UserConnectionDO.ts 1022 completedCorrelationIds entries leak when the awaited durable storage.put throws (no try/finally around add/delete; same pattern in the catalog path at lines 918/951) — a stale marker silently drops later legitimate responses for that id, and the rejection propagates unhandled into the no-catch waitUntil at line 527
services/session-ingest/src/dos/UserConnectionDO.ts 1325 countDurablePendingCommands() still counts 'done' entries (kept 35 s) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible
Files Reviewed (11 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 3 issues
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 0 issues
  • services/session-ingest/src/types/user-connection-protocol.ts - 0 issues
  • packages/cloud-agent-sdk/src/cloud-agent-transport.ts - 0 issues
  • packages/cloud-agent-sdk/src/cloud-agent-transport.test.ts - 0 issues
  • packages/event-service/src/client.ts - 0 issues
  • packages/event-service/src/__tests__/client.test.ts - 0 issues
  • services/event-service/src/do/user-session-do.ts - 0 issues
  • services/event-service/src/__tests__/user-session-do.test.ts - 0 issues
  • services/kiloclaw/plugins/kilo-chat/src/synced/schemas.ts - 0 issues
  • services/kiloclaw/plugins/kilo-chat/src/synced/events.ts - 0 issues

Fix these issues in Kilo Cloud

Previous review (commit b833068)

Status: 3 Issues Found | Recommendation: Address before merge

Executive Summary

The follow-up commit verifiably fixes 8 of 9 previous findings (durable result bounding below the 128 KiB DO limit, kiloclaw sync regeneration, rehydration/live expiry guards, bounded mutationId, fromId=0 replay closing the materialization gap, raised zombie-close threshold, resync state ordering), but the new completedCorrelationIds dedupe marker does not cover the rehydrated-path duplicate-delivery window and itself leaks if the durable put fails; the done-entry cap counting remains as a recorded D8 decision.

Overview

Severity Count
CRITICAL 0
WARNING 3
SUGGESTION 0
Issue Details (click to expand)

WARNING

File Line Issue
services/session-ingest/src/dos/UserConnectionDO.ts 839 Rehydrated-path duplicate window remains (previous line-830 finding only partially fixed): completedCorrelationIds is only populated in the live path (lines 918/1022), so after a DO restart two racing CLI responses for one correlation id can both read 'pending' — both storage gets issued before the first put — and be delivered twice
services/session-ingest/src/dos/UserConnectionDO.ts 1022 completedCorrelationIds entries leak when the awaited durable storage.put throws (no try/finally around add/delete) — stale marker silently drops later legitimate responses for that id, and the rejection propagates unhandled into the no-catch waitUntil at line 527
services/session-ingest/src/dos/UserConnectionDO.ts 1325 countDurablePendingCommands() still counts 'done' entries (kept 35 s) toward MAX_PENDING_COMMANDS — author confirmed intentional per recorded D8 decision (full-TTL retry dedupe); false PENDING_COMMAND_LIMIT rejections under retry loops remain possible
Files Reviewed (11 files, incremental)
  • services/session-ingest/src/dos/UserConnectionDO.ts - 3 issues
  • services/session-ingest/src/dos/UserConnectionDO.test.ts - 0 issues
  • services/session-ingest/src/types/user-connection-protocol.ts - 0 issues
  • packages/cloud-agent-sdk/src/cloud-agent-transport.ts - 0 issues
  • packages/cloud-agent-sdk/src/cloud-agent-transport.test.ts - 0 issues
  • packages/event-service/src/client.ts - 0 issues
  • packages/event-service/src/__tests__/client.test.ts - 0 issues
  • services/event-service/src/do/user-session-do.ts - 0 issues
  • services/event-service/src/__tests__/user-session-do.test.ts - 0 issues
  • services/kiloclaw/plugins/kilo-chat/src/synced/schemas.ts - 0 issues
  • services/kiloclaw/plugins/kilo-chat/src/synced/events.ts - 0 issues

Fix these issues in Kilo Cloud


Reviewed by kimi-k3 · Input: 97.7K · Output: 19.4K · Cached: 1.3M

Review guidance: REVIEW.md from base branch main

Comment thread services/session-ingest/src/dos/UserConnectionDO.ts Outdated
Comment thread services/session-ingest/src/dos/UserConnectionDO.test.ts
iscekic added 3 commits August 4, 2026 13:22
Persist terminal outcomes before delivery and prevent duplicate responses during CLI disconnect and ownership changes.
Keep durable write ordering assertions type-safe in the session ingest tests.
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts
@iscekic iscekic changed the title Fix mobile stream and event durability fix: mobile stream and event durability Aug 4, 2026
@iscekic iscekic changed the title fix: mobile stream and event durability fix(mobile): stream and event durability Aug 4, 2026
Keep terminal command outcomes durable while delayed initial writes settle.
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts
iscekic added 2 commits August 4, 2026 19:48
Reserve terminal durable correlations across concurrent sweep paths.
Comment thread services/session-ingest/src/dos/UserConnectionDO.ts
Comment thread services/session-ingest/src/dos/UserConnectionDO.test.ts Outdated
Drop expired durable correlation markers after their done entries expire.
@iscekic iscekic added the human-ready The PR is ready for human review. label Aug 4, 2026
…bility-75e6

# Conflicts:
#	apps/mobile/src/lib/trpc.test.ts
#	apps/mobile/src/lib/trpc.ts
@iscekic
iscekic enabled auto-merge (squash) August 5, 2026 14:26
@iscekic
iscekic disabled auto-merge August 5, 2026 14:27
@iscekic
iscekic enabled auto-merge (squash) August 5, 2026 14:28
set-environment -g fails when no tmux server is running, which broke
the test on CI runners without a leftover server (and on main).
@iscekic
iscekic merged commit 83c7c44 into main Aug 5, 2026
32 of 33 checks passed
@iscekic
iscekic deleted the audit-w2c-stream-durability-75e6 branch August 5, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

human-ready The PR is ready for human review.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants