fix(server): persist hook chat messages at occurred_at so spool replays keep conversation order#4277
Merged
daviddanialy merged 5 commits intoJul 16, 2026
Conversation
…scripts by (created_at, seq) The DNO-536 root cause, fixed at the right depth: chat_messages rows were stamped at insert time and read in insertion order (seq), so offline-spool backlog replayed after recovery sorted behind the newer live event that triggered the drain. - hooks ingest persists the envelope's occurred_at as created_at, clamped to arrival time so a skewed device clock can't sort a row into the future; replayed rows now carry their true timestamps too. - every transcript reader (full lists, keyset pages, risk/search windows, resolution boundaries) orders by (created_at, seq); seq stays the stable tiebreak and the public keyset cursor — the anchor row's position is resolved server-side, so the API shape is unchanged. - non-hook writers (playground, assistants, imports) leave created_at unset and the message store stamps each batch with one shared write-time value: rows tie on created_at and seq decides, preserving their exact pre-change ordering even when buffered rows are constructed out of order. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
🦋 Changeset detectedLatest commit: 581460b The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 16 files
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
- canonicalEventTime now clamps BOTH bounds — no future timestamps and no backdating past 14 days (mirrors the client spool expiry; occurred_at is fully client-controlled and an unbounded past value would pin a row to the transcript head forever) — and the clamp moved inside the helper so chat rows, ClickHouse telemetry, and enforcement all agree on one time per event. - chat-list recency orders by GREATEST(last_message_timestamp, updated_at), and hook message writes bump chats.updated_at (existing UpdateClaudeCodeSessionTimestamp query), so a chat receiving only spool-replayed backlog still surfaces as recently active; the displayed last_message_timestamp stays the pure message time. - keyset anchors are project-scoped and fall back to the plain seq comparison when the anchor row no longer resolves, instead of returning a silent empty page; the resolutions anchor is pinned to its chat. - BackfillLatestClaudeUserMessagePromptID picks the transcript-latest user row (created_at, seq), not the seq-latest. - all inserts route through one insertChatMessages chokepoint that stamps unset created_at (no call site can forget it), and the hand-rolled pgtype literals use the conv helpers. - the clamp test pins both bounds tightly with WithinDuration. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chat (LoadChat keyset paging): - a backdated row inserted last interleaves at its occurrence position and keyset pages agree with display order, with no dups or gaps - tied created_at rows (batch stamps) page by the seq tiebreak without skipping or repeating across page boundaries - an unresolvable cursor falls back to the plain seq comparison instead of a silent empty page, in both directions - a writer batch shares one stamp and keeps insertion order hooks (ingest): - missing and malformed occurred_at fall back to arrival time - equal occurred_at events read back in arrival order - a duplicate delivery under the same Idempotency-Key mints no second row and keeps the original timestamp even when the replay claims an older one - a backdated replay bumps chats.updated_at (arrival activity) while the row itself stays backdated Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chase-crumbaugh
approved these changes
Jul 16, 2026
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 18 files (changes from recent commits).
Tip: Review your code locally with the cubic CLI to iterate faster.
Fix all with cubic | Re-trigger cubic
… message-time recency - persistCanonicalConversationEvent takes the ingest-resolved canonicalEventTime instead of recomputing it, so the chat row, telemetry, and enforcement carry the exact same server-resolved time for one event (a recomputed fallback/clamp drifted by handler latency). - chat-list recency reverts to pure message time: folding in updated_at let title renames and pin toggles reorder recency. A chat whose only new traffic is spool-replayed backlog now deliberately keeps its occurred-time position — listings are a timeline of when conversations happened. The per-message updated_at touch is removed with it. - the composite (chat_id, generation, created_at, seq) index ships separately in the migration-only PR. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the out-of-order transcript rows after offline-spool recovery (DNO-536) at the root: the server now persists hook-captured chat messages at the event's original
occurred_at, and transcript readers order by(created_at, seq). Arrival order no longer decides conversation order, so backlog replayed after a live event still sorts before it — and replayed rows now display their true timestamps instead of drain time.This supersedes the client-side ordered-send-queue approach (#4275, closed unmerged): event-time ordering at the server covers every late-delivery path (spool replays, async hook races, retries) instead of best-effort sequencing at the sender.
Changes
server/internal/hooks/ingest_hooks.go): canonical conversation events persistcanonicalEventTime(payload)ascreated_at, clamped to arrival time so a skewed device clock cannot sort a row into the future. The envelope already carriedoccurred_at; telemetry rows already trusted it — chat rows now do too.server/internal/chat/queries.sql): every transcript query — full lists, latest/explicit generation, keyset pages, risk/search windows, resolution boundaries — orders by(created_at, seq).seqstays the stable tiebreak.before_seq/after_seqint cursors; the anchor row's(created_at, seq)position is resolved server-side with a subselect, so pages agree with display order without any client/SDK change.created_atunset andChatMessageWriterstamps each batch with one shared write-time value — rows tie oncreated_atandseqdecides, preserving their exact previous ordering even when buffered rows are constructed out of order (pinned by the existingTestCaptureMessage_FlushesPendingRowsAtomically).Tests
New
TestIngest_ReplayedMessageSortsByOccurredAtdrives the exact repro through the real handler + Postgres: a live event arrives first, an older replayed event arrives second, andListChatMessagesreturns them in occurrence order withcreated_atcarrying the originaloccurred_at; a future-dated event is clamped to arrival time. Fullchat,hooks,assistants,risk, andrisk_analysissuites pass, pluslint:server.Notes
created_atalready exists; the copyfrom query now supplies it explicitly.created_at.🤖 Generated with Claude Code
Summary by cubic
Persist hook-captured chat messages at their original occurred_at (clamped to [now−14d, now]) and read transcripts by (created_at, seq) so spool replays keep the correct conversation order. Fixes Linear DNO-536 where newer live messages appeared before older replayed ones; chat list recency now reflects pure message time.
Bug Fixes
Migration
Written for commit 581460b. Summary will update on new commits.