Skip to content

fix(server): persist hook chat messages at occurred_at so spool replays keep conversation order#4277

Merged
daviddanialy merged 5 commits into
mainfrom
daviddanialy/dno-536-server-respect-occurred-at
Jul 16, 2026
Merged

fix(server): persist hook chat messages at occurred_at so spool replays keep conversation order#4277
daviddanialy merged 5 commits into
mainfrom
daviddanialy/dno-536-server-respect-occurred-at

Conversation

@daviddanialy

@daviddanialy daviddanialy commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes the out-of-order transcript rows after offline-spool recovery (DNO-536) at the root: the server now persists hook-captured chat messages at the event's original occurred_at, and transcript readers order by (created_at, seq). Arrival order no longer decides conversation order, so backlog replayed after a live event still sorts before it — and replayed rows now display their true timestamps instead of drain time.

This supersedes the client-side ordered-send-queue approach (#4275, closed unmerged): event-time ordering at the server covers every late-delivery path (spool replays, async hook races, retries) instead of best-effort sequencing at the sender.

Changes

  • Ingest (server/internal/hooks/ingest_hooks.go): canonical conversation events persist canonicalEventTime(payload) as created_at, clamped to arrival time so a skewed device clock cannot sort a row into the future. The envelope already carried occurred_at; telemetry rows already trusted it — chat rows now do too.
  • Readers (server/internal/chat/queries.sql): every transcript query — full lists, latest/explicit generation, keyset pages, risk/search windows, resolution boundaries — orders by (created_at, seq). seq stays the stable tiebreak.
  • Cursors unchanged at the API: keyset pagination keeps its before_seq/after_seq int cursors; the anchor row's (created_at, seq) position is resolved server-side with a subselect, so pages agree with display order without any client/SDK change.
  • Non-hook writers unchanged in behavior: playground, assistants, and imports leave created_at unset and ChatMessageWriter stamps each batch with one shared write-time value — rows tie on created_at and seq decides, preserving their exact previous ordering even when buffered rows are constructed out of order (pinned by the existing TestCaptureMessage_FlushesPendingRowsAtomically).

Tests

New TestIngest_ReplayedMessageSortsByOccurredAt drives the exact repro through the real handler + Postgres: a live event arrives first, an older replayed event arrives second, and ListChatMessages returns them in occurrence order with created_at carrying the original occurred_at; a future-dated event is clamped to arrival time. Full chat, hooks, assistants, risk, and risk_analysis suites pass, plus lint:server.

Notes

  • No migration: created_at already exists; the copyfrom query now supplies it explicitly.
  • Rows without an explicit timestamp are stamped by the writer, so nothing can insert a NULL/zero created_at.

🤖 Generated with Claude Code


Summary by cubic

Persist hook-captured chat messages at their original occurred_at (clamped to [now−14d, now]) and read transcripts by (created_at, seq) so spool replays keep the correct conversation order. Fixes Linear DNO-536 where newer live messages appeared before older replayed ones; chat list recency now reflects pure message time.

  • Bug Fixes

    • Ingest: persist occurred_at as created_at via one canonical clamp (no future, no >14d past); missing/malformed occurred_at falls back to arrival; pass the resolved time into persistence so chat rows, telemetry, and enforcement share one event time; persist replayed.
    • Queries: order all transcript reads/windows by (created_at, seq); keyset pages resolve anchor position by (created_at, seq) within the same project/chat and fall back to plain seq if the anchor is missing; “latest user message” and resolution boundaries align to transcript order.
    • Writers: non-hook paths leave created_at unset; all inserts route through a single chokepoint that stamps one write-time per batch, preserving previous ordering when rows are constructed out of order.
    • Chat lists: recency uses message time only (no updated_at bump), so backlog-only replays keep their occurred-time position.
  • Migration

    • No schema or API changes; seq-based cursors are unchanged.

Written for commit 581460b. Summary will update on new commits.

Review in cubic

…scripts by (created_at, seq)

The DNO-536 root cause, fixed at the right depth: chat_messages rows were
stamped at insert time and read in insertion order (seq), so offline-spool
backlog replayed after recovery sorted behind the newer live event that
triggered the drain.

- hooks ingest persists the envelope's occurred_at as created_at, clamped
  to arrival time so a skewed device clock can't sort a row into the
  future; replayed rows now carry their true timestamps too.
- every transcript reader (full lists, keyset pages, risk/search windows,
  resolution boundaries) orders by (created_at, seq); seq stays the stable
  tiebreak and the public keyset cursor — the anchor row's position is
  resolved server-side, so the API shape is unchanged.
- non-hook writers (playground, assistants, imports) leave created_at
  unset and the message store stamps each batch with one shared write-time
  value: rows tie on created_at and seq decides, preserving their exact
  pre-change ordering even when buffered rows are constructed out of order.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@daviddanialy
daviddanialy requested a review from a team as a code owner July 16, 2026 19:52
@daviddanialy daviddanialy added the bug Something isn't working label Jul 16, 2026
@linear-code

linear-code Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

DNO-536

@changeset-bot

changeset-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 581460b

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
server Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 16 files

Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic

Comment thread server/internal/chat/queries.sql
- canonicalEventTime now clamps BOTH bounds — no future timestamps and no
  backdating past 14 days (mirrors the client spool expiry; occurred_at
  is fully client-controlled and an unbounded past value would pin a row
  to the transcript head forever) — and the clamp moved inside the helper
  so chat rows, ClickHouse telemetry, and enforcement all agree on one
  time per event.
- chat-list recency orders by GREATEST(last_message_timestamp,
  updated_at), and hook message writes bump chats.updated_at (existing
  UpdateClaudeCodeSessionTimestamp query), so a chat receiving only
  spool-replayed backlog still surfaces as recently active; the displayed
  last_message_timestamp stays the pure message time.
- keyset anchors are project-scoped and fall back to the plain seq
  comparison when the anchor row no longer resolves, instead of returning
  a silent empty page; the resolutions anchor is pinned to its chat.
- BackfillLatestClaudeUserMessagePromptID picks the transcript-latest
  user row (created_at, seq), not the seq-latest.
- all inserts route through one insertChatMessages chokepoint that stamps
  unset created_at (no call site can forget it), and the hand-rolled
  pgtype literals use the conv helpers.
- the clamp test pins both bounds tightly with WithinDuration.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
chat (LoadChat keyset paging):
- a backdated row inserted last interleaves at its occurrence position and
  keyset pages agree with display order, with no dups or gaps
- tied created_at rows (batch stamps) page by the seq tiebreak without
  skipping or repeating across page boundaries
- an unresolvable cursor falls back to the plain seq comparison instead of
  a silent empty page, in both directions
- a writer batch shares one stamp and keeps insertion order

hooks (ingest):
- missing and malformed occurred_at fall back to arrival time
- equal occurred_at events read back in arrival order
- a duplicate delivery under the same Idempotency-Key mints no second row
  and keeps the original timestamp even when the replay claims an older one
- a backdated replay bumps chats.updated_at (arrival activity) while the
  row itself stays backdated

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 18 files (changes from recent commits).

Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic

Comment thread server/internal/chat/queries.sql Outdated
Comment thread server/internal/hooks/ingest_hooks.go
daviddanialy and others added 2 commits July 16, 2026 14:00
… message-time recency

- persistCanonicalConversationEvent takes the ingest-resolved
  canonicalEventTime instead of recomputing it, so the chat row, telemetry,
  and enforcement carry the exact same server-resolved time for one event
  (a recomputed fallback/clamp drifted by handler latency).
- chat-list recency reverts to pure message time: folding in updated_at
  let title renames and pin toggles reorder recency. A chat whose only new
  traffic is spool-replayed backlog now deliberately keeps its
  occurred-time position — listings are a timeline of when conversations
  happened. The per-message updated_at touch is removed with it.
- the composite (chat_id, generation, created_at, seq) index ships
  separately in the migration-only PR.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@daviddanialy
daviddanialy added this pull request to the merge queue Jul 16, 2026
Merged via the queue into main with commit dae476c Jul 16, 2026
35 checks passed
@daviddanialy
daviddanialy deleted the daviddanialy/dno-536-server-respect-occurred-at branch July 16, 2026 21:27
@github-actions github-actions Bot locked and limited conversation to collaborators Jul 16, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants