feat(web): infinite-scroll conversation events

[Cha0os]

Reworked per @pikann's review: the client-side loop is gone. The conversation view now reads its events with useInfiniteQuery — the primitive the conversation and activity lists already use — instead of assembling the whole stream up front.

Why it matters

Measured on a live instance (249 conversations):

Conversations past a single 200-event page	51 (one in five)
Largest stream	3.4 MB stored / ~5 MB as JSON
Median event	~5.4 kB
Largest single event	781 kB (raw_output is 780 kB of it)

persist_conversation_event publishes one realtime message per event, and the view re-read the stream on each one — a 622-event turn re-read it 622 times. Opening that conversation now costs one request (~1.1 MB) instead of four (~5 MB).

Shape

• Opening fetches the newest page. event_count on the single-conversation read gives the tail offset with no extra round trip; without it the client asks for one row.
• Live events extend the window by exactly the pages it lacks. Scrolled away, the reader gets a count and a jump-to-latest affordance, and nothing is fetched.
• Scrolling up pages older events in and holds position.
• Both scopes: ConversationView serves the project and global routes, so the reader takes an optional projectId.

Also in here: realtime stopped refetching the conversation itself per event

Both realtime hooks invalidated the whole conversations prefix for every message, so each event refetched the conversation list and the detail. From a live instance's API log:

busiest second:  list=12  detail=12  events=2
3-hour totals:   list=240 detail=254 events=52

A persisted event carries an event_index and changes only the event stream; the conversation's own fields change on the lifecycle messages, which arrive separately and still invalidate exactly as before. A payload without an event_index is treated as lifecycle, so an API predating the field keeps today's behaviour.

Four things a reviewer should look at

Keys sit outside ["projects", …] and ["global-chat", …]. An invalidated infinite query refetches every page it holds — the cost this removes. The tail signal is also never fetched, so nothing can replace what was written into it. Two tests fail if either key moves back under a prefix.

Paging a stream that is still growing. getNextPageParam widens the end with the highest index realtime has reported, because a page's total is only as fresh as when it was fetched. The query stays disabled until at least one event exists, so a conversation empty at open begins fetching when realtime reports its first — and an empty page then unambiguously means end-of-stream.

A page carries its own limit. The last hop backwards is usually shorter than a full page; requesting a full one there refetched events already held, which showed up as duplicated messages at the start of a long conversation.

event_count is a pointer, omitted from list responses. Only the detail read loads it (both scopes share FindConversationByID), so list pages neither compute nor report it.

Verification

• 69 unit tests. The reader is driven against a growing stream through the HTTP client, so the query options are exercised rather than mocked over: newest-page open, paging back without duplication, paging to the start, realtime append, count-without-fetch while scrolled away, empty-at-open, a burst outrunning one page, the count probe, the global route. Plus two on which realtime messages invalidate what.
• biome check clean across 393 files; tsc -b && vite build green.
• go build, go vet, and the api handler + repository tests pass.

Not covered by tests: scroll behaviour — opening at the newest message, pinning while at the bottom, holding position on a prepend. It is reasoned from ThreadPrimitive.Viewport's implementation and wants a look in a browser.

Deliberately out of scope

• raw_output truncation — 40% of all payload bytes (63 MB of 157 MB) and the reason a page's size is unpredictable (usually ~1.1 MB, occasionally 10 MB). A bigger lever than paging, and a separate change.
• The chat floats still read a single page, exactly as today.
• Virtualization, and downward paging — unnecessary while the window is tail-anchored and nothing is evicted.

🤖 Generated with Claude Code

[pullfrog]

✅ No new issues found.

Reviewed changes

This PR fixes the web client so it pages through a conversation's full event stream instead of silently truncating at the server's 200-event limit.

• Shared pagination helper — fetchAllConversationEvents in apps/web/src/lib/agent-api.ts walks {offset, limit} pages for both the project-scoped and global event routes.
• Bounded loop — stops on a short page as well as on total, so a running conversation that appends events between requests cannot spin an unbounded request loop.
• Tests — covers single-page, multi-page, short-page, missing total, and global-route cases.

I verified the server contract (ConversationHandler.ListConversationEvents / GetGlobalConversationEvents return {items, total} and the repository orders by event_index ASC with positional OFFSET/LIMIT), ran the test file, and checked Biome on the changed files — all clean.

  ^{｜ View workflow run ｜ Using Kimi K2 (free via Pullfrog for OSS) ｜ 𝕏}

[pikann]

Thank you for your contribution! Please let me know if you are unable to make these updates, and I will gladly take over

[pikann]

Also, don't forget to commit using your GitHub account so you're added to the contributors list!

[Cha0os]

Thanks — you were right, and it's done: the client-side loop is gone. The view now pages with useInfiniteQuery, opening on the newest page and loading older events on demand.

A few notes on what changed since your review:

• No loop anywhere. fetchAllConversationEvents is removed rather than kept behind a flag. The chat floats still read a single page, exactly as they did before this PR.
• Live updates got cheaper too. While measuring this I found both realtime hooks were invalidating the whole conversations prefix per event, so every event refetched the conversation list and the detail. On a live instance a second carrying 12 events produced 12 list + 12 detail requests alongside 2 event requests. Persisted events now invalidate only that conversation's event key; lifecycle messages (agent.conversation.*, agent.session.started) invalidate the prefix as before.
• Two small server additions, both optional for the client: event_count on the single-conversation read so the view can open on the last page without first asking how many events exist (a pointer, omitted from list responses), and event_index in the realtime payload so a client can fetch only what it is missing. Without either, the client falls back to a one-row count request and a tick-only signal.
• Commits are attributed properly now — thanks for flagging that, the earlier one was authored to a local hostname address.

One thing I want to be straight about: the scroll behaviour — opening at the newest message, staying pinned while at the bottom, and holding position when older events are prepended — is reasoned from ThreadPrimitive.Viewport's implementation rather than covered by a test. Everything else has tests, including two that fail if the paged reader or the realtime signal is keyed back under an invalidated prefix. If you'd rather that scrolling logic lived elsewhere, or looks wrong in a browser, I'm happy to move or rework it.

[Cha0os]

Thanks for taking this into consideration, hope it's not getting too messy. This is the state I'm using locally and it's much easier to work with the agent conversations now for all the reasons listed above.

• Lukas

[pullfrog]

✅ No new issues found.

Reviewed changes

This run reviewed the complete rework that replaced the prior client-side pagination loop with a TanStack Query useInfiniteQuery window, added realtime tail signalling, and introduced load-older / jump-to-latest UI controls.

• Replaced the eager event fetch with an infinite-query window — useConversationEventWindow anchors the initial fetch at the newest page, pages older events on demand, and catches up to live events only while the reader is following the tail.
• Added scroll-affordance controls — LoadOlderEvents restores scroll position after prepending older messages, and TailFollowIndicator lets readers jump back when new events arrive while they are scrolled away.
• Reduced realtime refetch churn — persisted events now write a tail signal and invalidate only the event-stream key; lifecycle events still invalidate the conversations prefix.
• Added event_count to the single-conversation read — both project and global detail routes use it to open on the tail without a probe.
• Expanded test coverage — 69 unit tests cover window paging, realtime append/catch-up, empty-at-open, count probe, global route, and invalidation behavior.

  ^{｜ View workflow run ｜ Using Kimi K2 (free via Pullfrog for OSS) ｜ 𝕏}

[pikann]

Hi @Cha0os ,
Thank you so much for your contribution!
To maintain consistency across the project, we'd like to update this to use cursor-based pagination.

Would you mind if I take it over from here to apply those changes, or would you prefer to update it yourself?

[Cha0os]

Hi @Cha0os , Thank you so much for your contribution! To maintain consistency across the project, we'd like to update this to use cursor-based pagination.

Would you mind if I take it over from here to apply those changes, or would you prefer to update it yourself?

Of course, go ahead. Thank you!

[pullfrog]

Important

The cursor-pagination refactor looks correct, but an existing E2E test still asserts the old offset/limit contract and will fail when the E2E suite runs.

Reviewed changes

This run reviewed the incremental refactor that replaced offset-based paging with cursor-based keyset pagination for conversation events and removed the now-unnecessary event_count field.

• Switched the events endpoint to cursor pagination — parseConversationEventWindowQuery now accepts after/before/limit; writeConversationEventWindowResponse returns next_cursor/prev_cursor; the repository keyset-seeks by event_index.
• Removed event_count from the conversation read — the newest page is fetched with no cursor, so the client no longer needs the conversation's length upfront.
• Updated the client window hook — useConversationEventWindow no longer takes eventCount, handles empty conversations via refetch on the first realtime event, and drives older-page loading with an IntersectionObserver sentinel in LoadOlderEvents.
• Updated tests — handler unit tests cover the new query contract; agent-api.test.ts covers cursor forwarding; use-conversation-event-window.test.tsx exercises the window through the HTTP client.
• Verified Go build — go build ./... succeeds; go test ./internal/transport/http/handler/... -run TestListConversationEvents passes.

⚠️ E2E offset-limit validation test is stale

services/api/test/e2e/conversation_pagination_test.go still expects the events endpoint to reject offset=-1 and offset=abc with 400. The new parser no longer reads offset, so those requests return 200 and the E2E suite (PACA_E2E=1 go test ./test/e2e/...) will fail.

Technical details

# Stale E2E offset validation

## Affected sites
- `services/api/test/e2e/conversation_pagination_test.go:665` — `TestE2EListConversationEvents_OffsetLimitValidation` asserts 400 for `offset=-1` and `offset=abc`, but the endpoint now ignores `offset`.

## Required outcome
- Update or remove the offset cases so the E2E suite passes under the cursor contract.
- Add cursor-boundary coverage if missing: malformed `after`/`before` rejected with 400, mutually exclusive after+before rejected, and cursors round-trip correctly.

## Suggested approach
- Replace the offset invalid cases with cursor invalid cases (`after=not-base64`, `before=garbage`).
- Keep the `limit` invalid cases — they still apply.
- Add a happy-path case confirming a bare request returns the newest page with `next_cursor`/`prev_cursor` populated correctly.

  ^{｜ Fix it ➔ ｜ View workflow run ｜ Using Kimi K2 (free via Pullfrog for OSS) ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes

This run reviewed the incremental test-only commit (293ce77d), which reworks the E2E coverage for the cursor-paginated conversation-events endpoint.

• Replaced the stale offset-limit E2E test — TestE2EListConversationEvents_OffsetLimitValidation is now TestE2EListConversationEventsWindow_LimitValidation, dropping the dead offset cases (the parser no longer reads offset) and keeping the limit invalid cases, which still apply under the cursor contract.
• Added cursor rejection coverage — the new suite asserts after+before together, unparsable after, and unparsable before are all rejected (400 / AGENT_CONVERSATION_EVENT_INVALID_CURSOR), matching the handler and repository behavior.
• Added keyset-SQL E2E coverage — TestE2EListConversationEventsWindow_CursorBased drives the real event_index >/< queries and the reverse-then-flip-to-ascending ordering against a live database: newest-page open, before paging without duplication, after forward paging, empty page past the tail, and a full backward traversal returning every event exactly once.

Verified against the implementation: parseConversationEventWindowQuery rejects limit out of [1,200] and mutual after+before; cursor decoding lives in the repository and maps ErrConversationEventInvalidCursor to AGENT_CONVERSATION_EVENT_INVALID_CURSOR — which is exactly why the cursor-rejection cases seed a real conversation (via createConversationAt) while the limit/mutual-exclusion cases don't need one. go vet ./test/e2e/... compiles clean, and the suite runs in CI via api-pr-ci.yml's test-e2e job (go test ./test/e2e/... with PACA_E2E=1). This fully closes the previously flagged stale-test concern.

  ^{｜ View workflow run ｜ Using DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏}

[pullfrog]

✅ No new issues found.

Reviewed changes

This incremental review covers commit 92c62ea1 (the delta since prior pullfrog review at 293ce77d), which lands two defensive fixes:

• Hardened the scroll-anchor correction — LoadOlderEvents' ResizeObserver-driven apply() in event-window-controls.tsx now bails while an older-page fetch is in flight, so a resize that isn't the prepend (the reader typing, or the agent streaming at the bottom) no longer yanks the reader's position. The anchor survives and the correction still runs once the prepend commits and isLoadingOlder flips false.
• Made ListChatMessages fail loudly on a non-zero offset — agent_service.go now returns an explicit error instead of silently dropping the offset against the cursor-based ListConversationEvents. The method is unreached by any route (it only satisfies the agentdom.ChatSessionService interface), so this is pure defense with no behavioral risk.

Both changes are well-reasoned and correctly scoped to the incremental delta.

  ^{｜ View workflow run ｜ Using DeepSeek Flash (free via Pullfrog for OSS) ｜ 𝕏}