fix(sidebar): make each Load more click reveal newly fetched sessions

[ShiboSheng]

Description

While validating #555, the independent SDE Agent and Agent Org backend streams worked as intended, but a separate sidebar pagination problem became visible:

• Clicking Load more sometimes reveals sessions immediately.
• Other times, the same button must be clicked two or three times before any new session row appears.
• A Load more click shown under one agent section can add sessions to the Pinned section instead.
• After startup or a roster refresh, Load more can replay pages that are already present in the local cache before it reaches genuinely new sessions.

The click is usually not lost. The backend page can load successfully while every returned row is already present in the local cache, remains hidden by the frontend's per-section display limit, or is regrouped into a different section.

Steps to Reproduce

1. Open the session sidebar and group sessions by agent.
2. Have enough sessions for a backend category to contain multiple pages, with some sessions pinned.
3. Expand the visible local rows until the agent section is exactly at its current display limit.
4. Click that section's Load more button while the backend category still has another page.
5. Observe that the loading request completes, but sometimes no new row appears in the clicked section.
6. Click Load more again.
7. The second click expands the already-loaded local rows and the sessions finally become visible.
8. In some page distributions, the first click increases the Pinned section rather than the section where the button was displayed.

A minimal deterministic fixture is:

• The unpinned agent group has 20 loaded rows and a visibleCount of 20.
• The pinned group has 10 loaded rows and a visibleCount of 10.
• The backend category has more rows.
• The next backend page returns 9 unpinned rows and 1 pinned row.

After the backend click, the atom contains 29 unpinned and 11 pinned rows, but the UI still renders only 20 and 10. No newly fetched row is visible until a second local expansion click.

Expected Behavior

• One enabled Load more click should produce an immediate, visible result when matching rows exist.
• A button presented under a specific section should have clear ownership of the rows it loads.
• Loading more unpinned sessions for an agent section should not silently add rows only to Pinned.
• The UI should not require a second click merely to reveal rows fetched by the first click.
• When no matching rows remain, the button should disappear or transition to an explicit exhausted state.

Actual Behavior

The sidebar currently has two pagination layers that use the same Load more label:

1. Local reveal pagination increments groupVisibleCounts by the sidebar page size.
2. Backend category pagination advances a backend cursor and merges another page into the global session atom.

After a backend page is merged, the frontend:

• splits sessions into pinned and unpinned arrays;
• regroups unpinned sessions by agent, Agent Org, workspace, or date;
• applies the old local visibleCount to every destination group.

The backend fetch does not increase those destination groups' visible counts. Therefore, newly fetched rows can be present in state but remain entirely invisible.

Root Cause

1. Backend and local pagination do not share one owner

A backend pager is rendered when a category has no currently hidden local rows. After that pager fetches another page, the same group can immediately acquire hidden local rows because its visibleCount is unchanged. The row then changes from a backend pager into a local reveal pager, even though both controls look identical to the user.

Relevant areas:

• src/scaffold/NavigationSidebar/connectors/useWorkstationSidebarHandlers.ts
• src/scaffold/NavigationSidebar/connectors/useSessionMenuItems/paginationHelpers.tsx
• src/scaffold/NavigationSidebar/connectors/useSessionMenuItems/menuSectionBuilders.ts
• src/scaffold/NavigationSidebar/connectors/useSessionMenuItems/index.tsx
• src/store/session/sessionAtom/loaders.ts
• src/store/session/sessionAtom/paginationAtoms.ts

2. Pinned rows are included in category pages

Native sidebar SQL pages include the persisted pinned value but do not exclude pinned rows before LIMIT/OFFSET. After the page reaches the frontend, pinned rows are removed from the clicked agent group and inserted into the global Pinned section.

Relevant backend areas:

• src-tauri/crates/agent-core/src/core/coordination/agent_org_runs/store.rs
• src-tauri/src/agent_sessions/session_directory/aggregation.rs

3. Some backend categories are wider than the visible UI group

A single backend cursor can cover more than one visible group:

• standalone_agent can represent SDE, Wingman, and Custom Agent groups.
• agent_org_root can represent multiple Agent Org groups.

A button visually placed below one group can therefore fetch rows belonging to another group. This is another way for the clicked section to remain unchanged even though the request succeeded.

4. The local cache and backend cursor can describe different windows

The sidebar restores up to 200 session rows from local storage so it can render immediately after startup. A roster refresh resets each backend pagination cursor to the first page, but the refreshed native rows are merged into the existing session atom instead of replacing the cached native window.

That can leave the frontend in a state such as:

• the atom already contains 30 standalone sessions restored from local cache;
• standalone_agent.loaded has been reset to 10 after refreshing page 1;
• the backend therefore believes the next page starts at offset 10.

The next two Load more clicks request offsets 10 and 20. Those pages contain rows that are already present in the atom, so ID-based merging removes the duplicates and the rendered list does not change. Only the request at offset 30 reaches genuinely new rows.

This is another deterministic reason one click can appear to do nothing. It is separate from the local visibleCount problem: in this case the request may add no new atom rows at all because the backend cursor is catching up with a larger cached window.

Relevant areas:

• src/store/session/sessionAtom/atoms.ts
• src/store/session/sessionAtom/persistence.ts
• src/store/session/sessionAtom/loaders.ts

Affected Session Categories

These root causes do not affect every visible sidebar label in exactly the same way.

Visible sidebar section	Cached pages can be replayed after cursor reset	Newly fetched rows can remain hidden behind the local display limit	Rows can move to Pinned	One backend page can add rows under another visible label
SDE / Wingman / Custom Agent	Yes	Yes	Yes	Yes — they share the standalone_agent backend stream
Individual Agent Org groups	Yes	Yes	Yes	Yes — several Org groups share the agent_org_root backend stream
CLI Agent	Yes	Yes	Yes	No — it has its own cli_agent category
OS Agent	Yes	Yes	Yes	No — it has its own os_agent category
Work Logs / human sessions	Yes	Yes	Possible when the row is pinned	No — it has its own human_session category
Imported history such as Codex App, Claude Code, Cursor, and OpenCode	Not from the exact native cache/cursor mismatch described above: refresh replaces that source's previous top-level rows and pagination is tracked per source/date bucket	Yes	No in the current imported-history sidebar payload	No — each imported source has its own category

In plain language:

• The cache/cursor replay bug primarily affects ORGII-managed native categories: standalone agents, Agent Org roots, CLI, OS, and human sessions.
• The local display-limit bug can affect every section, including Codex and Claude Code: the frontend may receive new rows but continue showing only the previous number of rows.
• The cross-label ownership bug is concentrated in shared backend streams: SDE/Wingman/Custom and multiple Agent Org groups.
• Codex, Claude Code, Cursor, OpenCode, and other imported-history sources have independent source pagination. They still need one-click visibility coverage, but they should not be described as having the same native cache/cursor replay mechanism.

Evidence

A read-only inspection of one real local dataset reproduced the issue with this standalone-agent page distribution. No session names, IDs, prompts, or transcript content were inspected.

Backend page	Pinned rows	Unpinned rows	Cumulative pinned	Cumulative unpinned
1	5	5	5	5
2	4	6	9	11
3	1	9	10	20
4	1	9	11	29

After page 3, the local limits can be exactly Pinned=10 and SDE=20. Loading page 4 succeeds, but all 10 returned rows land beyond those limits. The screen shows no new session row until another local expansion click.

A second deterministic fixture covers the cache/cursor mismatch:

• restore 30 native rows from local storage;
• refresh the roster with a backend page size of 10;
• verify that the atom still contains 30 rows while the cursor reports 10 loaded rows;
• the first Load more requests offset 10 and adds 0 unique rows;
• the second requests offset 20 and adds 0 unique rows;
• the third requests offset 30 and finally adds new rows.

The existing focused test suites all pass (27 tests across pagination helpers, menu section builders, and sidebar loaders). They verify the loader and menu builder separately, but do not exercise the full sequence:

backend Load more click -> merge -> regroup -> local visibleCount -> rendered rows

There is also coverage that intentionally keeps an Agent Org backend pager when all loaded roots are pinned, but no assertion that the next click visibly updates the intended section.

Suggested Resolution

The implementation can vary, but pagination ownership should become unambiguous:

1. Give the Pinned section its own pagination behavior, or exclude pinned rows from unpinned section queries before LIMIT/OFFSET.
2. Make one visible Load more control correspond to one well-defined data stream.
3. When a backend page returns, immediately reveal the newly fetched rows in their destination group(s), or remove the second local pagination layer for that path.
4. If a backend cursor is intentionally shared by several visible groups, place the control at a neutral category-level footer rather than implying that it belongs to one label.
5. Preserve bounded page sizes, independent cursors, generation guards, and deterministic ordering.
6. Keep the backend cursor and retained native cache on the same window: either replace the relevant native window on refresh, restore a cursor consistent with the retained rows, or track page membership explicitly instead of inferring it from one loaded count.

This should not require a database schema change. The likely changes are query filtering/stream ownership plus frontend pagination and visibility state.

Acceptance Criteria

• Every enabled Load more click either reveals at least one newly available row immediately or clearly transitions the control to loading/exhausted/error state.
• A successful backend page cannot be completely hidden solely because the destination group's previous visibleCount was already full.
• Loading more for an unpinned agent section cannot produce only a silent increase in the Pinned section.
• Pinned pagination cannot consume the visible capacity or cursor of an unpinned section.
• A pager displayed under one agent label cannot silently load rows only for another label; shared-category pagers use a neutral placement or exact group filtering.
• SDE/Wingman/Custom and multiple Agent Org groups are covered explicitly.
• Native cache/cursor coverage includes standalone agents, Agent Org roots, CLI, OS, and human sessions.
• Imported-history coverage includes Codex App, Claude Code, and Cursor IDE, proving that one source-local Load more click reveals its newly fetched rows without relying on a second local reveal click.
• After cold-start hydration or a forced roster refresh, the next enabled Load more request cannot consist entirely of rows already retained from later cached pages.
• Tests cover a cached native list larger than page 1, followed by cursor reset and repeated loads.
• Repeated loads do not skip or duplicate rows.
• Rapid repeated clicks while loading=true do not start overlapping requests.
• Equivalent behavior is checked for by-agent, by-time, and by-workspace sidebar modes.
• Tests cover the integrated fetch/merge/regroup/render sequence, not only the loader and menu builder in isolation.
• A strict rendered E2E test proves that one click makes the expected new row visible and fails if the row appears only after another click.

Environment

• OS: macOS
• App version: v1.2.3 issue fix(sidebar): load more SDE Agent and Agent Org sessions separately #555 test build
• Grouping mode: By Agent
• Related issue: fix(sidebar): load more SDE Agent and Agent Org sessions separately #555

Additional Context

This is related to the pagination work in #555 but is a distinct problem. #555 separates normal standalone sessions from Agent Org roots correctly; this issue concerns coordination between backend category pagination, pinned regrouping, local per-section visibility limits, and restored cache state.