Skip to content

fix(email-agent): search_messages states an exact, stable message count - #2760

Merged
itomek merged 2 commits into
mainfrom
issue-2756
Aug 3, 2026
Merged

fix(email-agent): search_messages states an exact, stable message count#2760
itomek merged 2 commits into
mainfrom
issue-2756

Conversation

@itomek

@itomek itomek commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #2756

Ask the email agent "how many messages from X in the last two weeks?" and it returned a wrong number — and a different wrong number each time. Measured on a real Gmail mailbox: 12 real messages reported as 6, then as 4, while the same session could correctly list 10 of them by name. TechCrunch: 19 real, reported as 11, then 14. The search was always correct and the full result set always reached the model; it simply miscounted a list it had in front of it. The tool now counts the results itself and hands the model a finished number to state verbatim — the same remedy that fixed this exact defect in check_followups (#2622) and was never ported here.

The one-line version of the bug: the registered search_messages tool rebuilt its envelope from scratch (out = {"messages": ...}), reading only messages off the implementation and silently discarding everything else. operator_retry had been computed since the tool was written and has never once reached the model as a result. The fix lives at that merge layer, not in search_messages_impl — an implementation-only change would have been a no-op.

Test plan

  • pytest hub/agents/email/python/tests/test_search_messages_count_2756.py — 7 new tests, written red-first against the registered tool (_TOOL_REGISTRY["search_messages"]), not the impl, since testing the impl in isolation false-greens exactly the gap above. Initial run failed with KeyError: 'count' / KeyError: 'operator_retry'; now 7 passed.
  • pytest hub/agents/email/python/tests/ — 1633 passed.
  • python util/lint.py --all — 10/11 pass, 1 pre-existing MyPy warning (none in touched files, confirmed by diffing against main).
  • Live-inference check (blocked, see below). On a real Gmail mailbox, GPU: from:"The Neuron" newer_than:14d (true 12) must state 12 on 3 consecutive runs, with a follow-up enumerating all 12 rows.
Why truncated is not len(stubs) == max_results

It is bool(listing.get("nextPageToken")), from the provider's real cursor. This file already forbids the length heuristic in its own words at read_tools.py:994-998"honest only by coincidence and wrong the moment a mailbox's true size exactly equals the request" — and implements the correct form at :1026-1034. A sender with exactly 25 matches and no next page would otherwise be reported as "at least 25", a worse answer than the tool already had the data to give. There is a regression test for precisely that case.

next_page_token is deliberately not added to the envelope: no tool in this file accepts a page_token argument, so a propagated token is unconsumable, and merging tokens across two backends is a design list_inbox never solved (its wrapper hardcodes "next_page_token": None). Deferred with multi-mailbox aggregation.

Evidence this remedy works on this model class

The identical fix was validated on check_followups on 2026-08-03 (Gemma-4-E4B-it-GGUF / GPU / ctx 65536, Gmail-only): 22 stated and 22 enumerated on 6 of 6 runs — 3 REST, 3 TUI — against a true 22. That same tool reported 13 / ~19 / ~15–16 against that same 22 in July, so #2622 is confirmed genuinely fixed rather than presumed, and the precomputed-count approach demonstrably reaches this model and is used.

Scope — what this does NOT fix

A separate, reproducible defect remains on this tool: for a large-body sender the agent returns no answer at all. Verified on GPU/Gmail with from:Every newer_than:14d (15 messages, both phrasings, model-chosen max_results of 100 and 50): the tool call succeeds, then the agent hits the context-overflow fallback at src/gaia/agents/base/agent.py:3758"I had to trim the conversation to fit my memory…". A precomputed count cannot help a turn that never reaches the answering stage. Reproduced independently on two machines and two OSes, so it is not platform-specific.

That is tracked separately and is not addressed here. The underlying design smell is worth stating: search_messages ships full 4000-char message bodies for questions like "how many emails from X" that need zero body bytes. A metadata-only path would remove the overflow failure mode outright rather than mitigate it.

Multi-mailbox aggregation semantics (count/truncated across two backends, partial-failure counts) are likewise out of scope — this ran Gmail-only. The wrapper ORs truncated and documents that count covers only surviving mailboxes when mailbox_errors is present, so it is honest, but the contract is not designed here.

itomek added 2 commits August 3, 2026 17:50
…ator_retry (#2756)

Asserts the registered search_messages tool's envelope carries an exact
count, a truncated flag derived only from Gmail's real nextPageToken
(never a len==max_results heuristic), and a forwarded operator_retry.
Also asserts the docstring instructs the model to trust the precomputed
count and enumerate every returned entry.
…nt (#2756)

search_messages returned a correct, complete result set and the model
then reported a wrong, unstable aggregate over it -- 12 real messages
stated as 6 on one run and 4 on the next. The registered search_messages
tool's merge closure now precomputes count and truncated instead of
leaving the model to tally the messages list itself, the same remedy
check_followups already shipped for the identical defect (#2622).

truncated comes only from Gmail's real nextPageToken, never from
len(messages) == max_results -- that heuristic reports a false positive
the moment a mailbox's true size exactly equals the request. operator_retry,
computed by search_messages_impl since inception but discarded by the
wrapper before it ever reached the model, is now forwarded too, so a
broadened retry query is disclosed instead of presented as the user's
literal search.

Multi-mailbox aggregation semantics are untouched -- count/truncated/
operator_retry are combined per backend without redesigning that merge,
consistent with the current single-mailbox (Gmail) scope.
@github-actions github-actions Bot added the agent::email Email agent changes label Aug 3, 2026
@itomek
itomek marked this pull request as ready for review August 3, 2026 22:18
@itomek
itomek requested a review from kovtcharov-amd as a code owner August 3, 2026 22:18
@itomek

itomek commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

The unfixed case from this PR's scope section is now tracked as #2763 (p0) — search_messages returning no answer at all for a long-bodied sender, reproduced 8/8 across macOS/Metal and Linux/vulkan. It is a distinct defect: a precomputed count cannot help a turn that never reaches the answering stage.

@itomek

itomek commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

Live-inference evidence (AC7) — count fixed 3/3, enumeration still fails 10/12

Run against this branch's code, not main: daemon + email sidecar restarted from the PR worktree (source: confirmed pointing at it, api v2.10). Runtime-resolved model, verified at run time rather than assumed: Gemma-4-E4B-it-GGUF, device: gpu, ctx 65536. Gmail-only (Microsoft not configured), so single backend and no multi-mailbox merge involved.

Ground truth re-pulled immediately before the runs, via POST /v1/email/search with the same query string: 12.

Count — PASS, 3 of 3

run tool args the model chose stated
1 {"query": "from:'The Neuron' newer_than:14d"} 12
2 {"max_results": "50", "query": "from:\"The Neuron\" newer_than:14d"} 12
3 {"query": "from:\"The Neuron\" newer_than:14d"} 12

Before this change, the same probe on the same box and model stated 6, then 4. Three consecutive exact answers, including one run where the model widened max_results to 50 on its own. The precomputed count works.

Enumeration — FAIL, unchanged at 10 of 12

A list every message … follow-up returned 10 real rows (Aug 3 → Jul 24, plus one mis-dated Jul 22), the same 10 of 12 measured before the fix. The two genuine Jul 22 messages are still missing, and One Gemini Became Three is dated Jul 22 when its real header is Jul 23.

It also degraded in a new way — the answer now contains visible self-correction and an invented row:

Wed, Jul 15 (Note: This date is outside the last two weeks based on the provided results) - Wait, I see a message from July. Let me recheck my dates…

There is no Jul 15 message in the result set. That is fabrication, not omission.

So the REPORT-EVERY-ENTRY half of the remedy did not transfer. On check_followups the identical wording held enumeration complete at 22/22 across 6 runs; here it does not. The likely discriminator is payload shape — check_followups ships compact metadata rows, search_messages ships 4000-char bodies, so the model is enumerating from a far larger and noisier context. That is consistent with #2763, where the same body weight tips into outright context overflow for a longer-bodied sender.

What this means for merging

The count defect in #2756 is fixed and evidenced. Enumeration completeness is not fixed by this PR and should not be claimed. Recommend either narrowing this PR's stated scope to the count (the enumeration gap then needs its own issue, alongside #2763), or holding it until the enumeration path is addressed — maintainer's call.

@itomek
itomek enabled auto-merge August 3, 2026 22:29
@itomek itomek self-assigned this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent::email Email agent changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(email-agent): search_messages states a wrong, unstable count for an intact result set

2 participants