fix(email-agent): search_messages states an exact, stable message count - #2760
Conversation
…ator_retry (#2756) Asserts the registered search_messages tool's envelope carries an exact count, a truncated flag derived only from Gmail's real nextPageToken (never a len==max_results heuristic), and a forwarded operator_retry. Also asserts the docstring instructs the model to trust the precomputed count and enumerate every returned entry.
…nt (#2756) search_messages returned a correct, complete result set and the model then reported a wrong, unstable aggregate over it -- 12 real messages stated as 6 on one run and 4 on the next. The registered search_messages tool's merge closure now precomputes count and truncated instead of leaving the model to tally the messages list itself, the same remedy check_followups already shipped for the identical defect (#2622). truncated comes only from Gmail's real nextPageToken, never from len(messages) == max_results -- that heuristic reports a false positive the moment a mailbox's true size exactly equals the request. operator_retry, computed by search_messages_impl since inception but discarded by the wrapper before it ever reached the model, is now forwarded too, so a broadened retry query is disclosed instead of presented as the user's literal search. Multi-mailbox aggregation semantics are untouched -- count/truncated/ operator_retry are combined per backend without redesigning that merge, consistent with the current single-mailbox (Gmail) scope.
|
The unfixed case from this PR's scope section is now tracked as #2763 (p0) — |
Live-inference evidence (AC7) — count fixed 3/3, enumeration still fails 10/12Run against this branch's code, not Ground truth re-pulled immediately before the runs, via Count — PASS, 3 of 3
Before this change, the same probe on the same box and model stated 6, then 4. Three consecutive exact answers, including one run where the model widened Enumeration — FAIL, unchanged at 10 of 12A It also degraded in a new way — the answer now contains visible self-correction and an invented row:
There is no Jul 15 message in the result set. That is fabrication, not omission. So the REPORT-EVERY-ENTRY half of the remedy did not transfer. On What this means for mergingThe count defect in #2756 is fixed and evidenced. Enumeration completeness is not fixed by this PR and should not be claimed. Recommend either narrowing this PR's stated scope to the count (the enumeration gap then needs its own issue, alongside #2763), or holding it until the enumeration path is addressed — maintainer's call. |
Closes #2756
Ask the email agent "how many messages from X in the last two weeks?" and it returned a wrong number — and a different wrong number each time. Measured on a real Gmail mailbox: 12 real messages reported as 6, then as 4, while the same session could correctly list 10 of them by name. TechCrunch: 19 real, reported as 11, then 14. The search was always correct and the full result set always reached the model; it simply miscounted a list it had in front of it. The tool now counts the results itself and hands the model a finished number to state verbatim — the same remedy that fixed this exact defect in
check_followups(#2622) and was never ported here.The one-line version of the bug: the registered
search_messagestool rebuilt its envelope from scratch (out = {"messages": ...}), reading onlymessagesoff the implementation and silently discarding everything else.operator_retryhad been computed since the tool was written and has never once reached the model as a result. The fix lives at that merge layer, not insearch_messages_impl— an implementation-only change would have been a no-op.Test plan
pytest hub/agents/email/python/tests/test_search_messages_count_2756.py— 7 new tests, written red-first against the registered tool (_TOOL_REGISTRY["search_messages"]), not the impl, since testing the impl in isolation false-greens exactly the gap above. Initial run failed withKeyError: 'count'/KeyError: 'operator_retry'; now 7 passed.pytest hub/agents/email/python/tests/— 1633 passed.python util/lint.py --all— 10/11 pass, 1 pre-existing MyPy warning (none in touched files, confirmed by diffing againstmain).from:"The Neuron" newer_than:14d(true 12) must state 12 on 3 consecutive runs, with a follow-up enumerating all 12 rows.Why
truncatedis notlen(stubs) == max_resultsIt is
bool(listing.get("nextPageToken")), from the provider's real cursor. This file already forbids the length heuristic in its own words atread_tools.py:994-998— "honest only by coincidence and wrong the moment a mailbox's true size exactly equals the request" — and implements the correct form at:1026-1034. A sender with exactly 25 matches and no next page would otherwise be reported as "at least 25", a worse answer than the tool already had the data to give. There is a regression test for precisely that case.next_page_tokenis deliberately not added to the envelope: no tool in this file accepts apage_tokenargument, so a propagated token is unconsumable, and merging tokens across two backends is a designlist_inboxnever solved (its wrapper hardcodes"next_page_token": None). Deferred with multi-mailbox aggregation.Evidence this remedy works on this model class
The identical fix was validated on
check_followupson 2026-08-03 (Gemma-4-E4B-it-GGUF / GPU / ctx 65536, Gmail-only): 22 stated and 22 enumerated on 6 of 6 runs — 3 REST, 3 TUI — against a true 22. That same tool reported 13 / ~19 / ~15–16 against that same 22 in July, so #2622 is confirmed genuinely fixed rather than presumed, and the precomputed-countapproach demonstrably reaches this model and is used.Scope — what this does NOT fix
A separate, reproducible defect remains on this tool: for a large-body sender the agent returns no answer at all. Verified on GPU/Gmail with
from:Every newer_than:14d(15 messages, both phrasings, model-chosenmax_resultsof 100 and 50): the tool call succeeds, then the agent hits the context-overflow fallback atsrc/gaia/agents/base/agent.py:3758— "I had to trim the conversation to fit my memory…". A precomputed count cannot help a turn that never reaches the answering stage. Reproduced independently on two machines and two OSes, so it is not platform-specific.That is tracked separately and is not addressed here. The underlying design smell is worth stating:
search_messagesships full 4000-char message bodies for questions like "how many emails from X" that need zero body bytes. A metadata-only path would remove the overflow failure mode outright rather than mitigate it.Multi-mailbox aggregation semantics (
count/truncatedacross two backends, partial-failure counts) are likewise out of scope — this ran Gmail-only. The wrapper ORstruncatedand documents thatcountcovers only surviving mailboxes whenmailbox_errorsis present, so it is honest, but the contract is not designed here.