fix(email-agent): stop meeting/invite answers from inventing what tools never said

[itomek]

Supersedes #2784 — same branch, same commits, reopened from a fork because itomek no longer has write access to amd/gaia, which left #2784's head branch unpushable. Review history from #2784 still applies; the merge-blocking caveat from it is repeated in a comment below.

Asked about upcoming meetings or calendar invites, the email agent could invent them — attendee names and invite confirmations that exist nowhere in the mailbox or the tool trace. Part of this is misinterpretation, not invention: the agent reads the calendar's real organizer field correctly and then narrates "you sent an invite" from it, which isn't what that field means. The rest is that list_calendar_events/detect_meeting_request never gave the model a structured way to say who's attending or whether an invite was actually sent, so it composed one. Now list_calendar_events/detect_calendar_conflicts surface each event's real attendees (previously discarded), and two deterministic guards catch an invite claimed as sent/received or an attendee named for an event the tool result shows has none — the same "tool computes, model reports" pattern already used for this file's calendar-conflict and attention-card checks. Both guards are negation-aware (an honest "no attendees are listed" or "no invite was sent" is never corrected) and leave a correctly-reported organizer alone. These guards close the two specific fabrication patterns the issue measured (unconfirmed invite claims, invented attendees); they are not a general hallucination filter — a narrower fabrication shape they don't cover is filed separately as #2778.

The REST surface passed the same probe cleanly in the original report while the TUI didn't. Investigation found this is not a TUI-vs-REST code difference — the TUI calls its tools fresh every turn, and the on-open card never reaches the model's context — but multi-turn conversation history: a fresh isolated question stays grounded on both surfaces every time; the same question as a follow-up in an ongoing session is where the fabrication risk lives, on either surface. Full writeup with the probe table: #2766 (comment).

Closes #2766

Test plan

Evidence below is graded on two levels, not one — a guard correcting a fabrication is real progress but is a different (weaker) result than the model never fabricating in the first place:

• PASS — no fabricated invite claim and no fabricated attendee name in the model's own text; the guard never had to fire.

• GUARDED — the model fabricated and a guard appended a correction. Better than main, but not a clean run.

• python -m pytest hub/agents/email/python/tests/ -q — 1739 passed (39 new: 2 new guards' unit + wiring tests, a new attendees-field test file), no regressions

• python -m pytest tests/unit/ -q — 8820 passed, 8 pre-existing failures with zero overlap with the files this PR touches (CLI-binary-on-PATH, hub-wheel-install, Claude-judge API key, VLM PDF extraction — unrelated to calendar_tools.py/answer_grounding.py)

• python util/lint.py --all — clean on every line touched (two pre-existing black/isort drift spots in files this PR touches were left as-is — scope-clean, not missed; hub/ isn't in this repo's CI lint path today)

• Mandatory eval for a tool-docstring change (CLAUDE.md's LLM-affecting-change rule): assessed, not run — gaia eval benchmark (the hermetic email-triage harness) currently scores nothing on any branch, tracked in email-triage benchmark harness scores nothing — stale ctx envelope, id-mismatch scoring join, zero LLM escalation #2776. Live before/after evidence below substitutes.

• TUI + REST evidence, before (main) and after (this branch), 3x multi-turn repeat each, at the issue's own condition — see below

Evidence

The deterministic evidence is the real proof, not the live runs. Two of the 39 new unit tests replay the exact text this issue's own live fabrications produced — "an invite has been confirmed as sent" (the ObjectWin HR text) and "Tomasz Iniewicz sent you invites" — and confirm both are caught and corrected by the new guards. That's repeatable and condition-independent, unlike a live LLM sample. list_calendar_events/detect_calendar_conflicts also now return each event's real attendees ([] for every event in this mailbox) instead of discarding the field, which is what the live runs below show the model actually reading and citing.

Live runs are corroboration, presented with the base rate — not as standalone proof. This defect is stochastic (temp=1.0 sampling) and condition-specific:

• Ryzen AI NPU (gemma4-it-e2b-FLM, ctx 32768, the default profile that hardware resolves to): 0 of 3 fabricated on main, fresh multi-turn, 3 repeats. Doesn't reproduce there at all. Full table: issue comment.
• GPU (Gemma-4-E4B-it-GGUF, ctx 65536 — the issue's own condition), pooled across two machines: 3 of 6 fabricated on main (Radeon 2/3, this Mac 1/3) — roughly a 50% base rate.

Against a 50% base rate, three consecutive clean runs happen by chance alone about 1 time in 8 (0.5³ ≈ 12.5%) even with no fix at all. That is suggestive, not conclusive — stated plainly rather than presented as a clean "3/3, fix confirmed."

Before/after at the issue's own condition — GPU, Gemma-4-E4B-it-GGUF, ctx 65536, confirmed via GET /api/v1/health before each capture. Multi-turn config: turn 1 "Any meetings coming up?", turn 2 (same session) "Did anyone send me a meeting invite?" — the config that reproduces.

Before (main @ e135b8e):

Run	Turn 2 tool	Turn 2 result	Verdict
1	list_calendar_events	"Yes, you have three upcoming meetings/invitations on your calendar, all organized by Tomasz Iniewicz" — organizer misread as invite-sender	FABRICATED
2	search_messages	"no such invitations were found"	PASS
3	search_messages	"did not find any recent incoming meeting invitations"	PASS

1 of 3 fabricated on this machine — confirms reproduction at the issue's documented condition, same mechanism as the original report (real organizer field misread as "sent an invite").

After (issue-2766 @ 2a766c9, same worktree, branch switched in place, TUI rebuilt and daemon/sidecar restarted for each state):

Run	Turn 2 tool	Turn 2 result	Guard fired?	Verdict
1	list_calendar_events	"...none of these events currently list any other attendees in the details provided by your calendar system."	No	PASS
2	list_calendar_events (x2)	"...scheduled with Tomasz Iniewicz... If you were asking about any pending or unaccepted invitation emails... let me know!"	No	PASS (no invite reported, no attendee named — see caveat below)
3	list_calendar_events	"All of these were organized by Tomasz Iniewicz. However, none of the retrieved events currently list any named attendees besides the organizer."	No	PASS

3 of 3 PASS — read against the ~12.5%-by-chance figure above, not as a standalone "fix confirmed." Neither guard fired live in any of the 3 runs (grepped the sidecar log for every guard-fired warning across the whole session: zero matches), so this result cannot be attributed to the guards catching anything — they were never exercised here. What most plausibly explains the clean runs is the structural half of the fix: attendees is now a real, visible [] in the tool's JSON, and all 3 runs explicitly cite it ("no attendees were listed," "none... list any named attendees besides the organizer") rather than silently omitting or inventing one. The docstring wording added alongside it is unproven by this run — a sibling issue in this same batch measured a tool-docstring instruction NOT reliably changing model behavior on this stack when it was the only mechanism, only becoming reliable once made structural, and nothing here contradicts that.

A sharper comparison, conditioning on the tool actually used. The fabrication mechanism this fix targets is specifically the organizer field being misread inside list_calendar_events' own output (run 1 above, and Radeon's earlier run 3). The two clean before-runs called search_messages instead — a tool with no organizer field to misread, so they were never at risk and say nothing about whether the fix works. Conditioning on the tool that can actually exhibit the bug: before, list_calendar_events fabricated 1 of the 1 time it ran on turn 2; after, it fabricated 0 of 3. Still small-n, but it's a like-for-like comparison landing on exactly the field this PR changed, rather than diluted by runs the bug couldn't have touched.

Worth a reviewer's eye, stated as an n=3 observation rather than a claim: tool selection for turn 2 also converged on list_calendar_events after the fix — 1 of 3 before, 3 of 3 after. This PR changed that tool's schema (added attendees) and its docstring, and schema/docstring text is exactly what a tool-calling model routes on. So the docstring half may not be inert after all — it may be steering tool selection rather than preventing fabrication once called, which is a different mechanism than intended and the opposite of what #2763's measurement on this same stack would predict. Flagging this as something to watch, not something to conclude from n=3.

One phrasing worth flagging rather than silently passing: run 2's "scheduled with Tomasz Iniewicz" is loose enough it could be misread as him being a co-attendee rather than the organizer — it doesn't name him in an attendee position or claim an invite, so it doesn't trip either of the issue's precise grading rules, but it's not the tightest possible phrasing either.

Guard 6's assumption, checked

The bot flagged (non-blocking) that guard 6 assumes no tool can confirm a genuinely-received invite. Checked: the assumption holds today — list_calendar_events/detect_calendar_conflicts expose organizer.email but not Google's organizer.self flag, and accept_invite/decline_invite's result envelope carries no invite-provenance signal, so nothing callable today distinguishes "someone else invited you" from "you organized this." create_event_from_email (the guard's one exception) is the opposite direction — an outbound invite the agent sends, not an inbound one it receives.

This is bigger than a rare edge case, so it's filed rather than left as a PR-body note: #2787. organizer.self = false — someone else organized it, i.e. the user was invited — is the majority case for a typical work calendar, not the minority one; it's only invisible here because this PR's reference mailbox happens to be all self-organized (3 of 3 events, organizer.self = true), which is itself a corpus gap. As shipped, guard 6 will disclaim true "you received an invite" statements for most real users' calendars. The fix follows this PR's own pattern exactly: surface organizer.self the same way attendees was surfaced here, then ground guard 6 against it. Scoped out of this PR to avoid destabilizing an already-reviewed change, not because the gap is minor.

Re-verified after merging main (this PR's head)

main advanced past #2784's last sync (#2782, #2788 both touch this package). Merged upstream/main into the branch — clean, no conflicts, net diff unchanged (still only the 5 files above).

• pytest hub/agents/email/python/tests/ -q — 1762 passed, 4 skipped on the merged head (was 1739 at fix(email-agent): stop meeting/invite answers from inventing what tools never said #2784; the delta is main's own new tests from fix(email-agent): search_messages defaults to metadata-only, fixing context overflow on counting questions #2782/fix(email-agent): get_thread renders a table card instead of relying on model prose #2788, not new tests here)

[itomek]

Carrying forward the review context from #2784, since comments don't move with a reopened PR.

Read this before merging: the structural half of this fix (calendar events now carry their real attendees) reaches the user on every surface, but the two new guards do not. Per #2789, every append-style guard in answer_grounding.py corrects a return value the TUI's streaming path never re-reads — the correction fires, logs, and is then dropped before render. Merging this on its own therefore ships the grounding data but not the visible correction, and risks reading as "fabrication is fixed" when a TUI user would still see the fabricated text. Landing #2789 first makes this PR deliver what it claims.

🔍 The rest of #2784's review trail

• Bot review: "Approve with suggestions" — the one gap it raised (missing CHANGELOG entry) was closed in 484a4ef9, which is on this branch.
• Guard 6's assumption was challenged and checked. It holds for what any tool in this package can establish today: list_calendar_events/detect_calendar_conflicts expose organizer.email but not Google's organizer.self, and accept_invite/decline_invite return {"event_id", "status"} with no provenance signal. So nothing callable distinguishes "someone else invited you" from "you organized this."
• That gap is filed as fix(email-agent): guard 6 disclaims genuinely-received invites — organizer.self not surfaced #2787, not left as a note — organizer.self = false is the majority case on a real work calendar, so as shipped guard 6 will disclaim true "you received an invite" statements for most users. It only reads as rare here because this PR's reference mailbox is all self-organized (3 of 3), which is itself a corpus gap. The fix follows this PR's own pattern: surface organizer.self the way attendees was surfaced here, then ground guard 6 against it.
• Live evidence (3× multi-turn before/after at the issue's own GPU condition) is unchanged and in the description above, including the caveat that neither guard fired in any of the 3 clean after-runs — so those runs evidence the structural change, not the guards.

[github-actions]

Verdict: Approve — clean, well-tested bugfix that follows the email agent's established grounding pattern.

This PR closes #2766: when asked about upcoming meetings or invites, the agent could invent attendee names and claim invites were "sent/received" with nothing behind them. It fixes this two ways — the calendar tools now surface each event's real attendees (normalizing Google's omitted-key case to [] so there's actual data to ground against), and two new deterministic post-processing guards append an honest caveat when the model claims an invite was sent/confirmed with no mutation this turn, or names attendees for an event whose own tool result shows none. Correctly-reported organizers and honest "no attendees" answers are deliberately left alone. The bottom line: nothing blocking — the logic is sound, the guards reuse the existing append-only shape, and test coverage is thorough (per-guard unit tests, impl-level extraction tests, tool-docstring pins, and the four-guard composition test).

Real-world evidence

N/A in this run — no evidence-bundle.md was produced, and I couldn't fetch the PR description here (the shell environment was unavailable, so gh pr view didn't run). The verdict therefore rests on static review plus the committed test suite, which for deterministic post-processing logic like these guards is well-matched. One nudge, not a blocker: the email agent is exposed in the Agent UI, so a before→after screenshot of the "did anyone send me an invite?" flow (or the underlying route evidence the CI lane can produce) would be the ideal artifact to attach — the guards are pure logic backed by strong unit tests, so this is a nice-to-have for the record, not a merge gate.

🔍 Technical details

Strengths

• Correct pattern reuse: both guards follow the existing append-only guard shape and are wired into ground_final_answer without an early return, and the composition test (test_all_four_append_guards_fire_on_one_turn) pins the invariant that all four append guards fire in sequence — exactly the regression class called out in the file's own docstring.
• _extract_attendees (calendar_tools.py:292) returns only backend data (email/response_status), drops entries without an email, skips non-mapping rows, and never re-derives from organizer — no silent synthesis. Fail-loudly is respected throughout (no swallowed exceptions, corrections append rather than mask).
• Guards are conservative in the right direction: negation/hedging in the same clause suppresses, and the attendee guard stays silent when any listed event carries attendees (test_any_event_with_attendees_across_multiple_clears_it), avoiding false corrections.
• Doc-sync is satisfied: CHANGELOG.md is updated, and SPEC.md/SKILL.md/README.md don't document the calendar tool response fields or the individual guards, so adding attendees contradicts no shipped doc.

🟢 Minor — overlap with the success-claim guard (answer_grounding.py:697 vs :251)
find_ungrounded_success_claim matches was sent, so at the orchestration level the phrase "The invite was sent yesterday." is caught and replaced by that guard before find_ungrounded_invite_claim runs (the invite guard's own unit test exercises it in isolation, where it does fire). This isn't a bug — the answer is still corrected, arguably more strongly — but worth a one-line comment near guard 6 noting that was/has-been sent is already handled upstream and the invite guard is the backstop for the received/confirmed as sent/proposal phrasings the success guard misses.

🟢 Minor — invitation isn't matched. _INVITE_CLAIM_RE uses \binvite[sd]?\b, which covers invite/invites/invited but not invitation (e.g. "you received an invitation"). The guard is intentionally narrow, so this is a coverage gap, not a defect — mentioning only in case the reference corpus contains that phrasing.

Nothing else. Nits are within cap; no security concerns; no architecture/convention violations.

[itomek]

Live TUI evidence on the merged head, at the issue's own condition. PASS, with the grading caveat spelled out below rather than rounded up.

Setup: real Gmail mailbox, GAIA TUI built from this branch (688ee47a), email sidecar in dev mode on this branch's source, Lemonade 11.5.1 serving Gemma-4-E4B-it-GGUF on Metal GPU. Fresh session, two turns.

Turn 1 — Any meetings coming up? → routed to list_calendar_events (36.1s, 2 steps, 1 tool):

You have three meetings coming up in your calendar:
• Northgate migration sync — phase 2 sequencing: Tuesday, August 5th at 10:30 AM – 11:30 AM EDT
• Design review — revised dashboard concepts: Wednesday, August 6th at 9:00 AM – 10:00 AM EDT
• Board meeting — Q4 deck: Saturday, August 13th at 2:00 PM – 4:00 PM EDT

No invented attendee names, no invite claim, no organizer misread. This is the tool that can actually exhibit the defect, and it didn't.

Turn 2 — Did anyone send me a meeting invite? → routed to detect_waiting_on_you (21.7s, 2 steps, 1 tool):

Yes, I found one message in your inbox that requires your attention—it looks like a direct ask rather than a formal calendar invite: … Subject: Re: Fieldstone MSA — counter-signature needed before Friday

That is the exact discrimination #2766 is about, made explicitly and correctly.

Neither guard fired — grepped the sidecar log across the whole session for every guard-fired warning: zero matches. So, as with the runs already in the description, this evidences the structural half of the fix (real attendees in the tool payload), not the guards.

How to read the grade, honestly. Turn 2 routed to detect_waiting_on_you, not to a calendar tool — so like the two clean search_messages before-runs in the description, it was never at risk of the organizer-misread mechanism. The like-for-like signal here is turn 1's list_calendar_events read, which was clean. One run, so it moves the needle rather than settling it.

Tool trace from the sidecar log

10:16:49  tool_call   name=list_calendar_events
10:16:49  tool_result name=list_calendar_events ok=True latency=245ms
10:22:24  tool_call   name=detect_waiting_on_you
10:22:35  tool_result name=detect_waiting_on_you ok=True latency=11037ms

Unrelated to this PR but found in the same session and worth linking so it isn't rediscovered: #2834 — a thread found in one turn can't be referenced in the next, because no tool result carries its id into the TUI's context.