fix(email-agent): stop meeting/invite answers from inventing what tools never said

[itomek]

Asked about upcoming meetings or calendar invites, the email agent could invent them — attendee names and invite confirmations that exist nowhere in the mailbox or the tool trace. Part of this is misinterpretation, not invention: the agent reads the calendar's real organizer field correctly and then narrates "you sent an invite" from it, which isn't what that field means. The rest is that list_calendar_events/detect_meeting_request never gave the model a structured way to say who's attending or whether an invite was actually sent, so it composed one. Now list_calendar_events/detect_calendar_conflicts surface each event's real attendees (previously discarded), and two deterministic guards catch an invite claimed as sent/received or an attendee named for an event the tool result shows has none — the same "tool computes, model reports" pattern already used for this file's calendar-conflict and attention-card checks. Both guards are negation-aware (an honest "no attendees are listed" or "no invite was sent" is never corrected) and leave a correctly-reported organizer alone. These guards close the two specific fabrication patterns the issue measured (unconfirmed invite claims, invented attendees); they are not a general hallucination filter — a narrower fabrication shape they don't cover is filed separately as #2778.

The REST surface passed the same probe cleanly in the original report while the TUI didn't. Investigation found this is not a TUI-vs-REST code difference — the TUI calls its tools fresh every turn, and the on-open card never reaches the model's context — but multi-turn conversation history: a fresh isolated question stays grounded on both surfaces every time; the same question as a follow-up in an ongoing session is where the fabrication risk lives, on either surface. Full writeup with the probe table: #2766 (comment).

Closes #2766

Test plan

Evidence below is graded on two levels, not one — a guard correcting a fabrication is real progress but is a different (weaker) result than the model never fabricating in the first place:

• PASS — no fabricated invite claim and no fabricated attendee name in the model's own text; the guard never had to fire.

• GUARDED — the model fabricated and a guard appended a correction. Better than main, but not a clean run.

• python -m pytest hub/agents/email/python/tests/ -q — 1739 passed (39 new: 2 new guards' unit + wiring tests, a new attendees-field test file), no regressions

• python -m pytest tests/unit/ -q — 8820 passed, 8 pre-existing failures with zero overlap with the files this PR touches (CLI-binary-on-PATH, hub-wheel-install, Claude-judge API key, VLM PDF extraction — unrelated to calendar_tools.py/answer_grounding.py)

• python util/lint.py --all — clean on every line touched (two pre-existing black/isort drift spots in files this PR touches were left as-is — scope-clean, not missed; hub/ isn't in this repo's CI lint path today)

• Mandatory eval for a tool-docstring change (CLAUDE.md's LLM-affecting-change rule): assessed, not run — gaia eval benchmark (the hermetic email-triage harness) currently scores nothing on any branch, tracked in email-triage benchmark harness scores nothing — stale ctx envelope, id-mismatch scoring join, zero LLM escalation #2776. Live before/after evidence below substitutes.

• TUI + REST evidence, before (main) and after (this branch), 3x multi-turn repeat each, at the issue's own condition — see below

Evidence

The deterministic evidence is the real proof, not the live runs. Two of the 39 new unit tests replay the exact text this issue's own live fabrications produced — "an invite has been confirmed as sent" (the ObjectWin HR text) and "Tomasz Iniewicz sent you invites" — and confirm both are caught and corrected by the new guards. That's repeatable and condition-independent, unlike a live LLM sample. list_calendar_events/detect_calendar_conflicts also now return each event's real attendees ([] for every event in this mailbox) instead of discarding the field, which is what the live runs below show the model actually reading and citing.

Live runs are corroboration, presented with the base rate — not as standalone proof. This defect is stochastic (temp=1.0 sampling) and condition-specific:

• Ryzen AI NPU (gemma4-it-e2b-FLM, ctx 32768, the default profile that hardware resolves to): 0 of 3 fabricated on main, fresh multi-turn, 3 repeats. Doesn't reproduce there at all. Full table: issue comment.
• GPU (Gemma-4-E4B-it-GGUF, ctx 65536 — the issue's own condition), pooled across two machines: 3 of 6 fabricated on main (Radeon 2/3, this Mac 1/3) — roughly a 50% base rate.

Against a 50% base rate, three consecutive clean runs happen by chance alone about 1 time in 8 (0.5³ ≈ 12.5%) even with no fix at all. That is suggestive, not conclusive — stated plainly rather than presented as a clean "3/3, fix confirmed."

Before/after at the issue's own condition — GPU, Gemma-4-E4B-it-GGUF, ctx 65536, confirmed via GET /api/v1/health before each capture. Multi-turn config: turn 1 "Any meetings coming up?", turn 2 (same session) "Did anyone send me a meeting invite?" — the config that reproduces.

Before (main @ e135b8e):

Run	Turn 2 tool	Turn 2 result	Verdict
1	list_calendar_events	"Yes, you have three upcoming meetings/invitations on your calendar, all organized by Tomasz Iniewicz" — organizer misread as invite-sender	FABRICATED
2	search_messages	"no such invitations were found"	PASS
3	search_messages	"did not find any recent incoming meeting invitations"	PASS

1 of 3 fabricated on this machine — confirms reproduction at the issue's documented condition, same mechanism as the original report (real organizer field misread as "sent an invite").

After (issue-2766 @ 2a766c9, same worktree, branch switched in place, TUI rebuilt and daemon/sidecar restarted for each state):

Run	Turn 2 tool	Turn 2 result	Guard fired?	Verdict
1	list_calendar_events	"...none of these events currently list any other attendees in the details provided by your calendar system."	No	PASS
2	list_calendar_events (x2)	"...scheduled with Tomasz Iniewicz... If you were asking about any pending or unaccepted invitation emails... let me know!"	No	PASS (no invite reported, no attendee named — see caveat below)
3	list_calendar_events	"All of these were organized by Tomasz Iniewicz. However, none of the retrieved events currently list any named attendees besides the organizer."	No	PASS

3 of 3 PASS — read against the ~12.5%-by-chance figure above, not as a standalone "fix confirmed." Neither guard fired live in any of the 3 runs (grepped the sidecar log for every guard-fired warning across the whole session: zero matches), so this result cannot be attributed to the guards catching anything — they were never exercised here. What most plausibly explains the clean runs is the structural half of the fix: attendees is now a real, visible [] in the tool's JSON, and all 3 runs explicitly cite it ("no attendees were listed," "none... list any named attendees besides the organizer") rather than silently omitting or inventing one. The docstring wording added alongside it is unproven by this run — a sibling issue in this same batch measured a tool-docstring instruction NOT reliably changing model behavior on this stack when it was the only mechanism, only becoming reliable once made structural, and nothing here contradicts that.

A sharper comparison, conditioning on the tool actually used. The fabrication mechanism this fix targets is specifically the organizer field being misread inside list_calendar_events' own output (run 1 above, and Radeon's earlier run 3). The two clean before-runs called search_messages instead — a tool with no organizer field to misread, so they were never at risk and say nothing about whether the fix works. Conditioning on the tool that can actually exhibit the bug: before, list_calendar_events fabricated 1 of the 1 time it ran on turn 2; after, it fabricated 0 of 3. Still small-n, but it's a like-for-like comparison landing on exactly the field this PR changed, rather than diluted by runs the bug couldn't have touched.

Worth a reviewer's eye, stated as an n=3 observation rather than a claim: tool selection for turn 2 also converged on list_calendar_events after the fix — 1 of 3 before, 3 of 3 after. This PR changed that tool's schema (added attendees) and its docstring, and schema/docstring text is exactly what a tool-calling model routes on. So the docstring half may not be inert after all — it may be steering tool selection rather than preventing fabrication once called, which is a different mechanism than intended and the opposite of what #2763's measurement on this same stack would predict. Flagging this as something to watch, not something to conclude from n=3.

One phrasing worth flagging rather than silently passing: run 2's "scheduled with Tomasz Iniewicz" is loose enough it could be misread as him being a co-attendee rather than the organizer — it doesn't name him in an attendee position or claim an invite, so it doesn't trip either of the issue's precise grading rules, but it's not the tightest possible phrasing either.

Guard 6's assumption, checked

The bot flagged (non-blocking) that guard 6 assumes no tool can confirm a genuinely-received invite. Checked: the assumption holds today — list_calendar_events/detect_calendar_conflicts expose organizer.email but not Google's organizer.self flag, and accept_invite/decline_invite's result envelope carries no invite-provenance signal, so nothing callable today distinguishes "someone else invited you" from "you organized this." create_event_from_email (the guard's one exception) is the opposite direction — an outbound invite the agent sends, not an inbound one it receives.

This is bigger than a rare edge case, so it's filed rather than left as a PR-body note: #2787. organizer.self = false — someone else organized it, i.e. the user was invited — is the majority case for a typical work calendar, not the minority one; it's only invisible here because this PR's reference mailbox happens to be all self-organized (3 of 3 events, organizer.self = true), which is itself a corpus gap. As shipped, guard 6 will disclaim true "you received an invite" statements for most real users' calendars. The fix follows this PR's own pattern exactly: surface organizer.self the same way attendees was surfaced here, then ground guard 6 against it. Scoped out of this PR to avoid destabilizing an already-reviewed change, not because the gap is minor.

[github-actions]

Verdict: Approve with suggestions — safe to merge; one doc-sync gap worth closing first.

This adds two new answer-grounding guards to the email agent (#2766): one that appends a disclaimer when the model claims a calendar invite was "sent/received/confirmed" without a mutation tool actually creating one, and one that flags invented attendee names for events whose real attendees list is empty. It also surfaces each event's real attendees (normalizing Google's omit-the-key-when-empty behavior to []) so the model has something to ground against instead of free-composing names. The guards are append-only, independent, and deliberately conservative — they suppress on negation/hedging and stay silent whenever any listed event genuinely has attendees, so the risk of over-correcting a good answer is low.

The one thing to fix before merge: this is a notable behavioral change to a hub package whose CHANGELOG.md actively tracks fixes by issue number (its [Unreleased] block lists #2756, #2730), yet there's no #2766 entry. Add one so the published package's changelog matches what shipped.

Worth a moment's thought (not blocking): guard 6 assumes no tool can confirm a received invite, so if the mailbox literally contains an invite email and the model correctly says "you received an invite," the disclaimer still gets appended. It's hedged ("may be a proposal") and append-only, so it softens rather than corrupts — just confirm that tradeoff is intended for genuinely-received-invite emails.

Real-world evidence

The evidence bundle (CI ubuntu lane, no inference) exercised the real surfaces this PR changes and is consistent with the diff — this supports the verdict.

• Tool schema actually sent to the model: it constructed EmailTriageAgent and read the live _TOOL_REGISTRY, showing the new attendee/invite guidance reaching the list_calendar_events, detect_calendar_conflicts, and detect_meeting_request descriptions (bundle §2) — not just the source file.
• ground_final_answer request/response: fed a tool trace with attendees: [] plus a model answer fabricating an invite claim and two names; both corrections appended (never replaced) and both WARNING log lines fired on the same turn (bundle §3), matching the composition test.
• Spot regression: two adjacent untouched guards (find_ungrounded_calendar_conflict_claim, find_ungrounded_success_claim) still behaved per their docstrings — no collateral break.
• 135 targeted unit tests pass. Screenshot correctly marked N/A (no Agent UI surface touched); live Lemonade end-to-end deferred to the strix-halo lane. Verdict rests on static review + this CLI/contract evidence for the changed surface.

🔍 Technical details

Issues

🟡 Missing CHANGELOG entry for #2766 (hub/agents/email/python/CHANGELOG.md:8) — the [Unreleased] section documents every notable fix with its issue number (#2756, #2730 immediately above), and CLAUDE.md's doc-sync rule requires a functional change to update the package's CHANGELOG in the same change. This PR ships a user-observable behavior change (new grounding corrections + a new attendees field on calendar tool results) with no entry. Add one under ### Fixed naming #2766, in the same before→after style as the existing entries. No code change required.

🟢 Guard 6 disclaimer on genuinely-received invites (gaia_agent_email/answer_grounding.py:66) — find_ungrounded_invite_claim grounds only on create_event_from_email. An email in the mailbox that is a calendar invite makes "you received an invite" a true statement, but it still gets the "I don't have a way to confirm…" note appended. It's hedged and append-only so the answer isn't corrupted; flagging only to confirm the tradeoff is intended, not a bug to fix now.

Note on the negation heuristic (_CLAUSE_NEGATION_RE, answer_grounding.py:36): including modals (would/will/might/could/should) over the whole clause means a clause like "the invite was sent, and I will follow up" suppresses the guard. This is a conservative miss (leaves prior behavior), consistent with the append-only design, so no change requested — noting only that the false-negative surface is a bit wider than the docstring implies.

Strengths

• New guards mirror the established guard 1–5 shape exactly (find-reason → append-only correction, tools_called_this_turn grounding, __all__ exports), and _extract_attendees correctly normalizes Google's key-omitted-when-empty behavior to [] while passing through only email/response_status — never synthesizing from organizer.
• Genuinely thorough tests: per-helper unit coverage, the "any event with attendees clears it" conservative case, correct-denial and correct-organizer cases (guarding against over-correction), plus the four-guard composition test proving no guard short-circuits another. _extract_attendees is pinned against missing/None/empty/non-mapping inputs.
• Guard 7 runs after guard 6's appended text but the invite-correction contains no attendee vocabulary, and clause bounding keeps each match's negation check inside its own sentence — the ordering is safe.

[itomek]

CHANGELOG entry added and pushed (484a4ef) — [Unreleased] / Fixed, matching the before→after style of the surrounding entries, referencing #2766.

On guard 6's "genuinely-received invite" question: checked, and the assumption holds for what any tool in this package can currently establish. list_calendar_events/detect_calendar_conflicts surface an event's organizer.email but not Google's organizer.self flag, and accept_invite/decline_invite are RSVP mutations whose result envelope ({"event_id", "status"}) carries no invite-provenance signal either — so nothing the model can call today distinguishes "you were invited by someone else" from "you organized this yourself." create_event_from_email (guard 6's one exception) is the opposite direction — an outbound invite the agent sends, not an inbound one it received — so it doesn't touch this case.

One real gap worth naming, not fixed in this PR: Google's raw API does return organizer.self, and if that were ever surfaced, organizer.self == false would be a legitimate, tool-computed signal that an event's invite really was received from someone else. Right now that data is available from the backend but discarded at the same call site attendees used to be, so a genuinely-invited event still gets the disclaimer today, exactly as flagged. Scoping that out of this PR since no event in the reference mailbox currently exercises it (the 3 real events here are all organizer.self=true) — it's a natural next step if a mailbox with real received invites ever needs this guard's precision, not a bug in the current one.

On the negation-heuristic note: agreed with the read — a modal appearing anywhere in the clause (e.g. "...and I will follow up") is a conservative miss, not an over-correction, consistent with the rest of the design. No change made.

[itomek]

Filed the organizer.self gap as its own issue rather than leaving it as a PR-body note: #2787. It's larger than the earlier comment characterized — organizer.self=false (someone else organized it, i.e. genuinely invited) is the majority case for a typical calendar, not a rare one; it only reads as rare here because this PR's reference mailbox happens to be all self-organized, which the issue also calls out as a corpus gap.

[itomek]

Re-verified on a build combining all five PRs from this run (this one plus #2774, #2782, #2783, #2788) — the calendar-attendees fix reaches the model and the user regardless of surface. The two answer-grounding guards are correct as deterministic logic, proven against real fabrication text, but a broader gap this run found (#2789) means their correction currently never reaches a live TUI (or, very likely, Agent UI) user — see below before merging on the current wording.

How it was tested — three repeats, "did anyone send me a meeting invite?" as a follow-up turn

Live GAIA TUI against a real Gmail mailbox (account redacted), integrated build at 7804d3ae. Ground truth confirmed independently via list_calendar_events, bypassing the agent: zero real calendar invites exist — all upcoming events are self-organized with no external attendees.

• Repeat 1: model fabricated an "Invite [was] sent for tomorrow!" claim. The invite-claim guard fired correctly server-side (confirmed in the sidecar log) and computed the intended correction — but that correction never reached the rendered TUI transcript. Traced end to end and filed as Append-style answer_grounding guards never reach the live TUI (correction fires, never renders) #2789 (p1): the SSE relay this agent's live query path uses emits the terminal answer event from inside the base agent loop, before this agent's post-loop grounding pass runs, so the grounding correction mutates a value nobody re-streams. Confirmed this affects the TUI; confirmed it does not affect the standalone playground or a direct process_query() caller (both read the corrected value correctly).
• Repeat 2: PASS — "Based on my search of your mailboxes, there are currently no messages identified as meeting invitations," matching ground truth exactly. No guard needed.
• Repeat 3: verbatim answer opened "Yes, there are several recent emails proposing or discussing meetings!" and every specific claim in the response used "propose"/"request"/"confirm" language — the word "invite" does not appear anywhere in the model's text. Per this PR's own "proposals are not invites" distinction, that's the correct behavior: it never claimed an invite was sent, received, or confirmed. PASS. All 5 cited message IDs verified as real, existing mailbox content — nothing fabricated.

Three repeats, three different outcomes (fabricated-and-guarded, clean, differently-shaped-but-correct) — that instability is itself worth noting: it's what a single clean run can't tell you, and it's consistent with the calendar-attendee logic itself being sound while the surrounding conversation is inherently variable.

The calendar-side fix (list_calendar_events/detect_calendar_conflicts now surfacing real attendees instead of discarding them) reaches the model and the user regardless of the SSE gap above — that's the change that carried the clean runs.

Combined-tree suite: 10636 passed, 8 pre-existing failures (parity-confirmed against clean origin/main), 146 skipped. Lint clean.

Integrated tree: 7804d3ae · this branch's merged commit: 484a4ef9

[itomek]

Closing in favour of #2833, which is the same branch and the same commits reopened from a fork.

itomek no longer has write access to amd/gaia, so this PR's head branch (amd/gaia:issue-2766) can no longer be pushed to — and a PR's head repository can't be changed after it's opened. That left this PR unable to sync with main (it had gone CONFLICTING) or take review fixes.

#2833 carries the identical commits plus a clean merge of current main (#2782/#2788), re-verified at 1762 passed / 4 skipped. The review trail here — the bot's approve-with-suggestions, the guard-6 assumption check, the #2787 follow-up, and the #2789 caveat that gates merging — is repeated in a comment on #2833.