test(gemini): live cassette hook-system stress suite

[gold-silver-copper]

Summary

A Gemini cassette-backed stress-test suite for the merged hook system (v2, #2012): 30 long, realistic multi-turn workflows recorded against a real Gemini model and replayed deterministically (no API key needed for replay). Five cassette files under tests/providers/gemini/cassette/ (hook_stress, hook_stress_context, hook_stress_patch, hook_stress_tools, hook_stress_streaming) + 31 cassette fixtures, driven by a shared deterministic-fixtures module (hook_stress_support.rs). No production code changes.

Inspiration-library test patterns studied

I studied how six libraries test their hook/middleware/callback/guardrail systems and adopted the transferable assertion strategies (a recorded real Gemini is the deterministic script the fake-model scripting can't be):

Library	What their hook tests stress	Strategy adopted
LangChain (v1 agent middleware)	onion order, override immutability, tool arg-rewrite/skip, PII redaction	Paired positive+negative redaction; ordered call-log
LangGraph (prebuilt/pregel)	pre/post-model hooks, tool interceptor Flow matrix	Prove mutations via downstream observable effect (Gemini won't echo settings)
OpenAI Agents (lifecycle/guardrail)	callback counts+order, guardrail short-circuit	Zero-downstream invariant for skip/terminate; per-event scratchpad counters
pydantic-ai (capabilities/TestModel)	log-spy hook, capture-what-reached-the-model	The cassette is the deterministic multi-turn script
Semantic Kernel (filters)	registration-order breadcrumb, skip zero-downstream	Execution-order breadcrumb (event, turn); paired counters
Vercel AI SDK (generate/stream-text)	multi-step loop, per-step callbacks, override precedence	Ordered lifecycle event list; matrix the RequestPatch merge/precedence

What the new Gemini hook stress tests cover

hook_stress.rs (6) — the original core: HookContext + Scratchpad across a long chain; RequestPatch extra_context + active_tools narrowing; chained RewriteArgs→observe→RewriteResult redaction; streaming lifecycle ordering + is_streaming parity; per-turn atomic call/result pairing; Skip zero-execution.

hook_stress_context.rs (8) — HookContext identity (stable run_id, advancing turn, is_streaming, agent_name incl. the unset case); a shared Scratchpad written by one hook and read by a second, growing across turns; internal_call_id correlation of ToolCall↔ToolResult; two observe-only hooks both firing; add_hook appending across builder + request; CompletionCall patch accumulation; active_tools set-intersection across two narrowing hooks.

hook_stress_patch.rs (4) — preamble override; tool_choice=Required (first turn only); per-turn history replacement injecting a prior fact; a multi-field patch (preamble + extra_context).

hook_stress_tools.rs (6) — single-key and chained RewriteArgs; chained RewriteResult (redact→wrap) and truncation; Terminate from a ToolResult (post-execution — the tool body ran first); and model-driven recovery from a tool error (the tool errors with guidance → the model retries → succeeds).

hook_stress_streaming.rs (6) — TextDelta / StreamResponseFinish / ModelTurnFinished on the streaming surface; ModelTurnFinished on tool turns; RewriteResult redaction reaching the FinalResponse; active_tools narrowing and Skip on the streaming driver; and blocking-vs-streaming answer parity (two cassettes).

Aspects covered: HookContext (all accessors + unset name), Scratchpad (cross-hook, cross-turn, keyed correlation), HookStack composition (multi-hook order, both-fire, add_hook append, chained rewrites, patch accumulation, active_tools intersection), RequestPatch (preamble / temperature / tool_choice / active_tools / extra_context / history), tool lifecycle (arg rewrite / result redact-wrap-truncate / skip / terminate-on-result / tool-error recovery / execution start), streaming lifecycle + blocking-vs-streaming parity.

Bugs found and fixed

No hook-system bugs. Every scenario passed on live recording, confirming the merged system behaves as documented. One footgun surfaced: forcing tool_choice=Required on every turn loops until max_turns (each turn re-forces a tool call). It's expected per the per-turn/re-fire semantics, so rather than change production code I captured the intended pattern with a FirstTurnPatch fixture and a comment. No production code changed.

Cassette safety

• Recorded with RIG_PROVIDER_TEST_MODE=record + GEMINI_API_KEY, then auto-scrubbed and safety-checked by the recorder before writing (the run fails if unsafe artifacts remain).
• Manually reviewed: key query param → [REDACTED]; thoughtSignature/responseId → placeholders; grep across all fixtures for x-goog-api-key/gemini_api_key/AIza/authorization/bearer/set-cookie and the raw key value all return empty.
• Descriptive names, deterministic prompts (low temperature, simple arithmetic tools), targeted record commands only.

Commands run

• Record (targeted, live): RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test gemini hook_stress_<mod> -- --test-threads=1 → all passed.
• Replay (no key): cargo test -p rig --all-features --test gemini hook_stress -- --test-threads=1 → 30 passed (0.18s, deterministic).
• Full provider suite: cargo test -p rig --all-features --test gemini gemini::cassette -- --test-threads=1 → 115 passed, 0 failed.
• cargo fmt (clean) · cargo clippy -p rig --all-features --test gemini (clean).

Remaining risks / flakiness considerations

• Replay is byte-identical/deterministic (hooks use no clocks/RNG). Variability exists only at re-record time, which is why model-shaped values use loose assertions — a re-recording with a different call count/ordering still passes.
• Deferred as hard-to-force-live or already covered: invalid-tool-call recovery (Retry/Repair/Skip/Fail via unregistered/disallowed calls — Gemini won't emit those on purpose; covered by agent_run_recovery.rs + rig-core agent::runner unit tests) and structured-output Tool-mode finalization (structured_output.rs + rig-core unit tests for ModelTurnFinished content and the output-tool stream item). Per the guidance to prefer a few rich workflows over many flaky tiny live tests.