Skip to content

Add OpenRouter agent tool cassette regressions - #2006

Merged
gold-silver-copper merged 1 commit into
mainfrom
test/openrouter-cassette-regression
Jul 4, 2026
Merged

Add OpenRouter agent tool cassette regressions#2006
gold-silver-copper merged 1 commit into
mainfrom
test/openrouter-cassette-regression

Conversation

@gold-silver-copper

Copy link
Copy Markdown
Contributor

Summary

  • Adds a new OpenRouter cassette module, agent_tool_sessions, with 7 live-recorded regression scenarios for serious multi-turn/tool use.
  • Covers sequential non-streaming and streaming agent loops with four ordered tools and complex arguments: empty objects, nested JSON, arrays, escaped strings, backslashes, newlines, and unicode.
  • Covers multiple tool calls in one model turn for non-streaming and streaming, raw streamed tool-call deltas, long caller-owned chat-history replay with assistant tool calls/tool results, tool_choice required/none behavior, usage/finish reason/model preservation, routed provider preferences, and nested JSON-schema structured output.

Models/providers used

  • openai/gpt-4o via OpenRouter/OpenAI for stable tool calling, multi-tool turns, streaming tool-call deltas, and long chat-history replay.
  • google/gemini-2.5-flash via OpenRouter/Google AI Studio preference for inexpensive nested JSON-schema structured-output coverage with provider.require_parameters.

Inspiration references studied

  • tests/README.md, existing tests/providers/openrouter/cassette/*, tests/cassettes/openrouter/*, crates/rig-core/src/providers/openrouter/*, and OpenAI-compatible code under crates/rig-core/src/providers/openai/*.
  • tests/providers/openai/cassette/responses_sessions.rs for long-session/multi-tool regression shape.
  • /Users/kisaczka/Desktop/code/many_rigs/inspirations/vercel-ai-sdk/packages/openai-compatible/src/chat/convert-to-openai-compatible-chat-messages.test.ts and packages/openai/src/chat/openai-chat-language-model.ts for OpenAI-compatible tool message/result, streamed tool delta, usage, and finish-reason handling.
  • /Users/kisaczka/Desktop/code/many_rigs/inspirations/vercel-ai-sdk/packages/openai/src/responses/openai-responses-language-model.ts for Responses event/usage/tool continuation patterns.
  • /Users/kisaczka/Desktop/code/many_rigs/inspirations/pydantic-ai/tests/models/test_openrouter.py and tests/providers/test_openrouter.py for OpenRouter provider options, reasoning/tool preservation, nested schemas, usage/provider details, and media/document edge cases.
  • /Users/kisaczka/Desktop/code/many_rigs/inspirations/langchain/libs/partners/openrouter/tests/integration_tests/test_chat_models.py and test_standard.py for OpenRouter tool calling, structured output, multimodal support, usage, and multi-turn reasoning preservation.
  • /Users/kisaczka/Desktop/code/many_rigs/inspirations/semantic-kernel/dotnet/src/IntegrationTests/Data/BaseTextSearchTests.cs for required function/tool-choice behavior in agent loops.

Bugs found/fixed

  • No provider implementation bug was found while recording these cassettes.
  • Test-side replay note: caller-owned historical tool-call IDs use the cassette redacted placeholder form so the existing cassette scrubber can replay generated-looking OpenAI tool-call IDs deterministically.

Recording/replay commands run

  • Record mode, targeted new suite/tests using exported OPENROUTER_API_KEY only through RIG_PROVIDER_TEST_MODE=record, including:
    • RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test openrouter agent_tool_sessions -- --nocapture --test-threads=1
  • Replay/no-credential validation:
    • env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter agent_tool_sessions -- --nocapture --test-threads=1
    • env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter openrouter::cassette -- --nocapture --test-threads=1
    • env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter cassette_safety -- --nocapture --test-threads=1
  • Other validation:
    • cargo fmt --check
    • cargo clippy --all-targets --all-features
    • cargo test -p rig-core providers::openrouter -- --nocapture

Manual cassette inspection notes

  • Inspected new YAMLs under tests/cassettes/openrouter/agent_tool_sessions/ for expected /chat/completions requests, parallel_tool_calls: false, tool_choice: required/none, streamed tool-call deltas, response usage, finish_reason/native_finish_reason, routed provider/model metadata, and nested response_format schema payloads.
  • Ran cassette safety and an additional grep for API keys, bearer tokens, auth/cookie headers, and account/project-id shaped fields; no secrets or unrelated churn found.

Remaining known gaps / non-goals

  • No new OpenRouter Responses API cassette beyond existing OpenAI Responses compatibility coverage; this PR focuses on Rig's OpenRouter chat-completions path.
  • No new multimodal cassette in this module because existing OpenRouter cassette coverage already exercises image/audio/video/PDF paths.
  • Broad cargo test without feature flags fails on an existing unrelated Gemini image-generation feature-gating compile issue (rig::image_generation and GEMINI_2_5_FLASH_IMAGE are gated behind the image feature); targeted OpenRouter and all-features clippy validations passed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant