Add OpenRouter agent tool cassette regressions - #2006
Merged
Conversation
Merged
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
agent_tool_sessions, with 7 live-recorded regression scenarios for serious multi-turn/tool use.tool_choicerequired/none behavior, usage/finish reason/model preservation, routed provider preferences, and nested JSON-schema structured output.Models/providers used
openai/gpt-4ovia OpenRouter/OpenAI for stable tool calling, multi-tool turns, streaming tool-call deltas, and long chat-history replay.google/gemini-2.5-flashvia OpenRouter/Google AI Studio preference for inexpensive nested JSON-schema structured-output coverage withprovider.require_parameters.Inspiration references studied
tests/README.md, existingtests/providers/openrouter/cassette/*,tests/cassettes/openrouter/*,crates/rig-core/src/providers/openrouter/*, and OpenAI-compatible code undercrates/rig-core/src/providers/openai/*.tests/providers/openai/cassette/responses_sessions.rsfor long-session/multi-tool regression shape./Users/kisaczka/Desktop/code/many_rigs/inspirations/vercel-ai-sdk/packages/openai-compatible/src/chat/convert-to-openai-compatible-chat-messages.test.tsandpackages/openai/src/chat/openai-chat-language-model.tsfor OpenAI-compatible tool message/result, streamed tool delta, usage, and finish-reason handling./Users/kisaczka/Desktop/code/many_rigs/inspirations/vercel-ai-sdk/packages/openai/src/responses/openai-responses-language-model.tsfor Responses event/usage/tool continuation patterns./Users/kisaczka/Desktop/code/many_rigs/inspirations/pydantic-ai/tests/models/test_openrouter.pyandtests/providers/test_openrouter.pyfor OpenRouter provider options, reasoning/tool preservation, nested schemas, usage/provider details, and media/document edge cases./Users/kisaczka/Desktop/code/many_rigs/inspirations/langchain/libs/partners/openrouter/tests/integration_tests/test_chat_models.pyandtest_standard.pyfor OpenRouter tool calling, structured output, multimodal support, usage, and multi-turn reasoning preservation./Users/kisaczka/Desktop/code/many_rigs/inspirations/semantic-kernel/dotnet/src/IntegrationTests/Data/BaseTextSearchTests.csfor required function/tool-choice behavior in agent loops.Bugs found/fixed
Recording/replay commands run
OPENROUTER_API_KEYonly throughRIG_PROVIDER_TEST_MODE=record, including:RIG_PROVIDER_TEST_MODE=record cargo test -p rig --all-features --test openrouter agent_tool_sessions -- --nocapture --test-threads=1env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter agent_tool_sessions -- --nocapture --test-threads=1env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter openrouter::cassette -- --nocapture --test-threads=1env -u OPENROUTER_API_KEY cargo test -p rig --all-features --test openrouter cassette_safety -- --nocapture --test-threads=1cargo fmt --checkcargo clippy --all-targets --all-featurescargo test -p rig-core providers::openrouter -- --nocaptureManual cassette inspection notes
tests/cassettes/openrouter/agent_tool_sessions/for expected/chat/completionsrequests,parallel_tool_calls: false,tool_choice: required/none, streamed tool-call deltas, responseusage,finish_reason/native_finish_reason, routed provider/model metadata, and nestedresponse_formatschema payloads.Remaining known gaps / non-goals
cargo testwithout feature flags fails on an existing unrelated Gemini image-generation feature-gating compile issue (rig::image_generationandGEMINI_2_5_FLASH_IMAGEare gated behind the image feature); targeted OpenRouter and all-features clippy validations passed.