Skip to content

Improve OpenAI agent routing and scenario-suite runs - #487

Open
ShuxinLin wants to merge 23 commits into
IBM:mainfrom
ShuxinLin:codex/openai-agents-responses-routing
Open

Improve OpenAI agent routing and scenario-suite runs#487
ShuxinLin wants to merge 23 commits into
IBM:mainfrom
ShuxinLin:codex/openai-agents-responses-routing

Conversation

@ShuxinLin

@ShuxinLin ShuxinLin commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • upgrade to openai-agents>=0.18.3 and make openai-agent router-only, preventing fallback to direct OpenAI credentials
  • apply verified TokenRouter API routing and configurable reasoning effort, defaulting to medium
  • persist API-provided reasoning summaries and reasoning-token counts for GPT-5 Responses runs without requesting raw chain-of-thought
  • reuse MCP subprocesses, expose all configured MCP tools by default, and support a repeatable fail-closed tool allowlist
  • add opt-in file, Bash, edit, and public-web tools with per-run workspaces
  • extend the scenario-suite runner with openai_agent, named YAML profiles, agent/model output nesting, per-run workspaces, recursive evaluation aggregation, and --skip-existing
  • update the Lite profile to 10 CAR, FCC, FMSR, Health, and WOSR scenarios, with TSFM limited to 1001–1005

TokenRouter routing

Model API route Reasoning effort
tokenrouter/openai/gpt-5.* Responses supported
tokenrouter/MiniMax-M3 Responses supported
tokenrouter/google/gemini-3.6-flash Responses supported
tokenrouter/anthropic/claude-opus-4.8 Chat Completions ignored
tokenrouter/z-ai/glm-5.2 Chat Completions supported

Other router-backed models use Chat Completions. Reasoning effort defaults to medium; GPT-5 reasoning summaries default to auto and can be disabled with none.

Safety

  • raw chain-of-thought is neither requested nor stored
  • local file, Bash, edit, and web tools are disabled by default
  • file and edit tools enforce workspace containment
  • Bash uses the workspace and a credential-scrubbed environment, but is not an OS sandbox
  • web fetch permits public destinations only and revalidates redirects
  • an MCP allowlist is fail-closed whenever one or more --allow-mcp-tool flags are supplied

Validation

  • uv run pytest src/agent/openai_agent/tests src/benchmark/tests/test_scenario_suite_runner.py -q: 112 passed
  • live TokenRouter smoke checks passed for GPT-5.6 Sol, Claude Opus 4.8, MiniMax M3, Gemini 3.6 Flash, and GLM 5.2 using the routes above
  • git diff --check

Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin ShuxinLin changed the title Update OpenAI Agents SDK routing Update OpenAI agent routing and permissions Jul 28, 2026
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin ShuxinLin changed the title Update OpenAI agent routing and permissions Update OpenAI agent routing, permissions, and reasoning summaries Jul 28, 2026
ShuxinLin added 13 commits July 28, 2026 14:03
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
@ShuxinLin ShuxinLin changed the title Update OpenAI agent routing, permissions, and reasoning summaries Improve OpenAI agent routing and scenario-suite runs Jul 28, 2026
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Signed-off-by: Shuxin Lin <linshuhsin@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant