fix: route image_qa and self_reflection through the configured model (closes #3)#5
Merged
adamlu123 merged 1 commit intoMay 27, 2026
Conversation
…loses microsoft#3) Issue microsoft#3 reported that 'Fails to run with only ANTHROPIC_API_KEY'. The image_qa and self_reflection tools hardcoded the OpenAI Responses API through an _openai_config helper that read OPENAI_API_KEY even when the agent loop ran on Claude. This change adds a model_config flag to both tools, threads the resolved model: block from base.yaml (and model_claude.yaml's anchored tools.*.model blocks) into the subprocess invocation, and routes through the same models registry the outer loop uses. The existing OpenAI fallback path is preserved for direct CLI use. Closes microsoft#3.
adamlu123
added a commit
that referenced
this pull request
May 27, 2026
Follow-up to #5. Consolidates duplicated model-config helpers, drops legacy OpenAI HTTP code paths, and simplifies model_claude.yaml. Net -477 lines.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
webwright -c base.yaml -c model_claude.yamlnow runs end-to-end with onlyANTHROPIC_API_KEYset. Theimage_qaandself_reflectioninner tools route through the configured model registry instead of hardcoding the OpenAI Responses API.Why this matters
Issue #3 (reported 2026-05-26 by @leachuk) shows the symptom: an Anthropic-only run fails at step 8 with
RuntimeError: Missing OPENAI_API_KEYraised fromsrc/webwright/tools/image_qa.py:66. The README andbase.yaml:25say "Export credentials for the chosen backend (e.g. OPENAI_API_KEY or ANTHROPIC_API_KEY)" but the inner tools never honored that contract — the comment atbase.yaml:18explicitly admittedOPENAI_API_KEY (always — used by self_reflection and image_qa tools).The root cause was two
_openai_config(args)helpers intools/image_qa.pyandtools/self_reflection.pyreadingos.environ.get("OPENAI_API_KEY")and posting directly to the OpenAI Responses API. Browser-use and Stagehand both let every layer of the agent pick any supported model; this change brings webwright to parity.Changes
tools/image_qa.pyandtools/self_reflection.pyaccept--model-config <path>and route the tool's vision call throughwebwright.models.get_model(...). The existing--api-key/OPENAI_API_KEYpath is preserved so direct CLI use of these tools (without the agent loop) still works.agents/default.pywrites the resolvedmodel:(ortools.<name>.model:) block to a per-workspace JSON file and passes--model-config <path>to the subprocess invocations.models/base.pyadds_complete_text_asyncand__call__, so model wrappers can be reused for inner-tool calls without going through the agent's full query pipeline.OpenAIModelandOpenRouterModelget a matching_build_text_payload.AnthropicModelalready produces text via the same primitives the agent uses.model_claude.yamldeclarestools.image_qa.modelandtools.self_reflection.modelvia a YAML anchor, so an Anthropic run inherits Anthropic for inner tools automatically.base.yamland the README credentials section are updated to reflect the new reality.Testing
tests/unit/test_tool_model_routing.py(2 tests, both passing) verifies thatbase.yaml + model_claude.yamlresolves the inner-tool model class toanthropic, and that_extract_model_configfalls back to the top-levelmodel:block when no per-tool override is present.Fixes #3.