Vision Providers

Vision Providers

Screenwright's vision layer is fully optional and provider-swappable — describe() returns the same ScreenshotMetadata shape regardless of provider, so downstream output never changes.

Each provider's SDK is a separate pip extra, not a hard dependency of the base package — see Getting Started.

Claude Haiku (default — cloud)

pip install "screenwright[anthropic]"
export ANTHROPIC_API_KEY=your-api-key

[vision]
provider = "anthropic"
model    = "claude-haiku-4-5"
structured_metadata = true

Cost: ~$0.25/M input tokens — a fraction of a cent per screenshot.

OpenAI (cloud)

pip install "screenwright[openai]"
export OPENAI_API_KEY=your-api-key

[vision]
provider = "openai"
model    = "gpt-4o-mini"
structured_metadata = true

Useful when the rest of your workflow — e.g. driving Screenwright from Codex CLI — is already on OpenAI.

Moondream2 (local — no API key)

pip install "screenwright[ollama]"
# Install Ollama: https://ollama.com
ollama pull moondream

[vision]
provider = "ollama"
model    = "moondream"
structured_metadata = true

Recommended default for air-gapped environments or UIs with sensitive data that shouldn't leave your network. Other Ollama vision models (llava, qwen2-vl) also work — set model accordingly.

Note: Moondream2 is small (~1.6B) and can return an empty response when asked to follow the structured JSON prompt. Screenwright falls back gracefully to an empty description rather than failing the run — but for reliable structured metadata with Moondream, either set structured_metadata = false or use a larger local model such as llava or qwen2-vl.

Custom describe prompt

Every provider sends a built-in generic prompt ("Describe this UI screenshot for documentation purposes.") unless you override it — useful for steering descriptions toward accessibility review, a non-English language, or a specific documentation style:

[vision]
provider = "anthropic"
prompt   = "Focus on accessibility issues: missing labels, low contrast, unclear focus states."

Works with any provider, and with describe_screenshot's own prompt argument when calling it directly via MCP instead of through a flow's [vision] config. When structured_metadata = true (the default), the JSON-structure instruction is still appended after your prompt — you're overriding what the model is asked to look for, not the requirement that it respond as JSON.

Disable vision entirely

[screenwright]
vision_describe = false

Pure capture mode — no API calls, no metadata JSON, just PNGs (and video, if record = true).