Skip to content

fix(llm): gate OpenAI/Compatible supports_vision on model capability#6423

Merged
bug-ops merged 1 commit into
mainfrom
fix/6411-openai-supports-vision
Jul 17, 2026
Merged

fix(llm): gate OpenAI/Compatible supports_vision on model capability#6423
bug-ops merged 1 commit into
mainfrom
fix/6411-openai-supports-vision

Conversation

@bug-ops

@bug-ops bug-ops commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Summary

  • OpenAiProvider::supports_vision() unconditionally returned true regardless of the configured model, so a text-only model (e.g. gpt-3.5-turbo, or an arbitrary text-only type = "compatible" endpoint) would have images attached and sent, typically failing with a 400. CompatibleProvider inherited the same bug via delegation.
  • supports_vision() now resolves via an explicit vision override (new OpenAiConfig/CompatibleConfig field, with_vision builders, and a new vision field on [[llm.providers]] entries) falling back to a built-in model-name prefix table for well-known OpenAI vision families (gpt-4o, gpt-4-turbo, gpt-4-vision, gpt-4.1, gpt-5, vision-capable o-series), explicitly excluding text-only reasoning variants (o1-mini, o1-preview, o3-mini). Unrecognised model names fail safe to false — this is also the default for arbitrary compatible endpoints unless vision = true is set explicitly.
  • Same defect class as zeph-llm: Ollama provider's supports_vision() hardcoded true regardless of configured model #6377, already fixed for Ollama in fix(llm): gate Ollama supports_vision on actual model capability #6410.

Closes #6411

Test plan

  • Full local check suite: cargo +nightly fmt --check, cargo clippy --profile ci --workspace --all-targets --features "desktop,ide,server,chat,pdf,scheduler,testing" -- -D warnings, cargo nextest run --config-file .github/nextest.toml --workspace --features "desktop,ide,server,chat,pdf,scheduler" --lib --bins (14217 passed, 0 failed, 35 skipped; one unrelated concurrency test flake confirmed non-reproducing on re-run), rustdoc gate with RUSTFLAGS="-D warnings" RUSTDOCFLAGS="--deny rustdoc::broken_intra_doc_links" — all clean.
  • cargo test --doc --workspace — all 32 crates pass, including new with_vision doctests on OpenAiProvider/CompatibleProvider.
  • Per-crate rustdoc check for touched crates (zeph-llm, zeph-config) — clean.
  • New unit tests cover: known-vision-model auto-detect, gpt-3.5* false, unrecognised-model fail-safe false, explicit override in both directions, text-only o-series exclusion (o1-mini/o1-preview/o3-mini), CompatibleConfig.vision config-field forwarding (distinct from the .with_vision() builder path).
  • LLM serialization gate: live round-trip against a real gpt-4o-mini OpenAI provider confirmed the request reaches the server and is rejected only with 429 (rate-limited, a known pre-existing test-account constraint, unrelated to this change) — never a 400/422, confirming the request payload is well-formed. Verified in source that 400 responses are routed through a distinct error path (map_error_responseLlmError::ApiError) never conflated with the 429/503 retry loop that was actually hit.
  • Adversarial critique (impl-critic) and independent test-coverage validation (tester) both ran; one significant finding (o-series text-only false positive) was fixed and re-verified before code review approval.

OpenAiProvider::supports_vision() unconditionally returned true
regardless of the configured model, so a text-only model (e.g.
gpt-3.5-turbo, or an arbitrary text-only compatible endpoint) would
have images attached and sent, typically failing with a 400.
CompatibleProvider inherited the same bug via delegation.

supports_vision() now resolves via an explicit vision override
(OpenAiConfig::vision / CompatibleConfig::vision, or the with_vision
builder) when set, otherwise a built-in model-name prefix table for
known OpenAI vision-capable families (gpt-4o, gpt-4-turbo, gpt-4-vision,
gpt-4.1, gpt-5, and vision-capable o-series models, explicitly excluding
text-only o-series reasoning variants like o1-mini/o1-preview/o3-mini).
Unrecognised models fail safe to false, which is also the default for
arbitrary OpenAI-compatible endpoints unless vision is explicitly
configured. ProviderEntry.vision is wired through provider_factory.rs
so operators can override via config.toml.

Closes #6411
@github-actions github-actions Bot added documentation Improvements or additions to documentation llm zeph-llm crate (Ollama, Claude) skills zeph-skills crate rust Rust code changes core zeph-core crate config Configuration file changes bug Something isn't working size/XL Extra large PR (500+ lines) labels Jul 17, 2026
@bug-ops
bug-ops enabled auto-merge (squash) July 17, 2026 21:46
@bug-ops
bug-ops merged commit 4a1bdab into main Jul 17, 2026
47 checks passed
@bug-ops
bug-ops deleted the fix/6411-openai-supports-vision branch July 17, 2026 21:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working config Configuration file changes core zeph-core crate documentation Improvements or additions to documentation llm zeph-llm crate (Ollama, Claude) rust Rust code changes size/XL Extra large PR (500+ lines) skills zeph-skills crate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OpenAI/Compatible providers hardcode supports_vision() true regardless of model capability

1 participant