Image auto-routing cannot use native vision when auxiliary.vision is configured as fallback

[Vodkakakaka]

Bug Description

When agent.image_input_mode is left at the default auto, an explicit auxiliary.vision override forces all user-attached images through the text-description pipeline, even when the active main model supports native vision.

This makes it impossible to keep a dedicated auxiliary vision backend for text-only main models (for example DeepSeek) while still allowing a vision-capable main model (for example GPT-5.5 / OpenAI-family multimodal models) to inspect the original image pixels natively in the same profile.

In practice, the current config model creates a conflict:

• DeepSeek main model needs auxiliary.vision explicitly set to a vision-capable backend (Gemini flash), because DeepSeek itself does not support image input.
• GPT-5.5-style main models should receive attached images natively, because they support vision and should not be limited to a lossy auxiliary text summary.
• With the current routing logic, the explicit auxiliary.vision setting wins in auto mode and prevents the native path from being used for vision-capable main models.

Current Config Example

model:
  provider: deepseek
  default: deepseek-v4-pro

agent:
  image_input_mode: auto

auxiliary:
  vision:
    provider: gemini
    model: gemini-2.5-flash

This is a useful config for DeepSeek: uploaded images are analyzed by Gemini and converted to text for the text-only main model.

However, when switching the main model/session to a vision-capable GPT-5.5-style model, the same profile still routes uploaded images through vision_analyze instead of attaching the image natively to the main model.

Relevant Code Path

agent/image_routing.py documents and implements the decision:

# In auto mode:
#   - If the user has explicitly configured auxiliary.vision.provider
#     (i.e. not auto and not empty), we assume they want the text pipeline
#     regardless of the main model

And:

if _explicit_aux_vision_override(cfg):
    return "text"

supports = _lookup_supports_vision(provider, model)
if supports is True:
    return "native"
return "text"

So an explicit auxiliary vision backend has higher priority than the active model's supports_vision=True capability.

Expected Behavior

There should be a config-only way to express this policy:

Use native image input whenever the active main model supports vision; otherwise fall back to the configured auxiliary vision backend.

That would allow one profile to work correctly for both:

• text-only main models such as DeepSeek, using auxiliary.vision as fallback/text-description backend;
• vision-capable main models such as GPT-5.5, using native multimodal input.

Actual Behavior

With agent.image_input_mode: auto and any explicit auxiliary.vision provider/model, Hermes always chooses the text-description path for user-attached images, even if the active main model supports native vision.

The only current config workarounds are incomplete:

1. Set agent.image_input_mode: native

   • fixes vision-capable main models;
   • but breaks or risks API errors for text-only models such as DeepSeek.

2. Clear auxiliary.vision.provider/model

   • allows auto to choose native for vision-capable main models;
   • but removes the explicit fallback needed for text-only main models.

3. Use separate Hermes profiles

   • works operationally;
   • but splits sessions/config/state and is not a real fix for a routing-policy limitation.

Proposed Solution

Add a routing policy that gives native-capable main models priority while keeping auxiliary vision as fallback for text-only models.

Possible config shapes:

agent:
  image_input_mode: auto_native_first

or:

agent:
  image_input_mode: auto
  image_native_priority: true

or under auxiliary vision:

auxiliary:
  vision:
    provider: gemini
    model: gemini-2.5-flash
    use_as_fallback_only: true

Behavior:

1. If agent.image_input_mode: native, force native as today.
2. If agent.image_input_mode: text, force auxiliary/text as today.
3. If auto-native-first / fallback-only policy is enabled:
   • check active main model capabilities first;
   • if supports_vision=True, return native;
   • otherwise use the configured auxiliary.vision text pipeline.
4. Keep the current behavior as default if backward compatibility is preferred.

Why This Needs Source Changes

The conflict cannot be solved cleanly with current config fields because the only available modes are:

• auto: explicit auxiliary.vision forces text pipeline before checking main model vision capability;
• native: unsafe for text-only providers;
• text: disables native vision entirely.

There is no existing config value for “native if supported, otherwise use configured auxiliary vision fallback”.

Environment

• Hermes Agent source checkout: NousResearch/hermes-agent
• OS: WSL / Linux
• Main text model example: deepseek + deepseek-v4-pro
• Auxiliary vision example: gemini + gemini-2.5-flash
• Desired vision-capable main model behavior: GPT-5.5-style native multimodal input

[Vodkakakaka]

Additional clarification: clearing auxiliary.vision and relying on auto is not a reliable workaround in this setup.

For a DeepSeek main model, the expected behavior would be:

DeepSeek does not support image input, so image analysis should skip DeepSeek and fall through to a configured/available vision backend.

But in practice, when auxiliary.vision.provider/model is reset to auto, the DeepSeek path can get stuck/fail in the vision auto chain instead of cleanly falling back. The failure mode we observed/avoided is that the system attempts to route image content through DeepSeek (which does not support image_url / multimodal input), resulting in errors such as unsupported image content / unknown variant 'image_url'-style failures.

That is why the explicit Gemini auxiliary vision config exists in the first place:

auxiliary:
  vision:
    provider: gemini
    model: gemini-2.5-flash

So the current choices are all flawed:

1. Keep explicit Gemini auxiliary vision:

   • DeepSeek works;
   • vision-capable main models (GPT-5.5-style) lose native image input because auto chooses the text pipeline first.

2. Clear auxiliary vision and rely on auto:

   • may allow native image input for vision-capable main models;
   • but DeepSeek image handling can get stuck/fail instead of falling back as expected.

3. Force agent.image_input_mode: native:

   • works for vision-capable main models;
   • unsafe for DeepSeek/text-only providers.

So the requested source-level fix is specifically not just “let users clear the auxiliary override”. We need a routing mode/policy that supports:

If the active main model supports native vision, use native input. If it does not, use the configured auxiliary vision backend as fallback, without attempting to send image content through text-only providers like DeepSeek.

[Vodkakakaka]

More precise root-cause detail for why “just set auxiliary.vision back to auto” is not a valid workaround for DeepSeek users:

We previously traced the DeepSeek failure to a split between two different image-routing paths.

1. User-attached image routing uses model capability metadata

agent/image_routing.py::decide_image_input_mode() checks the active model capabilities through models.dev:

supports = _lookup_supports_vision(provider, model)
if supports is True:
    return "native"
return "text"

For deepseek-v4-pro, the model metadata is text-only (attachment: false, input modalities only include text), so this part correctly chooses the text pipeline.

2. But vision_analyze provider auto-routing uses a separate hardcoded skip list

After the text pipeline is selected, the image is passed to vision_analyze. Its auto provider resolution goes through agent/auxiliary_client.py::resolve_vision_provider_client().

That path does not use the same supports_vision metadata. Instead, it checks _PROVIDERS_WITHOUT_VISION:

_PROVIDERS_WITHOUT_VISION: frozenset = frozenset({
    "kimi-coding",
    "kimi-coding-cn",
})

deepseek is missing from that set.

So with:

model:
  provider: deepseek
  default: deepseek-v4-pro

agent:
  image_input_mode: auto

auxiliary:
  vision:
    provider: auto
    model: ''

the actual chain is:

1. decide_image_input_mode() sees DeepSeek is text-only and returns text.
2. The TUI/gateway calls vision_analyze to create a text description.
3. resolve_vision_provider_client(auto) reads the main provider as deepseek.
4. It looks for a provider-specific vision model in _PROVIDER_VISION_MODELS; none exists for DeepSeek, so it keeps deepseek-v4-pro.
5. It checks _PROVIDERS_WITHOUT_VISION; DeepSeek is not listed, so it does not skip the main provider.
6. It creates a DeepSeek client successfully because the API key/base URL are valid.
7. The image content is sent to the DeepSeek chat endpoint.
8. DeepSeek rejects the image content (image_url / multimodal input unsupported), producing errors such as unknown variant 'image_url' / unsupported image content.

That is why we explicitly configured Gemini for auxiliary vision locally:

auxiliary:
  vision:
    provider: gemini
    model: gemini-2.5-flash

This avoids the broken DeepSeek vision-auto chain, but it introduces the separate issue reported here: when switching to a vision-capable main model, the explicit Gemini override makes image_input_mode: auto choose the text-description path first and prevents native vision.

What this issue is asking for

The desired source-level behavior is not simply “clear auxiliary.vision and rely on auto”. That fails for DeepSeek today because DeepSeek is misclassified/not skipped in the vision auto chain.

The desired behavior is:

1. If the active main model supports native vision, attach images natively to the main model.
2. If the active main model is text-only, skip providers/endpoints known to be text-only (including DeepSeek’s api.deepseek.com chat endpoint) and use the configured auxiliary vision fallback.

This likely needs both:

• a routing policy for “native if main model supports vision, otherwise auxiliary fallback”; and/or
• making resolve_vision_provider_client(auto) consult the same model capability metadata, or at least include deepseek in _PROVIDERS_WITHOUT_VISION.

[Vodkakakaka]

One more summary to make the configuration dead-end explicit:

We have tried the available configuration-level approaches, and none can express the required behavior. This appears to require a source-level change rather than user configuration.

Config paths tested / considered

1. Keep explicit auxiliary vision backend

agent:
  image_input_mode: auto

auxiliary:
  vision:
    provider: gemini
    model: gemini-2.5-flash

Result:

• Works for DeepSeek/text-only main models.
• But prevents native image input for GPT-5.5-style vision-capable main models, because auto treats explicit auxiliary.vision as an instruction to always use the text-description pipeline.

Conclusion: not acceptable when switching to a native-vision main model.

2. Clear auxiliary vision and rely on auto

agent:
  image_input_mode: auto

auxiliary:
  vision:
    provider: auto
    model: ''

Result:

• Lets decide_image_input_mode() choose native for models whose metadata says supports_vision=True.
• But for DeepSeek/text-only main models, the later vision_analyze auto provider chain can still choose the DeepSeek client because deepseek is missing from _PROVIDERS_WITHOUT_VISION and that path does not reuse the same model capability check.
• The image is then sent to the DeepSeek chat endpoint and fails on unsupported image_url / multimodal input.

Conclusion: not reliable for DeepSeek. This is why the explicit Gemini fallback was needed in the first place.

3. Force native mode

agent:
  image_input_mode: native

Result:

• Works for GPT-5.5-style vision-capable main models.
• Unsafe for DeepSeek/text-only providers, because images can be sent directly to a provider that cannot accept them.

Conclusion: not usable for a profile that may run text-only providers.

4. Force text mode

agent:
  image_input_mode: text

Result:

• Works for text-only providers with a working auxiliary vision backend.
• Completely disables native vision for GPT-5.5-style main models.

Conclusion: not acceptable for native multimodal models.

5. Separate Hermes profiles

Result:

• Operational workaround: one profile for DeepSeek + Gemini auxiliary vision, another profile for GPT-5.5 + native image input.
• But this splits config/session/state and does not solve the routing limitation inside a single profile/session.

Conclusion: only a workaround, not a real fix.

Final conclusion

The current config surface (auto / native / text plus auxiliary.vision.provider/model) cannot express the desired policy:

Use native image input when the active main model supports vision; otherwise route to a configured auxiliary vision fallback, while never attempting to send image content through text-only providers such as DeepSeek.

This cannot be solved cleanly by changing config.yaml alone. The routing structure needs to change so that:

1. native-capable main models can take priority over auxiliary text-description routing;
2. text-only providers are skipped by the auxiliary vision auto chain using reliable capability metadata, not only a hardcoded _PROVIDERS_WITHOUT_VISION list;
3. explicit auxiliary.vision can be used as a fallback for text-only models without suppressing native vision for capable main models.