Skip to content

Only allow generative (text/vision) models in the chat model picker - #102

Merged
Lazarus-931 merged 3 commits into
mainfrom
fix/chat-model-picker-generative-only
Jul 27, 2026
Merged

Only allow generative (text/vision) models in the chat model picker#102
Lazarus-931 merged 3 commits into
mainfrom
fix/chat-model-picker-generative-only

Conversation

@Lazarus-931

@Lazarus-931 Lazarus-931 commented Jul 25, 2026

Copy link
Copy Markdown
Collaborator

Problem

The chat model picker is a denylist (LocalModel.isEligibleForLanguageModelPicker = !speechToText && !textToSpeech), so non-generative models a user downloads — encoders like google-bert/bert-base-uncased / FacebookAI/xlm-roberta-base, embeddings, ASR, time-series — still appear and can be selected as the chat model. Selecting one and starting the server pre-loads it as text_generation; mlx-vlm/mlx-lm has no matching architecture, so startup aborts:

ERROR - Error loading model google-bert/bert-base-uncased: Model type bert not supported
ModuleNotFoundError: No module named 'mlx_lm.models.bert'
Application startup failed. Exiting.  (exit status 3)

The app is then effectively bricked — the server won't come up and the menu bar is unresponsive, with no in-UI way to recover.

Fix

  • Picker becomes an allowlist of generative models: text and multimodal vision-language models both qualify (.text or .vision); only models with no generative capability — encoders, embeddings, ASR, forecasting, pure image-gen — drop out. Multimodal/VLM models are not excluded.
  • More robust generative detection: infer .text from causal/generation architectures (*ForCausalLM / *ForConditionalGeneration / *LMHeadModel) in addition to the existing name heuristics, so legitimate chat models (Phi, SmolLM, …) aren't accidentally filtered out.
  • A stale non-generative selection no longer bricks startup: when the configured chat model's local config.json shows only non-generative architectures, its --model pre-load flag is dropped (load-on-demand) so the server still starts, with a log note. The check is conservative — any uncertainty leaves the model untouched.

Screenshots to follow.

The chat model picker was a denylist (not speech), so encoders like
google-bert/bert-base-uncased, embeddings, ASR and time-series models stayed
selectable. Selecting one pre-loads it as text_generation and aborts server
startup (Model type bert not supported), bricking the app.

- Detect generative models by causal/seq2seq architecture in addition to the
  name heuristics, so real chat models (Phi, SmolLM, …) are recognized and
  encoders are not.
- Make the picker an allowlist: only text-generation-capable (.text/.vision)
  models are selectable.
- Skip pre-loading a stale non-generative selection so the server still starts.
@Lazarus-931 Lazarus-931 changed the title Only allow text-generation models to be picked as the chat model Only allow generative (text/vision) models in the chat model picker Jul 25, 2026
@Lazarus-931
Lazarus-931 requested a review from Blaizzy July 25, 2026 15:34
@Blaizzy

Blaizzy commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Please double check this doesn't affect images which are supported in the chat version in images tab

@Blaizzy

Blaizzy commented Jul 25, 2026

Copy link
Copy Markdown
Owner

Also, it doesn't affect omni models like Gemma 4 which might use that tag

@Lazarus-931

Copy link
Copy Markdown
Collaborator Author

yeah @Blaizzy, great point about omni models, added a small patch for that, for the img page, i think we can wait to merge this until a new pr for folding the img feature into the chat page. i think @lucasnewman said it'll be a follow up work

@lucasnewman

Copy link
Copy Markdown
Collaborator

yeah @Blaizzy, great point about omni models, added a small patch for that, for the img page, i think we can wait to merge this until a new pr for folding the img feature into the chat page. i think @lucasnewman said it'll be a follow up work

I added that in #93 so you should be good -- model availability is type-aware now for non-language models (image, stt, tts).

Comment thread Sources/Nativ/NativModel.swift Outdated

@lucasnewman lucasnewman left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM but see comment about making sure to pick up all possible model roots.

…olders

Add a shared NativSettings.modelSearchRoots (primary folder + user-added
additionalModelSearchPaths + HF hub cache) and use it in the chat-picker's
non-generative check instead of re-deriving the roots there, which had omitted
the user's custom model folders.
@Lazarus-931
Lazarus-931 merged commit ad108af into main Jul 27, 2026
2 checks passed
@Lazarus-931

Copy link
Copy Markdown
Collaborator Author

yeah @Blaizzy, great point about omni models, added a small patch for that, for the img page, i think we can wait to merge this until a new pr for folding the img feature into the chat page. i think @lucasnewman said it'll be a follow up work

I added that in #93 so you should be good -- model availability is type-aware now for non-language models (image, stt, tts).

wait, confused, the release shows sep img and chat page

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants