Skip to content

Family resolver misclassifies sam2_video as text-prompt; unknown model types fall through instead of refusing #456

Description

@JArmandoAnaya

The adapter that answers a connection is chosen from the model_type the model's own
config declares (visionset/inference/providers.py). Two defects there, one visible and
one latent.

1. sam2_video is classified as a text-prompt model

SEGMENTER_FAMILIES holds exactly {"sam2"}. The SAM 2 checkpoints published for image
and video work declare model_type: sam2_video — including
facebook/sam2-hiera-base-plus, which is the model id the product's own connection form
suggests. That name is not in the set, so resolution falls through to the detector
adapter, and a point suggestion is refused with a sentence that is confidently false:

$ family_of(connection) -> 'sam2_video'   in SEGMENTER_FAMILIES=False
UnsupportedPrompt: facebook/sam2-hiera-base-plus@main answers text prompts;
it was asked with 'points' points, which it has no way to interpret

Reproduced 2026-08-08 against a ready local connection whose weights are in the
workspace cache. The model is a point-prompt model; the message describes some other
model.

The checkpoint is servable as it stands — Sam2Model.from_pretrained loads a
sam2_video checkpoint and transformers names that case as expected:

You are using a model of type `sam2_video` to instantiate a model of type `sam2`.
This may be expected if you are loading a checkpoint that shares a subset of the
architecture (e.g., loading a `sam2_video` checkpoint into `Sam2Model`)

So this is a classification defect only. No adapter needs to change.

2. An unrecognised model_type falls through instead of refusing

family_of returns the declared type, or "" when the config cannot be read; anything
not in SEGMENTER_FAMILIES is handed to the detector adapter. So a model this build has
no adapter for — and a config nothing could parse — is served by whichever adapter
happens to be the fallback, and fails later with a message about that adapter's prompt
kind rather than about the model.

Defect 1 is one instance of this. A wrong classification produces error messages that lie
about the user's model, which is worse than an honest refusal.

What it should do

  • Every SAM 2 model_type the locked transformers can serve for image-level point
    prompting resolves to the point-prompt adapter.
  • An unrecognised model_type is refused with a message that says the model type is not
    supported and lists the ones that are — never resolved to a family by fallback.

Scope

The resolver and its error message. The suggested default model id is correct and does
not change; nothing about the suggest route or the editor changes.

cf. #424, #418

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendvisionset Python distribution — kernel, server, cli, mcp, formatsbugSomething isn't working

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions