Skip to content

Suggest tool picks any ready connection blindly; connections don't declare capability #480

Description

@JArmandoAnaya

Two defects, one missing fact. A connection row says where a model runs and
whether its weights are here, and nothing anywhere says what it can be asked
to do
. The editor therefore picks a connection by the only property it can see.

cf. #418, #424, #425, #469.

Observed

With grounding-dino-tiny as the only ready connection, every S click in the
editor round-trips to the server and comes back with the truthful-but-useless
UnsupportedPrompt refusal. The tool was offered where it could never work, and
the server said so one click at a time.

  1. The editor picks blindly. usableConnection is
    connections.find((row) => row.setup_state === "ready") — the first ready
    row, whatever model it names. A workspace whose ready connection answers text
    prompts sends every point-prompt click to it.
  2. Nothing declares capability. The family a model belongs to is resolved
    from its downloaded config on every provider build (family_of), used to pick
    an adapter, and then thrown away. No surface can read it, so no client can
    filter on it — and there is no state a client could filter on instead, because
    "which prompts does this model take" is a property of the weights rather than
    of the row.

The server refusal is correct and stays. What is missing is the client's ability
to not ask.

Design (directed by Armando, 2026-08-09)

A. Connections declare capability

  • Persist the resolved family when it becomes knowable — at download
    completion for a local connection, where the config is in the cache and the
    resolver the providers already use can read it. One resolver, no second
    spelling, and nothing is fetched at connection creation: the form's promise
    is that nothing is fetched until asked.
  • ConnectionOut gains a declared capability derived from that family, in
    the wire's existing grammar. Vocabulary: point_suggest (SAM 2 families),
    text_detect (grounding-dino family); unknown or not yet resolved is empty.
    One home for the family-to-capability mapping, beside the family sets it
    derives from.
  • HTTP connections declare nothing until the HTTP contract slice defines how
    a remote endpoint states what it can do. That is the slice's inherited
    question, recorded here rather than answered by default.
  • Backfill. A row that is already ready carries no stored family. Resolve
    from the cached config where it is present. A ready row whose config is not
    in the cache resolves to empty, is not usable for the suggest tool, and has
    the download and integrity actions as its visible remedy.

B. The editor filters, then lets the person choose

  • usableConnection becomes capability-aware: the suggest tool requires
    point_suggest, and only ready connections declaring it are candidates.
  • A new blocker state — connections exist, some even ready, and none carries
    the capability this tool needs. The panel says what happened ("your connection
    answers text prompts; this tool needs a point-prompt model — the SAM 2
    family") and what to do, through the existing configure callback. It is
    distinguishable from no-connections and not-ready, and the parked-class
    precedence settled in fix(annotator): the suggest tool survives a class switch; a select option is two lines #473 keeps its rank.
  • A selector when there is a choice. Two or more ready capable connections
    put a compact select in the suggest panel, remembered per project. Exactly one
    candidate renders no selector at all — zero friction for the common case.
  • The request still carries an explicit connection_id, and the server
    still refuses a wrong-family one. The UI filter is convenience; the kernel
    check is the law.

Out of scope

  • No batch-mode surface. text_detect gains a declared capability and no tool —
    that is Batch pre-labeling — unattended model annotation as a background job #425, and this slice makes the wiring trivial when it arrives.
  • No HTTP adapter work beyond the empty-capability statement above.
  • No change to providers, masks, or the suggest route's contract other than the
    new ConnectionOut field.

Not by model-id string matching

The family comes from the model's own downloaded config, never from matching on
the model id. Guessing by name is what #458 removed, and a resolver that guesses
is wrong invisibly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    backendvisionset Python distribution — kernel, server, cli, mcp, formatsenhancementNew feature or requestfrontendannotator / ui-core / app packageskernelvisionset.kernel — domain, ports, adapters, services

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions