Model picker: show what a model is good/bad at — and whether it runs locally

[eman717]

The problem

The model picker lists every model by name and endpoint, which answers "what can I pick?" but not the two questions that actually decide the pick:

1. What is this model good and bad at, on my hardware?
2. Does it run locally, or do my prompts leave the machine?

The second matters more than it looks. Locally-served models, cloud-hosted models behind a :cloud-style tag, and OpenAI-compatible proxies all appear in the same flat list, styled identically. On my own setup a Grok model is reached through a loopback URL (127.0.0.1:8645) that forwards to a third party — so neither "does the name say cloud" nor "is the base URL local" is a reliable tell, and nothing at the point of selection distinguishes "this stays on my GPU" from "this is a third-party API."

For anyone self-hosting because of unreleased product work, client data, or regulated content, that distinction is the entire reason they self-host — and it's currently invisible at exactly the moment it's being decided. Users can memorise which names are which; that's a burden the interface could carry instead, and memory fails quietly.

Suggested shape

Two surfaces sharing one content source:

• A docked context panel beside the chat column for the selected model — good at / watch out for / measured throughput / location.
• Per-row hover tips over the open picker, so a model can be judged before selecting it.

The design point worth arguing: capability and location are independent axes

I built this locally and got it wrong the first time by making cloud a capability tier. That forces a false choice — a cloud model is frequently both the most capable option available and the one that must not receive confidential input. Collapsing them hides the actual tradeoff.

Two fields instead:

• Tier — default / capable / specialist / unbenchmarked
• Where — local / cloud

rendered as two badges. A model reading MORE CAPABLE + CLOUD simultaneously is the honest description.

Unknown location must fail loud. No recorded location should render an amber WHERE? ("treat as if it leaves your PC until confirmed"), never defaulting to LOCAL. A confidentiality indicator that guesses is worse than none, because it gets trusted.

The cheap 80% — probably the only part worth upstreaming

Most of the value needs no user input at all: derive location automatically. Endpoint URL, provider type and tag suffix are already known to the picker — loopback and known-local backends are local, everything else is cloud. A single badge from data you already have.

Optional extras if there's appetite: shipped capability defaults per model family (user-overridable), and populating throughput from measurements Odysseus already records, so users see their tok/s rather than published benchmarks that don't reflect their hardware.

cookbook-hwfit.js ("What Fits?") already reasons about model size against hardware — this is arguably the same knowledge surfaced at selection time rather than serve time.

Implementation notes

Running locally against dev @ 25c9e73, deliberately built to avoid touching upstream files:

• No changes to modelPicker.js — the panel observes #model-picker-label via MutationObserver, and row tips are delegated on document matching .model-switch-item / .mp-model-name. If those identifiers change, the feature goes quiet rather than breaking the picker.
• One added <script type="module"> in static/index.html is the only upstream-file edit.
• Tips are position: fixed, pointer-events: none, flip side when short of room, clamp vertically.

Happy to open a PR if there's interest — or if it's better solved inside the picker itself, treat this as a design note. Mainly raising it to gauge whether the local/cloud indicator specifically is something maintainers would want.

(Related but distinct: #3151 proposes separate pickers for Chat vs Agent mode.)

[eman717]

Screenshots, as promised. Built against dev @ 25c9e73 and running locally.

Docked panel — local model selected

Per-row hover over the open picker — the tip follows the row under the cursor, so a model can be judged before selecting it rather than after.

The case the local/cloud badge exists for. grok-4.3 reads MORE CAPABLE and CLOUD at the same time — it is genuinely a strong model and it leaves the machine. It is reached through an OpenAI-compatible shim on 127.0.0.1:8645 that forwards to xAI, so both obvious heuristics fail: the model id carries no cloud marker, and the base URL is loopback.

[SirThinksAl0t]

I love this ❗

[WGlynn]

The fail-loud instinct here is the right one, and it is in tension with the "cheap 80%" you would upstream, worth resolving before it ships because the two halves disagree using your own example.

You open with the Grok model on 127.0.0.1:8645 that forwards to a third party, and correctly conclude the base URL is not a reliable tell. Then "derive location automatically" proposes: loopback and known-local backends are local, everything else cloud. That rule paints your own Grok setup green LOCAL, the exact case you led with, and the exact guess-that-gets-trusted the fail-loud rule exists to prevent.

The resolution keeps your principle intact: inference can only ever assert CLOUD or WHERE?, never LOCAL. A loopback endpoint is consistent with local but does not prove no egress, because a proxy can bind loopback and forward. So:

• URL or tag points off-box → CLOUD. Safe to assert; a false cloud only over-warns.
• Loopback or known-local backend → still WHERE?, not LOCAL. Cheap to compute, but it can only downgrade trust, never grant the green one.
• LOCAL requires a positive assertion: a config flag the operator sets, or a verified no-egress check. It is the only badge whose wrongness is dangerous, so it is the only one that should not be inferable.

That is still the cheap 80% (you derive CLOUD and WHERE? from data you already have), it just refuses to hand out the single label that, if wrong, does the harm you are trying to prevent. The asymmetry is the whole point: over-warning costs a glance, under-warning costs the confidential input.

[eman717]

You're right, and the example makes it undeniable: the derivation rule I proposed would have painted my own Grok-on-127.0.0.1:8645 setup green — the exact case I opened with. I wrote the fail-loud rule and then, in the same document, proposed a shortcut that defeats it.

The framing that fixed it is yours: LOCAL is the only badge whose wrongness is dangerous, so it is the only one that cannot be inferred. Everything else follows.

Signal	Infer	Why
Off-box URL, known third-party provider, or a :cloud tag	cloud	Safe to assert. A false cloud only over-warns.
Loopback, or a recognised local backend	unknown	Consistent with local; does not prove no egress. A proxy can bind loopback and forward.
Nothing recognised	unknown	The honest answer.
—	local	Never inferred. Operator config flag, or a verified no-egress check.

One case your rules didn't cover, resolved your way: a LAN address (192.168.x, 10.x, *.local). Not a third party if you own the box, but the prompt does leave the machine — so it infers cloud, and the tie breaks toward over-warning.

The other half turned out to be making unknown a first-class value rather than a fallback for a missing field. My local build's location column accepted only local/cloud, which quietly forces every row to take a side — so "I haven't verified this" gets written as a guess into the one field that must never hold one. It is now declarable, and renders as the amber WHERE? badge.

Implemented and tested locally. Posting it here rather than as a PR, since CONTRIBUTING.md routes ideas through Discussions first and nobody has said they want this built yet — happy to open one against dev if a maintainer does.

deriveWhere() — the whole rule

export const CLOUD = 'cloud';
export const LOCAL = 'local';
export const UNKNOWN = 'unknown';

/**
 * Infer where a model runs, from data the picker already has.
 *
 * Returns 'cloud' or 'unknown' — and 'local' if and only if the caller passes
 * a positive assertion. There is no input to this function that makes it
 * derive 'local' on its own, and that is not an oversight.
 */
export function deriveWhere(model = {}) {
  const { baseUrl, provider, id, assertedLocal } = model;

  // Positive assertion wins, and is the only thing that can produce LOCAL.
  if (assertedLocal === true) return LOCAL;

  const prov = provider ? String(provider).trim().toLowerCase() : null;
  if (prov && CLOUD_PROVIDERS.has(prov)) return CLOUD;

  if (id && CLOUD_TAG.test(String(id).trim())) return CLOUD;

  const host = hostOf(baseUrl);
  if (host) {
    if (isLoopback(host)) return UNKNOWN;      // consistent with local, proves nothing
    if (isPrivateLan(host)) return CLOUD;      // off this machine
    return CLOUD;                              // public host
  }

  // No URL, but a backend that runs on the box. Still not evidence of no
  // egress — a local Ollama can be configured to forward.
  if (prov && KNOWN_LOCAL_BACKENDS.has(prov)) return UNKNOWN;

  return UNKNOWN;
}

hostOf() tolerates a bare host:port, isLoopback() covers all of 127.0.0.0/8 plus ::1, and isPrivateLan() covers RFC1918 and .local/.internal.

Tests — the loopback proxy by name, plus the invariant

test('the loopback proxy that motivated the feature is not LOCAL', () => {
  const where = deriveWhere({ baseUrl: 'http://127.0.0.1:8645/v1', id: 'grok-4.3' });
  assert.notEqual(where, LOCAL);
  assert.equal(where, UNKNOWN);
});

test('LOCAL requires a positive assertion and nothing else produces it', () => {
  assert.equal(deriveWhere({ baseUrl: 'http://127.0.0.1:11434', assertedLocal: true }), LOCAL);
  // Strictly boolean true — a truthy string from a config parse must not be
  // enough to hand out the green badge.
  for (const v of ['true', 1, {}, [], 'yes']) {
    assert.notEqual(deriveWhere({ baseUrl: 'http://127.0.0.1:11434', assertedLocal: v }), LOCAL);
  }
});

test('the invariant holds across a spread of inputs', () => {
  // Every combination of inferable inputs, asserting none reaches LOCAL.
  for (const baseUrl of urls) {
    for (const provider of provs) {
      for (const id of ids) {
        const where = deriveWhere({ baseUrl, provider, id });
        assert.ok(isInferable(where));
        assert.notEqual(where, LOCAL);
      }
    }
  }
});

10 cases, node --test, no DOM and no clock. A parallel set on the table parser covers the three-state vocabulary.

Thanks for catching this before it shipped — it was load-bearing and I had it backwards.