Configurable AI model selection with automatic fallback (fixes #91)

[tbartik]

Description

The extension's AI features (skill generation, quizzes, code reviews, did-you-know, rule compilation, etc.) previously selected a single Copilot chat model by family (gpt-5.4-mini → gpt-5-mini → gpt-4.1-mini → gpt-4.1, then first-available). When that model is disabled for the user's Copilot plan or organization, it is still selectable via vscode.lm.selectChatModels but returns an empty response. The empty text then fails JSON parsing, so AI features error out — or silently produce nothing — with no actionable guidance for the user.

This PR makes model selection configurable and resilient:

• New src/core/llm-models.ts — a single source of truth for model selection:
  • New aiEngineerCoach.preferredModel setting plus an in-memory runtime override (so a change applies to the next request immediately and persists across reloads).
  • Auto mode orders available models by a vendor-neutral capability heuristic (scoreFamily) and iterates them, transparently skipping any model that returns an empty response instead of relying on a single pick.
  • De-duplicates models by id (Copilot can return the same model multiple times).
  • Pure helpers (scoreFamily, orderModelsByPreference, dedupeModelsById) are unit-tested.
• panel-llm.ts — callLlm/callLlmJson iterate candidate models and skip empty responses. When every model returns empty, they throw a clear message pointing the user at the new "AI Model" picker. Existing JSON-repair (balanceTruncatedJson) and structured-output fallback behaviour are preserved.
• Dashboard sidebar — adds an "AI Model" dropdown bound to the setting via new listModels / setModel RPCs (panel-html.ts, app.ts, panel-request-service.ts, rpc-types.ts).
• rule-compiler.ts — uses the shared candidate list instead of a hardcoded gpt-4.1 family, so rule compilation benefits from the same fallback.

The capability ordering is deliberately organization-neutral: it does not assume which models any particular plan has enabled — a disabled model simply returns empty and the loop falls through to the next candidate.

Related Issues

Fixes #91

Checklist

• npm run check passes (typecheck + lint + spellcheck + knip + tests)
  • Note: one pre-existing, environment-sensitive test (src/core/parser-vscode.test.ts › findVsCodeDirs) fails on a clean main checkout as well; it is unrelated to this change.
• Changes are covered by tests (if applicable) — src/core/llm-models.test.ts (11 cases for scoring, ordering, and de-duplication)
• Documentation updated (if applicable) — the new setting is self-documenting via its markdownDescription in package.json

[tbartik]

Heads-up on overlap with #98 ("add Gemini CLI support and provider-agnostic AI settings").

Both PRs add a user-configurable preferred-model setting + a model picker, so they touch some of the same files (panel-llm.ts, panel-request-service.ts, panel-html.ts, app.ts, package.json). A couple of things worth calling out so maintainers can decide how to sequence them:

This PR is specifically the fix for #91. #98 refactors model selection (preferred id + a hardcoded high-capability family list) but still does a single selectModel() returning models[0] with no empty-response handling. The root cause of #91 is that a model disabled for the user's Copilot plan/org is still selectable yet returns an empty response — so the single-pick path keeps crashing. This PR instead iterates all candidate models and skips empty responses, surfaces an actionable error pointing at the model picker when every model is empty, routes the rule compiler through the same logic, and adds unit tests for the selection/ordering/dedup helpers.

Possible conflict: the setting key differs — #98 uses aiEngineerCoach.preferredModelId, this PR uses aiEngineerCoach.preferredModel. Happy to align on whichever name maintainers prefer.

I'm glad to rebase this on top of #98 (layering the empty-response fallback onto their selection logic) or vice-versa — whatever ordering is easiest to review. The Gemini CLI parser in #98 is independent and valuable on its own.

[tbartik]

@microsoft-github-policy-service agree

@microsoft-github-policy-service agree