CS-11249: realm-independent fallback models + host wire/picker integration by FadhlanR · Pull Request #4949 · cardstack/boxel

FadhlanR · 2026-05-25T12:04:46Z

Summary

Adds DEFAULT_FALLBACK_MODELS (6 curated rows) + DEFAULT_FALLBACK_MODEL_ID to runtime-common/matrix-constants.ts so capability fields are available even when SystemCard.modelConfigurations is missing — fixing the silent-tools-off failure mode for the curated set.
New matrix-service.resolveActiveLLMConfig(roomId, model, callerOverrides?) — layered resolver: caller overrides → SystemCard → DEFAULT_FALLBACK_MODELS → most-recent valid prior app.boxel.active-llm event in this room → conservative floor ({ toolsSupported: false, inputModalities: ['text'] }). sendActiveLLMEvent calls it internally so every wire write carries defined toolsSupported + inputModalities.
FallbackModelConfig is intentionally minimal: { modelId, displayName, toolsSupported, inputModalities }. reasoningEffort is a user choice (never auto-filled from the curated fallback); supportsReasoning has no consumer in this project.
The picker's fallback branch in room.gts swaps the legacy 22-entry DEFAULT_LLM_LIST for the curated 6; the existing usedLLMs merge keeps non-curated history selectable.
T1 only. Out of scope (follow-up tickets): T2 (CS-11252) ai-bot fill on the receive side, T3 (CS-11254) fallback banner, T4 (CS-11255) retire DEFAULT_LLM / _LIST / _ID_TO_NAME.

Test plan

cd packages/realm-server && TEST_FILES=fallback-models-test pnpm test — shape unit tests pass locally
cd packages/host && pnpm lint:js — clean
cd packages/runtime-common && pnpm lint:js — clean
Host integration fallback-models-test.gts — 8 scenarios: resolver layers (curated caps fill-in, conservative floor for unknown model, caller override priority, rehydrate from prior valid event, skip prior broken event), wire writer (curated + override), picker renders the 6 displayNames

🤖 Generated with Claude Code

…ation Closes the silent-tools-off bug for the curated 6 models on the host side. sendActiveLLMEvent now fills missing toolsSupported/inputModalities from the bundled constant when SystemCard.modelConfigurations doesn't cover the chosen model; the AI assistant picker uses the same constant in its SystemCard-missing fallback path. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Adds reasoningEffort to FallbackModelConfig (default 'medium' for the curated rows) and changes sendActiveLLMEvent to fall back to it via ?? when caller / SystemCard hasn't set one. Inlines the refresh procedure into matrix-constants.ts and drops the scratch plan + refresh docs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

github-actions · 2026-05-25T12:11:00Z

Preview deployments

Host staging preview

Host production preview

Host Test Results

1 files ± 0 1 suites ±0 3h 35m 1s ⏱️ + 1h 43m 50s
2 825 tests + 93 2 808 ✅ + 91 15 💤 ± 0 1 ❌ +1 1 🔥 +1
5 447 runs +2 696 5 414 ✅ +2 678 30 💤 +15 2 ❌ +2 1 🔥 +1

Results for commit add370b. ± Comparison against earlier commit 789ad81.

For more details on these errors, see this check.

Realm Server Test Results

1 files ± 0 1 suites ±0 8m 27s ⏱️ +50s
1 507 tests +20 1 507 ✅ +20 0 💤 ±0 0 ❌ ±0
1 598 runs +20 1 598 ✅ +20 0 💤 ±0 0 ❌ ±0

Results for commit add370b. ± Comparison against earlier commit 789ad81.

Move capability resolution out of sendActiveLLMEvent into a layered resolveActiveLLMConfig on MatrixService so every wire write carries defined toolsSupported + inputModalities — caller overrides, then SystemCard, then DEFAULT_FALLBACK_MODELS, then prior valid event in the same room, then a conservative floor. sendActiveLLMEvent becomes a dumb wire writer with a required caps argument. Trim FallbackModelConfig to { modelId, displayName, toolsSupported, inputModalities }: reasoningEffort is a user choice (never auto-filled) and supportsReasoning had no consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Drop the two-step resolve-then-send dance at every call site by calling resolveActiveLLMConfig inside sendActiveLLMEvent and exposing the same callerOverrides arg on its signature. The resolver stays public for tests and any future query-only consumer. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

getRoomState returns the state event content directly, not a wrapper — the two sendActiveLLMEvent tests were dereferencing .content on the content and hitting undefined. The activateLLMTask edit was redundant once sendActiveLLMEvent's callerOverrides arg accepts the full caps shape; revert to main so the existing SystemCard-derived config flows through unchanged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

This PR introduces a realm-independent fallback model list and a layered resolver in the host Matrix service to ensure toolsSupported and inputModalities are always defined on app.boxel.active-llm wire events, even when SystemCard.modelConfigurations is unavailable. It also updates the host LLM picker fallback to use the curated fallback set while preserving previously-used (non-curated) models, and adds coverage across runtime-common, realm-server, and host integration tests.

Changes:

Add DEFAULT_FALLBACK_MODELS + DEFAULT_FALLBACK_MODEL_ID (curated capability surface) to runtime-common/matrix-constants.ts.
Add matrix-service.resolveActiveLLMConfig() and ensure sendActiveLLMEvent() always writes resolved capability fields.
Update the host room picker fallback branch to use the curated fallback list (merged with usedLLMs) and add tests validating resolver + picker behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
packages/runtime-common/tests/fallback-models-test.ts	Adds shared tests that validate the curated fallback list shape and invariants.
packages/runtime-common/matrix-constants.ts	Introduces curated fallback models + default fallback model id for capability fill-in.
packages/realm-server/tests/index.ts	Registers the new fallback-models test in the realm-server test suite.
packages/realm-server/tests/fallback-models-test.ts	Runs the runtime-common shared fallback-model tests under realm-server.
packages/host/tests/integration/components/ai-assistant-panel/fallback-models-test.gts	Adds host integration tests for resolver layering, wire writes, and picker rendering.
packages/host/app/services/matrix-service.ts	Implements layered resolver and ensures active-llm state events always include capability fields.
packages/host/app/components/matrix/room.gts	Switches picker fallback list from legacy defaults to curated fallback list + used history.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

FadhlanR · 2026-05-28T02:12:50Z

+// curated set changes. Derivation rules mirror
+// `packages/host/app/commands/sync-openrouter-models.ts:67-141`:


Good catch — fixed in add370b. The comment now points at the computed fields on the OpenRouterModel card itself (packages/openrouter-realm/openrouter-model.gts), which is where toolsSupported = supportedParameters.includes('tools') and inputModalities = architecture.inputModalities actually live. Dropped the line range so the reference doesn't bit-rot.

FadhlanR · 2026-05-28T02:12:57Z

+      let priorEvents = (roomData?.events ?? [])
+        .filter((e) => e.type === APP_BOXEL_ACTIVE_LLM)
+        .map((e) => e as ActiveLLMEvent)
+        .filter(
+          (e) =>
+            e.content.model === model &&
+            e.content.toolsSupported !== undefined &&
+            e.content.inputModalities !== undefined,
+        )
+        .sort((a, b) => (b.origin_server_ts ?? 0) - (a.origin_server_ts ?? 0));
+      let mostRecent = priorEvents[0];


Fixed in add370b — Layer 4 is now a single for…of pass that tracks the newest matching event by origin_server_ts directly, no intermediate filtered array and no sort.

lukemelia

Copilot's recommendations are good. Otherwise, looks good to me.

- Comment in matrix-constants.ts now references the OpenRouterModel card's computed fields (the actual home of the toolsSupported / inputModalities derivation), not sync-openrouter-models.ts which only writes card JSON. - Layer 4 of resolveActiveLLMConfig walks roomData.events once tracking the newest matching APP_BOXEL_ACTIVE_LLM event by origin_server_ts, instead of filtering into an array and sorting. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

FadhlanR and others added 2 commits May 25, 2026 13:14

FadhlanR and others added 3 commits May 26, 2026 11:50

FadhlanR marked this pull request as ready for review May 26, 2026 08:15

FadhlanR requested review from a team and lukemelia May 26, 2026 08:29

habdelra requested a review from Copilot May 26, 2026 13:11

Copilot started reviewing on behalf of habdelra May 26, 2026 13:12 View session

Copilot AI reviewed May 26, 2026

View reviewed changes

lukemelia reviewed May 26, 2026

View reviewed changes

FadhlanR merged commit 71e0d22 into main May 28, 2026
98 of 101 checks passed

FadhlanR mentioned this pull request May 29, 2026

Fill missing LLM capability fields from fallback constant (CS-11252) #5037

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CS-11249: realm-independent fallback models + host wire/picker integration#4949

CS-11249: realm-independent fallback models + host wire/picker integration#4949
FadhlanR merged 6 commits into
mainfrom
cs-11249-t1-fallback-models

FadhlanR commented May 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented May 25, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

FadhlanR May 28, 2026

Uh oh!

FadhlanR May 28, 2026

Uh oh!

lukemelia left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// curated set changes. Derivation rules mirror
		// `packages/host/app/commands/sync-openrouter-models.ts:67-141`:

Conversation

FadhlanR commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

github-actions Bot commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Preview deployments

Host Test Results

Realm Server Test Results

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

FadhlanR May 28, 2026

Choose a reason for hiding this comment

Uh oh!

FadhlanR May 28, 2026

Choose a reason for hiding this comment

Uh oh!

lukemelia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

FadhlanR commented May 25, 2026 •

edited

Loading

github-actions Bot commented May 25, 2026 •

edited

Loading