Skip to content

Spec2Contract Model Groups

Grzegorz Nowak edited this page Oct 12, 2026 · 4 revisions

Spec-to-contracts — Model-Groups (book ↔ in-tests contracts)

This area covers the Model Groups subsystem: registry-backed group definitions, their two storage scopes, the /model-groups TUI manager, the modality/constraint engine, spawn-time routing, and the #group autocomplete + agent guidance. Left-space IDs are PB-GROUPS-01…PB-GROUPS-47 (47 rows). Contracts are carried mainly by the dedicated fixture family T15–T24 (router, crud, constraints, modalities, modality lexicon, display, autocomplete, integration, TUI, frontmatter), with cross-area support from T01/T03 for spawn routing + identity and T48 for the compile-time type contract.

Junction rows

Claim ID Assertion (terse) Fixture(s) Status Notes
PB-GROUPS-01 A group maps to a list of provider/model refs, each optionally carrying a thinking level. T16, T48 ✅ asserted model-groups-crud.test.ts — "model groups store creates, round-trips, validates, renames, updates, deletes, and moves" (thinking "high" round-trips; absent thinkingLevel not persisted/invented). T48 tests/typecheck/model-groups-producer.ts pins ModelGroupModel {provider, modelId, thinkingLevel?}.
PB-GROUPS-02 Global groups persist at ~/.pi/agent/pi-schematic/model-groups.json. T16 ◐ partly asserted missing: no independent assertion of the global path literal — tests write/read via modelGroupsPath("global", …) ("model groups load recovery handles malformed, schema-invalid, unsupported version, and backup failure"), so a wrong global root would still pass. Would be sealed by asserting modelGroupsPath("global", cwd) === path.join(homedir(), ".pi/agent/pi-schematic/model-groups.json").
PB-GROUPS-03 Project groups persist at <cwd>/.pi/pi-schematic/model-groups.json (Pi project config dir name). T16 ✅ asserted "model groups use branded paths, global-only access, canonical own keys, and native max" asserts modelGroupsPath("project", cwd, "branded-pi") === path.join(cwd, "branded-pi", "pi-schematic", "model-groups.json"); the default segment is Pi's CONFIG_DIR_NAME.
PB-GROUPS-04 Persisted document uses version 2 and a groups map. T16 ✅ asserted "v1 migration is in-memory until the first successful mutation writes v2 without an invented override"; "model groups strictly partitions schema and legacy version domains"; "v2 constraint envelope preserves explicit empty and opaque slots canonically".
PB-GROUPS-05 Names are trimmed of surrounding whitespace; empty-after-trim names are rejected. T16, T23 ✅ asserted "model groups direct save canonicalizes unique keys and rejects empty/colliding keys before write"; "model groups use branded paths…" (` ${name} ` → trimmed); model-groups-tui.test.ts — "decodes then canonicalizes prototype-sensitive names and rejects malformed escapes".
PB-GROUPS-06 Project groups available only in a trusted project; otherwise access is global-only. T16, T22 ✅ asserted "model groups use branded paths, global-only access…" (project probe forbidden, project createGroup throws /global-only/); model-groups-integration.test.ts — "model groups untrusted root flow never probes project data and publishes global-only state to TUI".
PB-GROUPS-07 A project group shadows a same-named global group when resolving an effective role. T15, T16, T22 ✅ asserted model-groups-router.test.ts — "effective model group names use project-over-global names"; crud round-trip asserts validation.shadowedByProject; integration notifies 1 project overrides.
PB-GROUPS-08 Saves write a temporary JSON file and rename it into the selected scope's path. T16 ✅ asserted "model groups rename failure removes the generated temp file and preserves committed bytes" (.tmp name pattern, rename phase); "model groups persistence failures throw typed errors and preserve committed state" (temp-write/rename phases).
PB-GROUPS-09 /model-groups opens the interactive manager in TUI mode. T22 ✅ asserted "…/model-groups command registers and opens ctx.ui.custom with live registry/cwd".
PB-GROUPS-10 Manager cannot run in non-TUI mode; RPC receives a TUI-required notification. T22 ✅ asserted "root-recorded /model-groups rejects RPC and skips JSON/print before custom UI" (rpc → "/model-groups requires TUI mode"; json/print silent; custom never called).
PB-GROUPS-11 + Add group immediately creates an empty group, defaulting to project scope when trusted and global otherwise. T23 ✅ asserted model-groups-tui.test.ts — "computes unique new-group names and opens editor after create" (createGroup project, 0 models); "clamps final add row and enforces global-only access" (global-only policy → global).
PB-GROUPS-12 The group editor allows moving a group between project and global locations. T23 ✅ asserted "move, wizard add, model thinking, and remove persist through store calls" (move:review:global); "notifies and preserves location on move collision".
PB-GROUPS-13 Selecting the Name row activates inline input; committing renames the persisted group. T23 ✅ asserted "renders name editing inline and preserves edit/commit transitions" (Esc and Enter both commit renameGroup).
PB-GROUPS-14 Adding a member walks through provider, model, and thinking selection. T23 ✅ asserted "wizard renders provider/model/thinking steps and preserves state on add failure"; "Esc and left-arrow share wizard back-step behavior".
PB-GROUPS-15 The add-model picker offers only registry-available models with configured authentication. T23 ✅ asserted "Add-model picker shows capability chips per model" (gpt-no-auth absent); "model groups TUI wizard renders provider/model/thinking steps…"; "searchable Model step filters raw fields and handles no matches safely" (hidden absent).
PB-GROUPS-16 Picker searches the complete selected provider's results by model ID, provider, provider/ID, and display name. T23 ✅ asserted "model groups TUI searchable Model step filters raw fields and handles no matches safely" (queries alpha-id, openai/alpha-id, Friendly, openai); "fuzzy search excludes synthetic provider-space-id matches".
PB-GROUPS-17 Picker displays ten model rows at a time without limiting the underlying results to ten. T23 ✅ asserted "Model SelectList keeps all results behind a ten-row viewport and owns wrapping" (12 models, (1/12), navigable to model-11).
PB-GROUPS-18 The thinking options the manager offers for a member are inherit, the levels the screen derives for the member's selected model, and the member's stored level when that level falls outside that set. T23 ✅ asserted "model edit renders identity/status and filters thinking options" (claude → inherit only; gpt-5 → full list incl. max); "clamps final add row and enforces global-only access" offers native max; "MODEL_EDIT stale level on an available model keeps its row after inherit and Remove model last" and "MODEL_EDIT treats a stale off level on a non-reasoning model as current, unavailable" show the stored level added to the offered set.
PB-GROUPS-19 The edit-model screen changes a member's thinking level or removes that member. T23 ✅ asserted "notifies and preserves model edit state when updateGroup fails" (thinking change then D remove); "move, wizard add, model thinking, and remove persist through store calls".
PB-GROUPS-20 Deleting a group requires selecting Delete group rather than the default Keep group. T23 ✅ asserted "selected markers and primary labels use accent token" (default selected Keep group, Down → Delete group); "list renders validation summary, health tags, add row, no Validate row, and confirmed D delete".
PB-GROUPS-21 Deleting a same-named group in one scope leaves the other scope's group intact. T16, T23 ✅ asserted crud "model groups store creates, round-trips, validates, renames, updates, deletes, and moves" (otherScopeHasOverride === true; other scope survives the delete/move); TUI "…confirmed D delete" asserts "Same-name group in the other scope remains unaffected".
PB-GROUPS-22 Group rows identify scope, member count, thinking selections, capability letters, and validation warnings. T23 ✅ asserted "list renders validation summary, health tags, add row…" ([project], 1 models, ⚠ degraded, ✗ unavailable); "renders modality labels, warnings, and stale override choices" (T I, ⚠ no common modalities, ⚠ stale modality override: reasoning); "selected markers…" row label review [project] 1 models inherit.
PB-GROUPS-23 The production capability registry contains exactly one constraint type, modalities. T17 ✅ asserted model-groups-constraints.test.ts — "injected scalar traverses resolution, aggregation, persistence, reconciliation, and production isolation" (productionConstraintRegistry.descriptors keys ["modalities"]; injected key undefined).
PB-GROUPS-24 The supported modality vocabulary is text, image, and reasoning. T17, T16, T19 ✅ asserted T17 "descriptor codecs report errors and retain vocabulary ordering"; crud "v2 config load rejects non-array, duplicate, and out-of-vocabulary modality constraints"; model-groups-modality.test.ts — "MODALITY_LETTER / MODALITY_FG cover the established single-letter lexicon".
PB-GROUPS-25 A model's text/image capabilities follow its registry input modalities. T18 ✅ asserted model-groups-modalities.test.ts — "derives ordered common, supported, and override-effective modalities from the live registry".
PB-GROUPS-26 A model advertises reasoning when its registry model has reasoning: true. T18 ✅ asserted same "derives ordered common, supported…" test (rich reasoning:true → supported includes reasoning; text reasoning:false excluded); model-groups-constraints.test.ts — "engine preserves unresolved members as unknown facts".
PB-GROUPS-27 Without an override, a group's advertised capabilities are the union of its resolved members' modalities. T15, T18, T22 ✅ asserted model-groups-router.test.ts — "automatic mixed groups default effective to the union so a capability routes to a capable member"; T18 "derives ordered common, supported, and override-effective modalities…"; model-groups-integration.test.ts — "before_agent_start exposes union-effective modalities for automatic mixed groups".
PB-GROUPS-28 An override may narrow but cannot add a modality none of its members supports when creating/updating. T16, T18, T15 ✅ asserted crud "low-level save permits shape-valid overrides while CRUD rejects union-cap-invalid overrides" and "CRUD rejects unsupported injected overrides before any write"; T18 "caps stale overrides without mutation and restores them when catalog support returns"; T15 "explicit override stays a subtractive ceiling even when a capable member exists".
PB-GROUPS-29 Optional-media toggle edits image; text is a required base when supported; reasoning is not toggleable. T23 ✅ asserted "modality editor commits override and Automatic through updateGroup"; "text-only modalities screen is inert and states Automatic"; "editor rows show per-model capability chips and modalities screen shows dynamic Automatic/Override status" (T text [required], I image [on], no reasoning row).
PB-GROUPS-30 Add-model and member rows show compact T/I chips; reasoning is omitted from these media chips. T19, T23 ✅ asserted model-groups-modality.test.ts — "modalityLetterRun canonicalizes order and excludes reasoning by default"; TUI "Add-model picker shows capability chips per model" (T, T I); "editor rows show per-model capability chips…".
PB-GROUPS-31 Group guidance includes only effective group names and capabilities, not membership or thinking levels. T22 ✅ asserted model-groups-integration.test.ts — "before_agent_start injects fresh names-and-effective-modalities guidance" (Available Model Groups: review (text, image, reasoning); doesNotMatch /gpt-5/, /model-groups\.json/).
PB-GROUPS-32 A #-prefixed group mention autocompletes effective names and shows model/thinking details. T21 ✅ asserted model-groups-autocomplete.test.ts — "#group autocomplete suggests effective live group names and delegates elsewhere" (openai/gpt-5 • high; anthropic/claude-sonnet-4 • inherit); "registerModelGroupAutocomplete uses ctx.ui.addAutocompleteProvider once".
PB-GROUPS-33 Boot validation marks refs missing from the registry as unavailable and warns when counts are nonzero. T16, T22 ✅ asserted crud "creates, round-trips…" (unavailableRefs: [{missing/nope}]); integration "index session_start stores model group validation and notifies load and validation issues" (notifies 1 unavailable model references, and "index session_start does not notify when load and validation issues are absent" seals the zero-count branch).
PB-GROUPS-33 Boot validation marks present-but-unauthenticated refs as unavailable. T15 ◐ partly asserted Router excludes unauthenticated usable members (model-groups-router.test.ts — "capability pre-selection excludes unauthenticated capable models and picks randomly from the capable set"), but no test drives validateModelGroups with a registry-present model whose hasConfiguredAuth is false to assert it enters validation.unavailableRefs. Would be sealed by such a boot-validation case.
PB-GROUPS-34 Corrupt/schema-invalid JSON loads as an empty configuration for that scope with a backup attempt. T16 ✅ asserted model-groups-crud.test.ts — "model groups load recovery handles malformed, schema-invalid, unsupported version, and backup failure" (empty groups, .bak written, backupFailed branch).
PB-GROUPS-34 Load issues are surfaced to the operator as a warning. T22, T20 ✅ asserted integration "index session_start notifies corrupt/schema/unsupported load issues", "…schema-invalid load issues", "…includes backup-failure detail…", "…boot load-issue notifications escape hostile source, backup, and error detail fields"; model-groups-display.test.ts seals the escape codec the warning paths/messages pass through.
PB-GROUPS-35 spawn accepts an optional exact group name; omitting it inherits the parent's model and thinking. T15, T01 ✅ asserted model-groups-router.test.ts — "omitted and unknown groups inherit parent route with fallback metadata" (status:"inherited"); spawn.test.ts — "spawn execute composes Model Group routing with readonly child guards" (explicit group routed) and "spawn forwards requested thinking and reports the session effective thinking" (no group → details.route { status: "inherited" }).
PB-GROUPS-36 Unknown group names visibly fall back to the parent model/thinking rather than selecting a different group. T15, T03 ✅ asserted T15 "omitted and unknown groups inherit parent route with fallback metadata" (status:"unknown-fallback", requestedGroup); spawn-render.test.ts — "spawn result identity formats default routed and unknown fallback" (rev? fallback → openai/parent • medium).
PB-GROUPS-37 A known group with zero members fails before a child session is created. T15, T01, T24 ✅ asserted T15 "known empty and all-unusable groups fail clearly" (reason:"empty"); spawn.test.ts — "executeSpawn propagates unusable-group errors before creating child work" (sessionFactory not called); model-group-frontmatter.test.ts — "empty model-group (SpawnRouteError) blocks execution with error".
PB-GROUPS-38 A known group with no registry-resolved usable member fails before a child session is created. T15, T01 ✅ asserted T15 "known empty and all-unusable groups fail clearly" (reason:"no-usable-models"); spawn.test.ts — "executeSpawn propagates unusable-group errors before creating child work" (no child registered).
PB-GROUPS-38 A known group whose resolved members all lack configured authentication fails the same way. T15 ◐ partly asserted T15's auth fixture has auth-configured members alongside the unauthenticated one, so it proves exclusion but not the all-unauthenticated → no-usable-models failure. Would be sealed by a group whose only member is registry-present with hasConfiguredAuth === false.
PB-GROUPS-39 A usable routed member is drawn uniformly by entry position on every spawn; repeated calls can draw the same entry. T15, T01 ◐ partly asserted T15 "routed empty modality requirements select by RNG across usable members" (exactly one RNG draw, index from rng) and "capability pre-selection excludes unauthenticated capable models and picks randomly from the capable set" (every selection consults the RNG, no cursor short-circuit). missing: no assertion of uniform distribution or that an unchanged pool can redraw the same entry across calls. Would be sealed by repeated-draw / repeat-entry assertions.
PB-GROUPS-40 spawn.constraints.modalities requires { required: [...] }, not a bare array; unknown/invalid keys/values are rejected. T01, T15 ✅ asserted spawn.test.ts — "spawn tool schema validates constraints via Value.Check" (bare array, duplicates, out-of-vocab, non-object, stringified, array-with-embedded-required → false); "spawn requirements validate the canonical envelope and keep its shape"; "executeSpawn rejects unknown requirements before factory or session publication"; T15 "bare-array constraint requirements are rejected by the descriptor codec".
PB-GROUPS-41 For declared capabilities, the router preferentially samples usable members satisfying the exact model requirement. T15, T01 ✅ asserted T15 "required modalities prefer a capable member and still reject a wholly-incapable group" and "automatic mixed groups default effective to the union so a capability routes to a capable member"; spawn.test.ts — "spawn routes capability requirements to capable members via random selection on the filtered pool".
PB-GROUPS-42 A declared capability must pass both the group's effective ceiling and the selected model's check; failure occurs before child creation. T15, T01 ✅ asserted T15 "explicit override stays a subtractive ceiling even when a capable member exists" (missingFromGroup set, missingFromModel empty); spawn.test.ts — "executeSpawn propagates missing modalities before creating child work" and "registered spawn tool rejects missing modalities before creating child work" (sessionFactory/factory not called).
PB-GROUPS-43 A routed member's thinking override wins over spawn.thinking/inherited; the result is clamped to the model's supported levels. T01, T24 ✅ asserted spawn.test.ts — "spawn execute composes Model Group routing with readonly child guards" (group entry low beats requested xhigh); model-group-frontmatter.test.ts — "model-group-only frontmatter applies and records the group's effective thinking" ("group thinking is clamped for the routed model").
PB-GROUPS-44 Routed identity shows group → provider/model • thinking; unknown-group identity shows group? fallback → provider/model • thinking. T03 ✅ asserted spawn-render.test.ts — "spawn result identity formats default routed and unknown fallback" (review → openai/gpt-routed • high, rev? fallback → openai/parent • medium; controlled chars escaped).
PB-GROUPS-45 A #group-name mention directs the agent to route the spawn through that exact group (exact name, no fallback). — ⚠️ no test yet No test asserts the #group-name sentence of modelGroupsPromptSection (index.ts:457-464); model-groups-integration.test.ts asserts only the section header, the Available Model Groups: … line, and /constraints/. A test would assert the guidance matches /#group-name/ and the exact-name/no-fallback sentence.
PB-GROUPS-46 The edit-model screen opens with the cursor on the member's configured thinking level and marks that row ✓ current when the screen offers it. The marker reflects the screen's offered options, independently of authentication, which the Status: line reports separately. T23 ✅ asserted model-groups-tui.test.ts — "MODEL_EDIT opens on the configured thinking level and preserves it on Enter", "MODEL_EDIT opens on a mid-list configured level and marks only that row current", "MODEL_EDIT marks the default inherit row current when no level is configured", "MODEL_EDIT opens on a supported level for a registered-but-unauthenticated model" (marker independent of Status: unavailable).
PB-GROUPS-47 A member's stored thinking level that the edit screen does not offer for its model — a level the model no longer lists, off on a model that does not reason, or any level when the model is not in the registry — stays on screen as its own row immediately after inherit, marked ⚠ current, unavailable, and is preserved unless the operator deliberately chooses another option. T23 ✅ asserted "MODEL_EDIT shows an unsupported/unavailable current level as current, unavailable and preserves it", "MODEL_EDIT stale level on an available model keeps its row after inherit and Remove model last", "MODEL_EDIT keeps an unsupported level after inherit when the model has other supported rows", "MODEL_EDIT stale level on an unavailable model keeps Status unavailable and preserves the value", "MODEL_EDIT treats a stale off level on a non-reasoning model as current, unavailable".

Empty surfaces in this area

No PB-GROUPS claim is entirely unrepresented except PB-GROUPS-45 (⚠️): no test asserts the #group-name mention sentence in the agent guidance. The prompt-section test hangs on the header, the names list, and /constraints/ only; it would be sealed by asserting the guidance matches /#group-name/ together with the exact-name/no-fallback sentence.

Partial surfaces (◐) and what would seal the missing portion:

  • PB-GROUPS-02 — no independent assertion of the global config path literal; tests reuse modelGroupsPath("global", …). Seal by asserting the home-rooted path equals ~/.pi/agent/pi-schematic/model-groups.json.
  • PB-GROUPS-33 (auth aspect) — no boot-validation test with a registry-present, unauthenticated ref entering validation.unavailableRefs. Seal by driving validateModelGroups with hasConfiguredAuth → false.
  • PB-GROUPS-38 (auth aspect) — no test for a group whose only member is present but unauthenticated failing with reason:"no-usable-models". Seal by that routing case.
  • PB-GROUPS-39 (uniformity aspect) — RNG-based selection is asserted deterministically, but uniform distribution and repeat-of-same-entry across calls are not. Seal by repeated-draw assertions over an unchanged pool.

Area coverage summary

Claims Junction rows ✅ ◐ ⚠️
47 50 45 4 1

Clone this wiki locally