Skip to content

feat: add Model Lifecycle Intelligence for host inventory, upgrades, and swaps #110

Open
Open
feat: add Model Lifecycle Intelligence for host inventory, upgrades, and swaps#110
Labels
enhancementNew feature or request

Description

@pacphi
pacphi
opened on Aug 5, 2026

Executive brief

Agentic Kit can already report which CLI/package versions are installed and which releases are available, and it already knows the configured per-activity host/model routes. It cannot yet answer the equivalent model-lifecycle questions across Claude Code, Codex, and OpenCode:

  • Which models are configured, effectively selected, and actually observed in use on each host?
  • Which models were added, removed, hidden, deprecated, or assigned a migration target since the last trustworthy catalog snapshot?
  • Which aliases silently resolved to a different concrete model?
  • Which configured routes now point at a model that is unavailable, policy-blocked, stale, or approaching retirement?
  • Which models are mechanically compatible swap candidates, and which are evidence-backed upgrades or cheaper equivalents?
  • Which Ruflo router candidates and Agentic-QE model overrides have drifted from the models the hosts can actually use?

This proposal adds Model Lifecycle Intelligence: a local-first, provenance-aware inventory and change-detection layer for host model catalogs, configured bindings, observed use, lifecycle notices, and route impact. It introduces a dedicated read-only ak models command family, adds a compact model-health row to ak status, and adds a Models view beneath the Dashboard's existing Usage area.

The central rule is that newer, discoverable, entitled, configured, observed, compatible, and better are different claims. A host or provider catalog may prove that a model exists; it does not prove that the current account can use it. A migration target may prove a supported replacement; it does not prove equal quality or lower cost. Evidence-backed quality and economic recommendations remain the responsibility of Route Intelligence in #109.

Representative output:

$ ak models status
Host       Configured / effective       Observed       Catalog                         Attention
Claude     sonnet → claude-sonnet-…     sonnet         4 visible · fresh 3h           alias target changed
Codex      gpt-5.4 / high               gpt-5.4        7 visible · fresh 18m           upgrade → gpt-5.6-terra
OpenCode   openrouter/…                  provider/…     126 project-eligible · 2d old   refresh available

2 migration opportunities · 1 alias change · 1 stale catalog
Run `ak models diff` for changes or `ak models plan --activity coder` for route impact.

All names and counts above are illustrative. Shipped output must come from source-attributed local evidence.

Table of contents

Background and current state

Agentic Kit already has most of the facts needed to build the feature, but they are fragmented:

  • src/lib/routing.mjs owns the canonical activity routes and currently carries a manually verified, static known-good model catalog (MODEL_CATALOG, PROVIDER_MODEL_CATALOG, and MODEL_CATALOG_VERIFIED). The help text correctly allows any model the host accepts, but the static catalog cannot detect per-machine additions, retirement notices, aliases, entitlement, or change over time.
  • src/lib/adapters/registries.mjs already defines capability-driven host/provider/projection/observability registries. Provider capabilities include modelDiscovery and runtimeDiscovery, and observability supports catalog evidence. Today only Ollama advertises those capabilities; the seam exists but is not yet generalized into a host-scoped inventory.
  • src/commands/status.mjs already aggregates installed/latest versions, health, drift, and fixes into human and JSON rows. It is the right place for a terse, cache-only summary, not a full model matrix.
  • src/commands/x/host.mjs and src/lib/routing.mjs already own route mutation. A new inventory feature should not create a competing model-policy source of truth.
  • src/lib/providers.mjs projects routing into Agentic-QE's .agentic-qe/llm-config.json, including per-agent overrides when supported.
  • Transcript indexing already observes models used by Claude Code, Codex, and OpenCode, while ak run records selected routes and bounded attempts. These are usage facts, not catalog or entitlement facts.

There is also a concrete opportunity in the current host surfaces:

  • The local Codex model cache includes catalog freshness, client version, visibility, reasoning levels, context windows, and explicit upgrade.model migration targets for retiring entries. That proves a useful adapter input exists on this host, but its schema must be treated as host-owned and versioned rather than copied into core policy.
  • opencode models [provider] provides a project/provider-scoped model list, optional verbose metadata, and explicit --refresh of the Models.dev-backed cache.
  • Claude Code supports configured model values, aliases, model overrides, managed availableModels, and gateway model discovery, but an alias or published model list does not by itself establish current-account entitlement.

The governing ADRs are living plans:

  • ADR-0001 is superseded in part by ADR-0020 (2026-07-23; updated 2026-07-30). Its one-routing-policy principle remains; historical compatibility surfaces do not.
  • ADR-0002 is amended by ADR-0020 (2026-07-23; updated 2026-07-30). The activity vocabulary remains and is owned by the in-repository runner.
  • ADR-0005 is Implemented (2026-07-23; updated 2026-08-04). The Dashboard remains read-only and organized into three primary areas.
  • ADR-0009 is Implemented (2026-07-25; updated 2026-08-04). It owns local transcript analytics, graded evidence, and Usage presentation.
  • ADR-0011 is Proposed (2026-07-27; updated 2026-07-30). Its local-model claims remain unmeasured and must not be treated as accepted behavior.
  • ADR-0016 is Accepted, with compatibility clauses superseded by ADR-0020 (2026-07-28; updated 2026-07-30). It is the architectural basis for capability-driven discovery adapters.
  • ADR-0017 is Accepted, with compatibility references amended by ADR-0020 (2026-07-28; updated 2026-08-04). OpenCode remains an opt-in managed host and its provider identity must stay independent.
  • ADR-0020 is Implemented (2026-07-30). There must be one stable GA surface per capability.
  • ADR-0021 is Accepted (2026-07-31; updated 2026-08-03). Host, inference provider, provider provenance, and model remain independent facts.

No governing ADR currently defines a normalized, snapshot-based model inventory or model-lifecycle contract. This issue should add a new Proposed ADR and amend the existing ADRs only when their shipped behavior changes.

Problem statement

Package upgrades are mostly scalar: installed version X, registry version Y, upgrade available. Models are not.

A model name can be:

  • a mutable alias such as sonnet, default, or another host-defined label;
  • a concrete host model slug;
  • a provider-qualified id such as provider/model;
  • a private gateway deployment id;
  • a local model name plus a mutable digest;
  • visible in a public catalog but not entitled for the current account;
  • entitled but blocked by managed policy;
  • accepted in configuration but not currently routable;
  • deprecated with a first-party migration target;
  • mechanically compatible but behaviorally different; or
  • demonstrably better or cheaper only for certain operations.

Consequently, a single installed → latest column would be misleading. Agentic Kit needs a multidimensional inventory and a provenance-aware diff that can say exactly which source established each claim.

The feature must also reconcile multiple consumers. The same model reference may appear in canonical kit.json routes, escalation ladders, Claude/Codex/OpenCode host config, Agentic-QE agent overrides or fallbacks, Ruflo candidate registries, learned Route Intelligence evidence, and historical transcripts. A model change can make any of these stale without changing the package version.

Goals

  1. Inventory configured, effective, observed, discoverable, entitled, policy-allowed, routable, and lifecycle state independently for every managed host.
  2. Detect trustworthy additions, removals, visibility changes, alias-target changes, deprecations, and first-party migration targets over time.
  3. Explain every model fact with source, evidence class, freshness, account/project scope, and confidence.
  4. Identify route and projection drift caused by model-catalog changes.
  5. Produce safe mechanical swap plans without claiming equivalent quality.
  6. Feed the compatible candidate set and invalidation events to Route Intelligence (feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109), Ruflo routing, and Agentic-QE integrations.
  7. Add an operator-friendly read-only ak models command family, a concise ak status summary, and a local Dashboard presentation.
  8. Keep normal status and Dashboard reads offline and side-effect free.

Non-goals

  • Creating a universal model leaderboard.
  • Treating a newer version number as proof of higher quality.
  • Treating public API availability as proof of Claude Code, Codex subscription, or OpenCode project entitlement.
  • Probing models by spending tokens during normal discovery.
  • Automatically changing kit.json, host settings, Ruflo candidates, or Agentic-QE overrides.
  • Replacing ak host pick as the canonical route-policy mutation surface.
  • Inferring provider/vendor identity from a host or model-name substring.
  • Scraping interactive pickers when the host provides no stable machine interface.
  • Claiming a missing model was removed when a source timed out, returned a partial list, changed account scope, or served a stale cache.
  • Reimplementing Ruflo's model router or Agentic-QE's LLM router inside Agentic Kit.

Product principles

  1. Inventory before recommendation. Establish what exists and is usable before asking what is best.
  2. No collapsed state. Configured, effective, observed, discoverable, entitled, allowed, and routable remain independent.
  3. No fabricated removals. A failed or partial refresh creates a diagnostic, not a tombstone.
  4. Aliases are pointers. Store both the alias and its concrete resolution, with change history.
  5. Scope is identity. Catalog snapshots are keyed by host, provider, account/profile scope, project scope, host version, and source schema.
  6. Provenance travels with the field. A model record is not more authoritative than its source.
  7. Compatibility is not quality. A valid migration target or route plan is not an effectiveness recommendation.
  8. One policy, many projections. kit.json routing stays canonical; inventory diagnoses consumers and plans changes.
  9. Offline by default. Cached/local status is automatic; network refresh and token-consuming verification are explicit.
  10. Unknown is a first-class answer. Absence of evidence is not evidence of absence.

Core model and evidence semantics

Model identity

Use a host-scoped identity rather than a global model-name string:

{
  key: { host, provider, modelId, scopeId },
  displayName,
  aliases: [{ name, resolvesTo, provenance, observedAt }],
  variant: { reasoningEffort, serviceTier, contextWindow, modalities },
  lifecycle: { state, replacement, notice, effectiveAt },
  capabilities: { tools, vision, parallelTools, local, apiSupported },
  pricing: { basis, input, output, currency, effectiveAt, provenance },
  evidence: [{ field, source, class, capturedAt, freshness, completeness }]
}

host + provider + modelId + scopeId prevents two accounts, gateways, projects, or providers that reuse a model string from being conflated. Private deployment names and local digests may require sanitized display values while retaining a local stable key.

Independent state dimensions

Dimension Meaning Typical evidence
Configured A policy/config references the model or alias kit.json, host config, AQE/Ruflo projection
Effective Precedence and alias resolution select a concrete model host-resolved state, invocation metadata
Observed A transcript or structured worker actually used it transcript/runtime record
Discoverable A host/provider catalog lists it in this scope host command/API/cache
Entitled The current account/profile is allowed to invoke it host entitlement response or successful observed use
Policy allowed Managed/user allowlists permit selection managed settings, availableModels, project config
Routable The complete host/provider/auth/capability path is currently viable validated adapter facts; optional explicit health check
Lifecycle Active, preview, hidden, deprecated, retiring, removed, or unknown first-party catalog/migration notice
Recommended A source recommends migration or Route Intelligence recommends a swap first-party migration metadata or #109 evidence

One record can legitimately be configured but not discoverable, discoverable but not entitled, observed but hidden, or entitled but policy-blocked. The UI must display these combinations rather than reducing them to red/green availability.

Lifecycle and compatibility graph

Model “upgrade” should be represented as typed edges, not guessed semver ordering:

alias ──resolves-to──▶ concrete model
old model ──first-party-migration──▶ replacement
model A ──same-family-newer──▶ model B
model A ──mechanically-compatible──▶ model B
model A ──tier-up / tier-down──▶ model B
model A ──specialized-alternative──▶ model B
model A ──evidence-backed-equivalent──▶ model B   (#109 only)

Every edge carries provenance and scope. first-party-migration is strong lifecycle evidence but not a quality assertion. same-family-newer may be curated or vendor-published. mechanically-compatible means the route can be expressed and required features are present. Only Route Intelligence may create an evidence-backed quality/economics edge.

Opportunity levels:

  1. Newly discovered — informational; no action recommendation.
  2. Configuration drift — configured model is missing, hidden, blocked, or deprecated.
  3. Supported migration — first-party replacement target exists and the local route can express it.
  4. Mechanical swap candidate — capabilities and route syntax are compatible; quality remains unknown.
  5. Evidence-backed opportunity — feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109 shows activity-specific non-inferiority or premium value.
  6. Urgent migration — published retirement/removal is imminent or the route has already failed for lifecycle reasons.

Evidence and confidence

Suggested evidence classes, strongest first:

  1. observed successful invocation with concrete model/provider identity;
  2. host-owned entitled catalog or explicit migration metadata;
  3. host-owned discoverable catalog/cache scoped to the active profile/project;
  4. managed policy/config after precedence resolution;
  5. canonical Agentic Kit routing policy and generated projections;
  6. provider-published public catalog;
  7. locally curated catalog with verification date;
  8. inferred family/version relationship;
  9. unknown.

Negative evidence requires extra care. A model becomes removed only after an authoritative removal/retirement signal or repeated complete snapshots in the same scope. A source failure, auth failure, schema mismatch, account switch, or incomplete list yields unknown plus a diagnostic.

Discovery strategy by host

Host Configured/effective inputs Discoverable/lifecycle inputs Observed inputs Important limits
Claude Code settings precedence, model, aliases, environment overrides, model overrides, managed availableModels, gateway config host-supported aliases; gateway /v1/models only when configured; documented lifecycle data where available Claude transcripts, status/runtime metadata Alias targets can change; picker/public docs do not prove entitlement; managed policy may hide models
Codex ~/.codex/config.toml, profiles, model, model_provider, reasoning effort, invocation overrides host-owned model cache and/or stable app-server model-list protocol; cache carries visibility/capability and can carry explicit upgrade targets rollout session_meta, turn/runtime context, structured workers Cache/protocol are host-owned and versioned; public OpenAI API model list is not the Codex subscription catalog
OpenCode global/project config, provider config, agent/command model overrides opencode models [provider], verbose metadata, explicit --refresh, Models.dev/provider/config merge OpenCode SQLite/message records with provider/model/cost Catalog is provider/project scoped; connected provider and config determine picker eligibility
Local/provider routes canonical policy plus provider projection provider adapter such as Ollama catalog/digest transcript/runtime records Local name is not immutable identity; digest and endpoint matter

Adapter requirements:

  • Add host/catalog observability descriptors rather than branching on hard-coded host ids in ak models.
  • A discovery adapter returns complete, partial, stale, unavailable, or unsupported, never merely an array.
  • Capture source version/schema, host CLI version, cache freshness, and a non-identifying scope fingerprint.
  • Keep raw credentials and private provider config out of snapshots.
  • Support fixture-driven parsers so upstream schema changes fail visibly.
  • Prefer stable host-owned CLI/protocol/cache surfaces. Do not automate interactive model pickers.

Proposed architecture

 kit.json routes        host config          host catalogs/caches        transcripts / ak run
       │                    │                         │                           │
       └────────────┬───────┴──────────────┬──────────┴──────────────┬────────────┘
                    ▼                      ▼                         ▼
             binding collectors     discovery adapters       observed-use collector
                    │                      │                         │
                    └──────────────────────┴─────────────────────────┘
                                           ▼
                              normalized inventory snapshot
                         (field provenance + scope + completeness)
                                           │
                      ┌────────────────────┼─────────────────────┐
                      ▼                    ▼                     ▼
                 snapshot diff      route-impact graph     candidate projection
                      │                    │                     │
                      ▼                    ▼                     ├──▶ Ruflo router diagnostics
              `ak models diff`      `ak models plan`            ├──▶ AQE drift diagnostics
                      │                    │                     └──▶ Route Intelligence #109
                      └──────────────┬─────┴─────────────────────┘
                                     ▼
                       `ak status` + Dashboard read model

Suggested snapshot contract:

{
  schemaVersion,
  snapshotId,
  capturedAt,
  scope: { machine, project, hosts, profileFingerprints },
  sources: [{ id, status, complete, capturedAt, sourceVersion, diagnostics }],
  models: [],
  bindings: [{ consumer, activity, host, provider, configured, effective, provenance }],
  changes: [{ kind, subject, before, after, severity, provenance }],
  opportunities: [{ kind, from, to, scope, confidence, reason, evidenceRefs }],
  diagnostics: []
}

Storage rules:

  • Persist sanitized snapshots in a machine-local cache; they are rebuildable operational evidence, not canonical policy.
  • Advance the comparison baseline only after a successful, sufficiently complete refresh.
  • Keep bounded history to explain when a model or alias changed.
  • Hash or sanitize account/project scope; never compare snapshots from different scopes as one lifecycle sequence.
  • Mark Route Intelligence evidence stale when its concrete model, provider binding, host version, harness fingerprint, or relevant capabilities change.

Ruflo MetaHarness and Agentic-QE overlap

Ruflo / MetaHarness

Ruflo already exposes two adjacent surfaces:

  • ruflo providers manages providers/models/configuration and can list provider models.
  • ruflo neural router exposes the cost-optimal router lifecycle, including candidate models, prices, configuration, decisions, outcomes, savings, and bandit state.

Ruflo's ADR-149 is Proposed, not shipped-contract authority. It describes a per-model candidate registry and quality-bar routing direction grounded in measured outcomes. This issue should consume runtime-proven surfaces and report their provenance; it must not present a Proposed ADR as installed behavior.

Boundary:

  • Agentic Kit owns host-scoped inventory, configuration/projection drift, and user-facing lifecycle explanation.
  • Ruflo owns its candidate registry, learned routing decisions, prices, and outcome-learning implementation.
  • The inventory can diagnose Ruflo candidates that are no longer discoverable/routable, surface candidates absent from the host inventory, and supply validated candidates to feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109.
  • A catalog change invalidates or narrows learned recommendations; it does not retrain or silently rewrite Ruflo state.
  • Do not duplicate MetaHarness selection logic or label an Agentic Kit heuristic as MetaHarness.

Agentic-QE

Agentic-QE exposes aqe llm providers, aqe llm models, router config/health/cost, and routing economics/accuracy/metrics. Its separate aqe upgrade command detects optional native bindings, not model upgrades.

Boundary:

Relationship to Route Intelligence (#109)

This issue answers what models exist, changed, are configured, and appear mechanically usable. #109 answers which model was effective for an operation and whether a cheaper candidate delivered materially equivalent benefit.

Model Lifecycle Intelligence                         Route Intelligence (#109)
catalog + entitlement + lifecycle + compatibility ──▶ eligible candidate set
catalog/alias/capability changes                    ──▶ evidence staleness/invalidation
first-party migration target                       ──▶ experiment candidate, not quality claim
                                                     ◀── evidence-backed equivalent/premium edge

The two features should share concrete model identity, provider provenance, harness fingerprint, and freshness semantics, but retain distinct evidence grades and UI language.

Proposed ak command surface

Add one stable plural noun: ak models. The initial family is read-only except for explicit cache refresh.

ak models status

ak models status
ak models status --host codex
ak models status --json
ak models status --all

Default output is concise and uses the most recent local snapshot plus current local config/observed evidence. --all expands hidden, unconfigured, and low-confidence entries. JSON returns the versioned snapshot/read-model contract.

Recommended columns:

  • host and provider;
  • configured value and effective concrete model;
  • last observed concrete model and timestamp;
  • discovery/entitlement/policy/routability badges;
  • lifecycle state and first-party migration target;
  • source freshness/completeness; and
  • affected activity count.

ak models refresh

ak models refresh
ak models refresh --host opencode
ak models refresh --online
ak models refresh --host claude --online --json
  • Without --online, refresh local config, caches, protocols, and transcript evidence only.
  • --online explicitly permits host/provider catalog network access such as OpenCode's cache refresh or a configured gateway model endpoint.
  • Never invoke a model or consume inference tokens.
  • A later --verify mode, if needed, must be separately designed and confirmed because it may spend tokens or cross provider boundaries.
  • Failed/partial refreshes do not replace the last complete diff baseline.

ak models diff

ak models diff
ak models diff --since <snapshot-id>
ak models diff --host claude --json

Report:

  • new, removed, hidden, and visibility-changed models;
  • alias-target changes;
  • lifecycle notices and first-party migration targets;
  • capability/reasoning/context changes;
  • configured or observed bindings affected; and
  • source/scope changes that make comparison invalid.

ak models explain

ak models explain codex:gpt-5.4
ak models explain claude:sonnet
ak models explain opencode:openrouter/provider-model --json

Show the evidence chain: where the reference is configured, precedence and alias resolution, last observed use, provider/source, catalog membership, entitlement/policy state, lifecycle edge, affected routes/escalations/AQE/Ruflo consumers, and why a candidate is eligible or ineligible.

ak models plan

ak models plan --activity coder
ak models plan --from codex:gpt-5.4 --to codex:gpt-5.6-terra
ak models plan --activity reviewer --to claude:sonnet
ak models plan --json

Produce a read-only impact plan:

  • canonical kit.json route/escalation bindings affected;
  • expected Claude/Codex/OpenCode projections;
  • Agentic-QE overrides/fallbacks affected;
  • Ruflo candidate/price/learning references affected;
  • compatibility, entitlement, and provenance warnings;
  • Route Intelligence evidence that would become stale; and
  • an exact copyable ak host pick --route … command when the change is expressible.

Do not add ak models apply in the first implementation. Route mutation remains ak host pick, preserving ADR-0001/ADR-0020. A future guarded ak models swap is acceptable only if it becomes a single transactional plan/apply/verify/undo contract over the canonical policy rather than a second policy owner.

Adapt ak status

ak status remains the quick dashboard. Add one cache-only aggregate row:

models  ⚠ 9 configured · 7 observed · 2 migrations · 1 alias change · catalog refreshed 3h ago

Behavior:

  • ordinary ak status never refreshes a network catalog;
  • ak status --deep may invoke bounded local host discovery protocols but still avoids remote refresh unless an existing status policy explicitly allows it;
  • stale/unavailable sources are named, not converted into “no models”; and
  • the fix string directs users to ak models refresh, ak models diff, or ak models plan, never auto-applies a route.

Status and Dashboard experience

Preserve the existing three primary Dashboard areas. Add a Models secondary destination under Usage, and add an Overview summary card that links to it. Because ADR-0009 currently governs Usage navigation, implementation must amend its Updated note/date when this destination ships.

Overview summary card

Models
9 configured · 7 recently observed
2 supported migrations · 1 alias change
Catalog: Claude 3h · Codex 18m · OpenCode 2d ⚠

Usage → Models view

Recommended sections:

  1. Attention — migration targets, configured-but-unavailable routes, alias changes, stale/incomplete discovery.
  2. Host inventory — configured/effective/observed/discoverable/entitled/policy/lifecycle matrix.
  3. Change history — additions, removals, visibility/capability changes, and snapshot scope.
  4. Consumers — activity routes, escalation rungs, AQE overrides/fallbacks, Ruflo candidates, and evidence cohorts.
  5. Swap impact — read-only plan with compatibility facts and a copyable ak host pick command.
  6. Evidence disclosure — source, freshness, scope, completeness, and uncertainty for every claim.

Visual language:

  • Upgrade available only for a first-party migration/supersession edge.
  • New model discovered for catalog additions, without implying superiority.
  • Compatible candidate for mechanically expressible swaps.
  • Cheaper equivalent or premium justified only when feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109 provides the required evidence.
  • Not verified for this account when discoverability exists but entitlement is unknown.
  • Catalog incomplete rather than removed after source failure.

The Dashboard stays read-only, loopback-only, CSP-protected, network-silent on reads, and free of raw credentials/config secrets.

User interactions and scenarios

Scenario 1 — A Codex model receives a first-party migration target

  1. A successful local refresh observes that the configured Codex model now carries a first-party replacement target.
  2. ak status shows one model migration requiring attention.
  3. ak models diff shows the lifecycle notice, replacement, catalog freshness, and affected coder/tester routes.
  4. ak models plan --from … --to … checks reasoning-effort/capability compatibility and enumerates projections and stale Route Intelligence evidence.
  5. The user applies the copyable ak host pick --route … command explicitly.
  6. A later refresh verifies that canonical and projected bindings converge.

Scenario 2 — A Claude alias silently resolves to a different concrete model

The configured route still says sonnet, but the effective concrete target changes. The diff reports an alias-target-changed event, marks outcome cohorts learned on the prior concrete model stale, and explains that the alias remains valid. It does not label the change an upgrade unless first-party lifecycle evidence supports that language.

Scenario 3 — OpenCode exposes new provider models

The user runs ak models refresh --host opencode --online. New provider/project-eligible models appear as newly discovered. They are not automatically inserted into routes, Agentic-QE, or Ruflo. Route Intelligence may later evaluate eligible candidates.

Scenario 4 — Configured does not mean entitled

A model is present in kit.json and a public provider catalog but absent from the active host-scoped entitled list. The UI reports configured · entitlement unknown or configured · policy blocked based on evidence. It does not claim removal and does not spend tokens to test it.

Scenario 5 — Partial discovery failure

The Claude gateway times out while Codex and OpenCode discovery succeed. The new snapshot records a partial source diagnostic. The previous complete Claude baseline remains active; no Claude model is marked removed. ak status reports stale Claude discovery with the exact refresh action.

Scenario 6 — AQE and Ruflo drift from canonical routes

A stale Agentic-QE override names a retired model and Ruflo's candidate registry still contains it. ak models explain shows both consumers. ak models plan describes the canonical route change and follow-on projection/invalidation work without editing either system.

Scenario 7 — A cheaper candidate exists but quality is unknown

The inventory knows that a lower-priced compatible model is selectable. The Dashboard labels it compatible candidate · quality not evaluated. Only after #109 produces qualified evidence may the label become cheaper equivalent for a particular activity.

Scenario 8 — Multiple account or project scopes

The user changes a Codex profile or OpenCode project. Agentic Kit creates a different scope fingerprint and refuses to call catalog differences additions/removals across scopes. It can display the inventories side by side, but lifecycle history remains scoped.

Phased technical implementation plan

Phase 0 — Specification, ADR, and DDD contract

Deliverables

  • Add a new Proposed ADR for Model Lifecycle Intelligence.
  • Define ModelIdentity, ModelBinding, CatalogSnapshot, CatalogSource, LifecycleEdge, CompatibilityEdge, ModelChange, and SwapPlan in the DDD vocabulary/context map.
  • Document evidence precedence, scope identity, completeness, freshness, tombstone rules, and privacy boundaries.
  • Define the versioned JSON contract for ak models status/diff/explain/plan.
  • Record how the new ADR relies on ADR-0001, ADR-0016, ADR-0017, ADR-0020, and ADR-0021 and amends ADR-0009 only when the Dashboard view ships.

Exit gate

  • Accepted semantics for all independent state dimensions.
  • Representative fixtures for aliases, explicit migrations, partial catalogs, scope changes, local digests, and unknown entitlement.
  • No contradiction with the current ADR statuses.

Phase 1 — Binding and observed-use inventory

Likely files

  • src/lib/routing.mjs
  • src/lib/providers.mjs
  • src/lib/adapters/registries.mjs
  • transcript/usage readers
  • new pure modules under src/lib/model-inventory/

Deliverables

  • Enumerate canonical routes, escalation rungs, host config, AQE overrides/fallbacks, and Ruflo references.
  • Join last-observed host/provider/model evidence without upgrading configured/inferred provenance to observed.
  • Resolve local precedence where stable and report unresolved aliases honestly.
  • Emit ak models status --json from current local evidence; no historical diff yet.

Exit gate

  • Claude, Codex, and OpenCode fixtures produce deterministic binding/observed records.
  • Provider identity remains independent from host/model.
  • Secrets and raw private config values are absent.

Phase 2 — Capability-driven discovery adapters

Likely files

  • src/lib/adapters/registries.mjs
  • src/lib/adapters/index.mjs
  • host-specific adapter modules
  • pure parser fixtures

Deliverables

  • Extend adapter contracts for catalog discovery, lifecycle metadata, entitlement/policy evidence, source completeness, and scope fingerprints.
  • Implement local Codex cache/protocol parsing with schema/version guards.
  • Implement OpenCode models discovery and explicit online refresh adapter.
  • Implement Claude configured/policy/alias discovery and bounded gateway discovery where configured; leave unsupported entitlement unknown.
  • Preserve Ollama digest/runtime discovery through the shared contract.

Exit gate

  • Upstream schema changes fail to unsupported-schema diagnostics.
  • No interactive picker scraping.
  • No inference request or token spend.
  • Partial failures cannot manufacture removals.

Phase 3 — Snapshot store and lifecycle diff

Deliverables

  • Persist sanitized, versioned, bounded snapshots.
  • Implement field-level diff for additions, removals, visibility, alias targets, lifecycle, capability, reasoning, context, and price metadata.
  • Gate baseline advancement on completeness and stable scope.
  • Add tombstone confidence and repeated-snapshot rules.
  • Ship ak models refresh, ak models diff, and machine-readable diagnostics.

Exit gate

  • Identical snapshots produce no changes.
  • Alias target changes are detected even when the configured alias string is unchanged.
  • Account/project scope changes are not reported as mass additions/removals.
  • A failed source leaves the last complete baseline intact.

Phase 4 — Impact graph, explain, and plan

Deliverables

Exit gate

Phase 5 — ak status and Dashboard presentation

Likely files

  • src/commands/status.mjs
  • src/lib/dashboard-server.mjs
  • src/lib/dashboard/client.mjs
  • src/lib/dashboard/page.mjs
  • src/lib/dashboard/styles.mjs
  • Dashboard/status tests

Deliverables

  • Add one cache-only aggregate models row to ak status and JSON output.
  • Add the Overview model card and Usage → Models destination.
  • Render attention, inventory, history, consumers, plan, and evidence disclosure.
  • Preserve local read-only/browser security boundaries and accessibility.
  • Amend ADR-0009 and ADR-0005 update notes/dates when shipped behavior changes.

Exit gate

  • Normal status/Dashboard reads create no network traffic.
  • Every badge and change is traceable to source evidence.
  • Keyboard, responsive, and screen-reader contracts pass.
  • Dashboard cannot apply a swap.

Phase 6 — Ruflo/AQE synchronization diagnostics and #109 integration

Deliverables

  • Compare host-scoped inventory against runtime-proven Ruflo candidate/model/price surfaces.
  • Compare canonical projections against Agentic-QE providers/models/overrides/fallbacks.
  • Feed eligible candidates and lifecycle invalidations to Route Intelligence.
  • Distinguish Ruflo/AQE configured, reported, and runtime-proven states.
  • Add quality unknown, evidence stale, and evidence-backed opportunity promotion rules.

Exit gate

  • No silent rewriting of Ruflo or AQE state.
  • Proposed upstream ADRs are not presented as shipped behavior.
  • A model lifecycle change reliably invalidates affected recommendations without deleting their audit history.

Phase 7 — Optional transactional swap workflow

This phase is deliberately deferred.

Evaluate a guarded ak models swap only after planning and projection convergence are proven. It must:

  • modify only canonical routing policy;
  • preview every affected projection and evidence invalidation;
  • require explicit confirmation;
  • reuse normal sync/projection machinery;
  • verify convergence; and
  • support a bounded undo record.

If those guarantees cannot be met, retain ak host pick as the only mutation path.

Phase 8 — Documentation, QE, and closure proof

Deliverables

  • Update ADRs, DDD, CLI reference, routing, provider, transcript, Dashboard, and troubleshooting docs.
  • Exercise parser/diff/plan fixtures and property tests through Agentic-QE where appropriate.
  • Run the repository's full pnpm run check plus security/privacy/network-silence proofs.
  • Link exact implementation PR, exact-head CI, QE evidence, and release proof before closure.

Exit gate

  • ADR statuses and update notes describe shipped code exactly.
  • All model claims in docs/UI are backed by a named source and evidence class.
  • Full project and Agentic-QE gates pass.

Security privacy and network constraints

  1. Ordinary ak status, ak models status, and Dashboard reads are offline and cache-only.
  2. ak models refresh --online is the explicit network boundary and names which hosts/providers will be contacted.
  3. Discovery never invokes a model, sends a prompt, or consumes inference tokens.
  4. A future availability verification mode must require separate confirmation and disclose provider, cost/capacity, and evidence sent.
  5. Credentials, auth tokens, raw provider configuration, prompt content, and reasoning traces never enter snapshots or Dashboard payloads.
  6. Account/profile/project identity is represented by a non-identifying scope fingerprint.
  7. Private gateway deployment ids and local endpoints are sanitized for normal display; exact local CLI disclosure should be deliberate.
  8. Host-owned cache/protocol data is treated as untrusted input with size, timeout, schema, and enum validation.
  9. Command execution uses literal argv, bounded timeouts, and no shell interpolation.
  10. Snapshot retention, rotation, deletion, and scope invalidation are documented and tested.
  11. Dashboard aggregate routes remain free of raw transcript/config bodies and keep existing token/CSP/origin protections.

Risks and mitigations

Risk Consequence Mitigation
Public catalog confused with entitlement Invalid “you can upgrade” claim Separate discoverable, entitled, policy-allowed, and routable dimensions
Alias changes invisibly Learned evidence and operator expectations attach to wrong concrete model Persist alias and resolution; diff target; invalidate affected evidence
Partial source looks like removal Mass false alarms Completeness status, stable scope, baseline gating, cautious tombstones
Upstream private cache/protocol changes Parser silently corrupts inventory Adapter schema/version guards, fixtures, unsupported-schema diagnostics
Account/project changes Entire catalog appears added/removed Scope fingerprints; refuse cross-scope lifecycle diff
Newer mistaken for better Unsafe routing change Typed lifecycle vs compatibility vs evidence-backed edges; #109 owns quality
Route mutation has two owners Projection divergence Read-only ak models; retain ak host pick as canonical mutation
Ruflo/AQE intent presented as host truth Invalid availability/provider claims Preserve source and runtime-proof levels; host adapter is entitlement authority
Model discovery spends tokens or leaks code Cost/privacy surprise Catalog-only discovery; explicit separate verification design
Private deployment names leak on screen/API Operational disclosure Sanitized display ids and bounded local-only exact disclosure
Stale learned recommendations survive model changes Bad “cheaper equivalent” claim Concrete identity/harness cohorts and lifecycle invalidation events
Static catalog and live inventory disagree Confusing help/validation Make static catalog a dated fallback/seed; prefer scoped live evidence with provenance

Acceptance criteria

Domain and evidence integrity

  • Configured, effective, observed, discoverable, entitled, policy-allowed, routable, lifecycle, and recommended are distinct fields.
  • Model identity includes host, provider, concrete model id, scope, and relevant variant/digest data.
  • Every model fact and edge carries source, evidence class, freshness, and completeness.
  • Alias values and concrete resolutions are both retained and diffed.
  • Lifecycle migration, mechanical compatibility, and evidence-backed equivalence cannot be conflated.
  • Unknown/partial evidence never becomes removal or unavailability.

Host discovery

  • Claude, Codex, OpenCode, and local-provider fixtures normalize deterministically.
  • Codex lifecycle/upgrade metadata is parsed behind a versioned adapter contract.
  • OpenCode catalog refresh is explicit and project/provider scoped.
  • Claude aliases, overrides, allowlists, and gateway discovery preserve precedence and entitlement uncertainty.
  • Interactive picker scraping and inference-token probes are absent.
  • Account/profile/project scope changes cannot generate mass false diffs.

Command surface

  • ak models status, refresh, diff, explain, and plan have stable human and JSON contracts.
  • Default commands use local evidence and make no network request.
  • refresh --online clearly names and bounds external contacts.
  • plan enumerates affected canonical routes, projections, AQE/Ruflo consumers, and evidence invalidation.
  • Initial implementation has no ak models apply and does not mutate routing.
  • Copyable actions use the canonical ak host pick --route … surface.

Diff and lifecycle correctness

  • Complete same-scope snapshots detect add/remove/visibility/alias/lifecycle/capability changes.
  • Failed, partial, stale, or schema-invalid sources retain the last complete baseline.
  • First-party migration targets are labeled supported migrations, not quality upgrades.
  • Removal requires authoritative evidence or the documented repeated-complete-snapshot threshold.
  • Snapshot history is bounded, auditable, and contains no credentials or raw prompts.

Ruflo AQE and Route Intelligence boundaries

Status Dashboard and security

  • ak status adds one truthful cache-only models row with freshness and actionable diagnostics.
  • The Dashboard preserves three primary areas and adds Models under Usage.
  • Every badge and count has evidence disclosure.
  • Dashboard remains read-only, loopback-only, CSP/origin/token protected, and network-silent on reads.
  • No secret, raw provider config, private transcript content, or unsanitized deployment id reaches aggregate APIs.
  • Keyboard, responsive, and screen-reader behavior passes.

Documentation and closure

  • A new ADR is accepted before lifecycle recommendations ship and marked Implemented only with proof.
  • ADR-0009/0005 are updated when the Dashboard navigation/presentation changes.
  • ADR-0016/0017/0021 remain accurate or are amended in the same change that alters their contracts.
  • CLI, routing, provider, transcript, Dashboard, DDD, and troubleshooting docs are updated.
  • Full pnpm run check, Agentic-QE gates, network-silence, privacy, and exact-head release evidence are linked before closure.

Success measures

  • percentage of configured bindings with concrete/effective identity;
  • percentage with fresh host-scoped discovery evidence;
  • percentage with proven entitlement versus unknown entitlement;
  • alias-target and lifecycle changes detected before a route failure;
  • configured deprecated/unavailable routes identified;
  • mean time from first-party migration notice to explicit route convergence;
  • number of false removals caused by partial/scope-changing discovery (target: zero);
  • number of AQE/Ruflo consumer drifts explained;
  • percentage of feat: add evidence-backed Route Intelligence for model effectiveness and cheaper-equivalent routing #109 recommendations carrying current inventory compatibility/freshness;
  • model findings withheld because entitlement, lifecycle, or quality evidence is insufficient; and
  • operator use of diff, explain, and plan before route changes.

Withheld findings are a trust measure: a system that never says “entitlement unknown” or “quality not evaluated” is overstating what it knows.

Open product decisions

  1. Should the command noun be permanently plural (ak models) or singular (ak model)? Plural is recommended for an inventory.
  2. What snapshot retention window balances change history and local footprint?
  3. Which host surfaces are stable enough to support by contract versus best-effort adapters?
  4. What repeated-complete-snapshot threshold permits a non-authoritative removal tombstone?
  5. Should ak status --deep ever perform remote catalog refresh, or should all network remain exclusive to ak models refresh --online? The latter is recommended.
  6. How should private gateway deployment names be sanitized while retaining stable local identity?
  7. Which capability fields are required before a swap is called mechanically compatible?
  8. Should reasoning effort be part of model identity, a route binding, or both?
  9. When an alias target changes, should Route Intelligence evidence expire immediately or remain visible as stale historical evidence? Visible-but-stale is recommended.
  10. Should a future transactional ak models swap exist, or is plan plus ak host pick the permanent clean separation?
  11. Should Usage → Models ship as a new secondary destination or initially as an expanded Findings disclosure? A dedicated secondary destination is recommended because inventory/change history is richer than a finding card.
  12. Which upstream lifecycle notices may be displayed verbatim versus normalized/summarized for licensing and UX consistency?

Related work and source grounding

Agentic Kit

Host-owned model sources

  • Claude Code model configuration — configuration precedence, aliases, allowlists, overrides, and gateway discovery.
  • OpenCode models — provider-qualified model selection and project/provider catalog behavior.
  • Local codex --help, codex app-server --help, and the versioned host-owned Codex model cache — model selection and machine-readable catalog/lifecycle evidence. Implementation should replace local research notes with a stable upstream protocol/source citation where available.

Ruflo / MetaHarness / Agentic-QE grounding

  • ruflo/v3/docs/adr/ADR-149-per-model-cost-optimal-routing.md — Proposed per-model candidate registry and cost-optimal routing design; useful direction, not proof of shipped behavior.
  • agentic-qe/src/integrations/agentic-flow/model-router/router.ts — implemented multi-model router orchestration boundary for complexity, budget, metrics, and decisions.
  • Runtime CLI interfaces verified during this proposal:
    • ruflo providers exposes provider/model management.
    • ruflo neural router exposes candidate models, prices, configuration, decisions, outcomes, and savings surfaces.
    • aqe llm exposes provider/model/router management.
    • aqe routing exposes economics, accuracy, and per-agent metrics.
    • aqe upgrade is a read-only optional-native recommendation command and is not a model-upgrade command.

If a required upstream contract is unavailable or unstable, implementation must surface that limitation and keep the state unknown rather than silently replacing it with an Agentic Kit guess.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions