Skip to content

feat: reuse warmed model store, add picker timing, and fix discovery error caching#3285

Merged
Sayt-0 merged 3 commits into
docker:mainfrom
dgageot:worktree-board-598065d1432da6d4
Jun 27, 2026
Merged

feat: reuse warmed model store, add picker timing, and fix discovery error caching#3285
Sayt-0 merged 3 commits into
docker:mainfrom
dgageot:worktree-board-598065d1432da6d4

Conversation

@dgageot

@dgageot dgageot commented Jun 27, 2026

Copy link
Copy Markdown
Member

The model picker pipeline was doing redundant work: each runtime session, server session, and embedded-chat path independently warmed its own models.dev store, even though a single warm copy is sufficient for the lifetime of the process. On top of that, context-cancelled discovery errors were being cached, causing subsequent requests to fail silently, and the picker had no way to understand why discovery was slow or where time was being spent.

This change wires a single shared ModelStore through the CLI entry point (cmd/root/run.go), the app layer, and both the runtime and server session constructors, so the expensive initial fetch happens once and the result is reused everywhere. The model picker now also accepts custom provider references directly, lifting a restriction that forced all providers through the standard discovery path. Timing instrumentation has been added throughout the picker pipeline — from initial store lookup through to final model selection — so latency spikes are visible in debug logs. Finally, context-cancellation errors are no longer cached in the discovery layer; a cancelled lookup leaves the cache empty so the next request retries cleanly.

Validated with task --force lint and task --force test.

dgageot added 3 commits June 27, 2026 11:41
Log durations at each stage of model discovery, filtering, and rendering
to help diagnose model picker latency.

Assisted-By: claude-opus-4-5
@dgageot dgageot requested a review from a team as a code owner June 27, 2026 14:07
@aheritier aheritier added area/cli CLI commands, flags, output formatting area/models LLM model integrations and model providers area/providers/docker-model-runner Docker Model Runner (DMR) local inference area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/tui For features/issues/fixes related to the TUI kind/feat PR adds a new feature (maps to feat:). Use on PRs only. labels Jun 27, 2026
@Sayt-0 Sayt-0 merged commit aa1dc17 into docker:main Jun 27, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli CLI commands, flags, output formatting area/models LLM model integrations and model providers area/providers/docker-model-runner Docker Model Runner (DMR) local inference area/runtime Runtime engine, agent loop execution, tool dispatch, loop detection area/tui For features/issues/fixes related to the TUI kind/feat PR adds a new feature (maps to feat:). Use on PRs only.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants