Skip to content

test: make models-list tests hermetic and parallelizable#3388

Merged
Sayt-0 merged 1 commit into
docker:mainfrom
dgageot:worktree-board-a6ce8a6c6df805e0
Jul 1, 2026
Merged

test: make models-list tests hermetic and parallelizable#3388
Sayt-0 merged 1 commit into
docker:mainfrom
dgageot:worktree-board-a6ce8a6c6df805e0

Conversation

@dgageot

@dgageot dgageot commented Jul 1, 2026

Copy link
Copy Markdown
Member

What

Make the TestModelsListCommand_* tests hermetic and parallelizable.

Why

Each of the five TestModelsListCommand_* tests ran newModelsCmd().Execute()
against the real environment, which made them slow (~2–3s each, ~12–16s for
the whole cmd/root package):

  1. config.AvailableProviders probes ~19 providers × several env vars. For every
    missing API key, the default env-provider chain shells out to the macOS
    security keychain (plus pass/op/Docker-Desktop socket). Measured at
    ~1.48s for 20 lookups.
  2. The real models.dev store fetches https://models.dev/api.json (30s timeout)
    on a cold cache — a network dependency in a unit test.
  3. They used t.Setenv + paths.SetConfigDir, so they could not run in parallel.

How

  • Add a small modelsCmdOption variadic seam to newModelsCmd / newModelsListCmd.
    Production calls newModelsCmd() with no args, so behavior is unchanged.
    Tests inject a hermetic env provider (environment.NewMapEnvProvider) and an
    in-memory models.dev store (modelsdev.NewDatabaseStore) via the existing
    RuntimeConfig.EnvProviderForTests / RuntimeConfig.ModelsDevStoreOverride
    fields.
  • Rewrite models_test.go: drop t.Setenv / paths.SetConfigDir, add
    t.Parallel() to every test, and add a tiny in-memory testCatalog() database.
  • The catalog includes a catalog-only model so the tests actually verify the
    injected store is read (not just the per-provider defaults); the default-marker
    test now asserts exactly one default row matching the auto-selected model; and
    the canceled-context test asserts the server is never reached.

Result

  • Each models test: ~2–3s → 0.00s
  • cmd/root package: ~12–16s → ~1.7s (~3.7s with -race)
  • Validation: golangci-lint 0 issues, go test -race ./cmd/root/ green,
    full suite passes with no regressions.

Inject an in-memory env provider and a stub models.dev store via a new
modelsCmdOption seam, replacing t.Setenv + paths.SetConfigDir hacks.
@dgageot dgageot requested a review from a team as a code owner July 1, 2026 16:35
@aheritier aheritier added area/cli CLI commands, flags, output formatting kind/test Test-only changes labels Jul 1, 2026
@Sayt-0 Sayt-0 merged commit 2ad82da into docker:main Jul 1, 2026
9 of 10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli CLI commands, flags, output formatting kind/test Test-only changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants