Skip to content

Fireworks default/example model reference (kimi-k2-instruct) doesn't exist in models.dev, silently skipped by CI #4132

Description

@aheritier

What's wrong

pkg/config/auto.go's DefaultModels["fireworks"] and examples/fireworks.yaml both reference:

accounts/fireworks/models/kimi-k2-instruct

This model ID does not exist in the fireworks-ai catalog on models.dev — and, as far as I can tell from both the current and previous embedded pkg/modelsdev/snapshot.json, it never has. models.dev's naming for this lineup uses a different scheme entirely (e.g. accounts/fireworks/models/kimi-k2p6, .../kimi-k2p7-code, .../kimi-k3).

Also referenced (with the same staleness) in:

  • pkg/config/auto_test.go (two assert.Equal checks mirroring the value in auto.go, not validated independently)
  • docs/providers/fireworks/index.md (model: snippets ×4, plus a stale "Available Models" table listing accounts/fireworks/models/kimi-k2-instruct, .../llama-v3p3-70b-instruct, .../qwen3-235b-a22b — none of which exist in the current fireworks-ai catalog; Llama has no successor there at all anymore)

Why it's invisible to CI

pkg/config/examples_test.go maintains a modelsDevAbsentProviders map that skips models.dev catalog validation entirely for several providers, including fireworks, with the comment:

"fireworks": true, // models.dev catalogs Fireworks under the "fireworks-ai" id, not "fireworks"

This is accurate — but the consequence is that no test anywhere ever checks that the model string configured for the fireworks provider actually exists, so this reference has silently drifted (or was simply wrong from the start) without any CI signal. The same blind spot applies to the other skipped providers (together, moonshot, chatgpt, ovhcloud, vercel, opencode-zen, dmr, cloudflare-workers-ai, cloudflare-ai-gateway), though this issue is specifically about the concrete Fireworks breakage found.

Suggested fix direction

  • Update the stale references to a model that currently exists in the fireworks-ai catalog (e.g. accounts/fireworks/models/kimi-k3) — tracked separately as part of the broader model-reference refresh for PR chore: refresh models.dev snapshot (+79 -75 ~196) #4121.
  • Longer-term: teach examples_test.go (or a dedicated test) to resolve the actual models.dev catalog id for these "absent" providers (e.g. fireworksfireworks-ai, togethertogetherai, moonshotmoonshotai) instead of skipping validation outright, so this class of staleness gets caught automatically going forward.

References

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

area/configFor configuration parsing, YAML, environment variablesarea/docsDocumentation changesarea/modelsLLM model integrations and model providersarea/providersFor features/issues/fixes related to LLM providers (Bedrock, LiteLLM, Qwen, custom, etc.)area/testingTest infrastructure, CI/CD, test runners, evaluation

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions