Autodiscover models for custom providers

[nikaro]

It would be nice to have an autodiscovery of models through the /models API endpoint that most providers seem to support.

Something that could be configured like that:

{
  "$schema": "https://charm.land/crush.json",
  "providers": {
    "litellm": {
      "type": "openai",
      "base_url": "http://localhost:4000",
      "name": "LiteLLM",
      "autodiscover_models": true
    }
}

Then when launched it would query {base_url}/models to get the list of models.

[meowgorithm]

Good call. This would be awesome.

[yuseferi]

Sounds interesting, So, what is required to be implemented? Get list of the models and update the config provider?

[caarlos0]

I guess we need to discuss in more detail how this could work.

I'm assuming you mean only for local LLMs (like ollama and whatnot)? How do you envision this should work?

[nikaro]

Not necessarily only for local, but probably for any custom endpoint that is OpenAI/Anthropic//models-compatible. Actually i had LiteLLM in mind when opening this discussion. I guess Ollama works the same 🤷

I've seen that Crush has a update-providers command, i've not looked at the code but i guess it updates the ~/.local/share/crush/providers.json file. So it would be something like that:

• read config
• if provider type: openai or anthropic (don't know for others) is detected
• if new settings autodiscover_models: is true (maybe: OR if models: array is not defined)
• fetch {base_url}/models and update ~/.local/share/crush/providers.json with provider and its exposed models

[smlx]

LiteLLM exposes the OpenAI /v1/models endpoint which would be very convenient to have support for in crush!

https://docs.litellm.ai/docs/proxy/model_discovery
https://developers.openai.com/api/reference/resources/models/methods/list

[klarkc]

I'm looking for any provider that runs locally and has autodiscovery enabled

[yuseferi]

I've created this PR to what we needed for a while : #3019. it make litellm models automatically loaded to Crush.
So please review 🙏