Allow multiple providers to be used simultaniously

[tarasglek]

At the moment to switch providers we have to go to preferences and swap out provider + key. Would like to just have a set of providers, with keys and switch between them and their models as easily as we do within one provider.

Currently openrouter has some really unique models eg perplexity model that can query the internet:

Useless openai response:

However there are some downsides of accessing gpt through openrouter(robustness, speed, privacy, etc). Also in the future we might want to use dedicated openai within Azure. For all these reasons we would like to be able to list models across multiple providers and switch between then when doing retry-with...

[humphd]

@kliu57 I don't know if you'd find this interesting to work on, but it dovetails with other changes you've done in the past. If you're interested, I could talk you through some of the aspects you'd need to consider when solving this.

[kliu57]

@humphd Yes I could work on this

[kliu57]

@humphd @tarasglek following today's meeting, this is my proposed implementation for storing multiple api keys of different providers

[kliu57]

@humphd @tarasglek I have a question about the remove button

[humphd]

Let me suggest some ideas:

1. By default, we would expect a providers key to exist on the settings object. This would be an Array of: `{ id: "unique-provider-id", name: "Provider Name", apiUrl: "https://provider/api", apiKey: "secret..." }.
2. On startup, if settings has apiUrl and apiKey, we know we are using an old version, and we migrate them to the new format: create providers and add an item for either OpenAI or OpenRouter.ai (it's one or the other), delete apiUrl and `apiKey.
3. We create a new file in /src/lib named ChatCraftProvider, and it would be a class that represents the info above. Look at how https://github.com/tarasglek/chatcraft.org/blob/main/src/lib/ChatCraftModel.ts works for a good example. Internally, our code can work with instances of ChatCraftProvider (e.g., provider.apiUrl or provider.apiKey).
4. We can write a context and hook for useAiProviders() or something, which returns the providers array anywhere in our code (e.g., const providers = useAiProviders()). Take a look at how https://github.com/tarasglek/chatcraft.org/blob/main/src/hooks/use-models.tsx works, and how we use it

   chatcraft.org/src/main.tsx

   Line 18 in 34fbb43

   <ModelsProvider>

   .
5. The UI in the settings would change to iterate over the list of providers, and show the various fields for each (i.e., show the Name, URL, API Key for each).
6. The "Remove" button would become a button to remove a provider from the list vs. the key itself.

What do you think about this?

I realize that this expands the scope of this issue, so feel free to break it up into smaller pieces that happen over time (e.g., add ChatCraftProvider type, create context/hook, do settings UI, do migration, etc).

[tarasglek]

Note providers should not be a list. it should be associative array of {name1: {url:, key:}, name2}.. where names would be openrouter, openai for now. Arrays will get us into trouble when we want to support more providers. Agree re wiping.

[humphd]

@kliu57 don't be afraid to break this into smaller pieces--we only need some of it to happen by Friday. For example, you could land the code that supports the providers in settings, but not use it yet (e.g., we continue to use the old settings for now).

[kliu57]

@humphd Hi Dave, I've broken this up like you suggested. Here is the first part of this issue. #383