[Feature]: Too few models are supported

[2catycm]

The Feature

Many famous LLM model are not included, while other similar github repo like https://github.com/songquanpeng/one-api and https://github.com/llmapi-io/llmapi-server supports them.

For example, this repo don't include:

• Baidu ERINE （文心一言）
• Tsinghua&ZhiPu ChatGLM_turbo （智谱清言）
• Ali QWEN （通义千问）

and these models are of high interest to me.

Motivation, pitch

This Repo seems elegant and simple, which is good compared to others. But is doesn't support some models that are famous in some community.

Twitter / LinkedIn details

No response

[2catycm]

Oh, I see gemini and claude in the doc now, my bad. Edited.

[ishaan-jaff]

what doc did you see that made it seem we don't support enough providers @2catycm ?

[krrishdholakia]

Hi @2catycm, please feel free to add any specific models / providers you'd like support for, in this thread - #1294

Closing this issue in favor of tracking it on that thread.

Similar to @ishaan-jaff, curious what made you feel we didn't support a lot of providers?

[yuanjiexiong]

@krrishdholakia
I also need qwen、 ChatGLM .
qwen、 ChatGLM is most popular in China and have opensource and api version(Because gpt is illegal in china and need to use vpn to visit)！！！！
But I don't know whether api can visit or not in other country.

[yuanjiexiong]

what made you feel we didn't support a lot of providers?
@krrishdholakia
You can see, the 1st is illegal in China and 2nd 3rd 4th are missing.

Because most chinese people can't not use other country model(don't have vpn), so we need 2nd 3rd 4th.

[Undertone0809]

There are relevant docs:

• Baidu ERINE: https://cloud.baidu.com/doc/WENXINWORKSHOP/s/clntwmv7t
• ZhiPu AI ChatGLM: https://open.bigmodel.cn/dev/howuse/glm4
• Ali QWEN: https://help.aliyun.com/zh/dashscope/developer-reference/model-introduction?spm=a2c4g.11186623.0.i2

[2catycm]

what doc did you see that made it seem we don't support enough providers @2catycm ?

Yes, some models are illegal in China because the data is stored in server outside China (for which the law maker might think it dangerous), and the algorithm is not put on records. Just like why Tiktok is now becoming illegal in America. With this context, a lack of support for legal LLMs in China would seem "too few" for some users.

[2catycm]

@krrishdholakia I also need qwen、 ChatGLM . qwen、 ChatGLM is most popular in China and have opensource and api version(Because gpt is illegal in china and need to use vpn to visit)！！！！ But I don't know whether api can visit or not in other country.

So maybe for the same reason these models on the list might not be legal in America or other countries, not sure about that

[2catycm]

@krrishdholakia I also need qwen、 ChatGLM . qwen、 ChatGLM is most popular in China and have opensource and api version(Because gpt is illegal in china and need to use vpn to visit)！！！！ But I don't know whether api can visit or not in other country.

But open source models would definitely not be a problem for law.
And this litellm library supports huggingface, which means most open source transformers are supported.

[dl942702882]

hope for support for Ali QWen。

[NeverOccurs]

It seems Kimi and ChatGLM are now using similar format as openAI, which means with just a little work you can use them via LiteLLM. I have not tested Qwen but it seems plausible.

[dl942702882]

It seems Kimi and ChatGLM are now using similar format as openAI, which means with just a little work you can use them via LiteLLM. I have not tested Qwen but it seems plausible.

qwen request format is quite different；

[NeverOccurs]

It seems Kimi and ChatGLM are now using similar format as openAI, which means with just a little work you can use them via LiteLLM. I have not tested Qwen but it seems plausible.

qwen request format is quite different；

Yeah After taking some look into it I noticed that too. Seems dashscope is a must. Will look into it but normally glm and Kimi work smoothly.

[krrishdholakia]

Hey @dl942702882 @NeverOccurs why use litellm to proxy openai-compatible models?

[NeverOccurs]

Hey @dl942702882 @NeverOccurs why use litellm to proxy openai-compatible models?

At first I used it to proxy ollama in order to use function calling. Then I tried to use it proxy some OpenAI compatible LLMs to see if I can achieve easier management of LLMs. Mostly it is because I'm new to this so just trying around.
Speaking of that there is indeed a question. I use outside app to call Litellm endpoint but I cannot set temperature in that app and it by default send requests with temperature 0.0, which is not acceptable for some LLMs. Is there a way to fix a temperature parameter in LiteLLM setting ignoring the parameter passed to it?

[dl942702882]

Hey @dl942702882 @NeverOccurs why use litellm to proxy openai-compatible models?

cause qwen is not openai-compatible, the chat request format is different. see here qwen api doc
litellm is a very good project。
actually, i am looking for litellm can custom model integration , that i can integrate any models that i need。

[zryf2000]

Langchain already has corresponding examples and hopes to access them as soon as possible. Here are some links I hope you find helpful.
https://python.langchain.com/docs/integrations/chat/tongyi/
https://python.langchain.com/docs/integrations/chat/baidu_qianfan_endpoint/
https://python.langchain.com/docs/integrations/chat/zhipuai/