How to setup local http api_endpoint

[kevint324]

Hi,

I've hosted a local api gateway. It works fine and can be tested with below code.

def gateway(prompt_to_send, model):
    url = 'http://172.16.32.20:8089/api/paas/v4/chat/completions'

    headers = {
        'Authorization': 'Bearer my_key',
        'Content-Type': 'application/json'
    }

    data = {
        "model": model,
        "messages": [
            {
                "role": "user",
                "content": prompt_to_send,
                "temperature" : "0.0"
            }
        ]
    }

    response = requests.post(url, headers=headers, data=json.dumps(data))
    result = json.loads(response.content.decode('utf-8'))
    content = result['choices'][0]['message']['content']
    return content

I've set up tabby's ~/.tabby/config.toml as below

[model.completion.http]
kind = "openai/completion"
model_name = "glm-4"
api_endpoint = "http://172.16.32.20:8089/api/pass/v4/chat/completions"
api_key = "my_key"


[model.chat.http]
kind = "openai/chat"
model_name = "glm-4"
api_endpoint = "http://172.16.32.20:8089/api/pass/v4/chat/completions"
api_key = "my_key"

Now the problem is chat and generation stops working.
There is no response.

Is there anything I missed. Really appreciate your help.

Tabby Version 0.15.0

[wsxiaoys]

OpenAI SDK requires setting the base URL in a format similar to https://api.openai.com/v1. Therefore, in your case, it should be http://172.16.32.20:8089/api/pass/v4.

Besides - chat completion model endpoint is not really usable with model.completion.http. See https://tabby.tabbyml.com/docs/references/models-http-api/openai/ for details.

[kevint324]

Hi

Thanks for the reply.

I've tried
api_endpoint = "http://172.16.32.20:8089/api/pass/v4"

but still got no luck. There is no output.

Can I turn on some debug log to find out what's going on?

[wapleeeeee]

Is the typo "pass" cause the error? I've noticed the following successful ways all use "paas" at endpoint url.

[kevint324]

hi @wsxiaoys

Could you point a direction about how to debug this issue?
I'm stuck here trying to hook tabby to my local deployed LLM.

Thanks a lot!
Kevin

[kevint324]

I also test it with OpenAI api which works just fine.

from openai import OpenAI 

client = OpenAI(
    api_key="<my_api_key>",
    base_url="https://open.bigmodel.cn/api/paas/v4/"
) 

completion = client.chat.completions.create(
    model="glm-4",  
    messages=[    
        {"role": "user", "content": "Hello."} 
    ],
    top_p=0.7,
    temperature=0.9
) 
 
print(completion.choices[0].message)

Output $ python test_zp.py

ChatCompletionMessage(content="Hello 👋! I'm ZhiPuQingYan（智谱清言）, you can call me Xiaozhi🤖, the artificial intelligence assistant, nice to meet you. Feel free to ask me any questions.", role='assistant', function_call=None, tool_calls=None)

~/.tabby/config.toml

[model.completion.http]
kind = "openai/completion"
model_name = "glm-4"
api_endpoint = "https://open.bigmodel.cn/api/paas/v4"
api_key = "<my_api_key>"

[model.chat.http]
kind = "openai/chat"
model_name = "glm-4"
api_endpoint = "https://open.bigmodel.cn/api/paas/v4"
api_key = "<my_api_key>"

But I cannot get any response from tabby chatbox.

[cbenitez81]

i need to test this but i think that the v1 should come after your base url. Check this out https://platform.openai.com/docs/api-reference/making-requests , e.g., https://api.openai.com/v1/chat/completions

[kevint324]

I switched to twinny and use ollama mode and this kind of url http://my_gateway_ip_address/api/paas/v4/chat/completions works just fine.

I thinks it'd be better that tabby provide some sort of debugging method say curl some http address to test if the service is available.

[b4ushankar]

Hi Kevin,

I am also facing the same issue. Thanks for your idea. I tried changing like below as suggested. Tabby failed at the startup.

Completion model

[model.completion.http]
kind = "ollama/completion"
model_name = "deepseek"
api_endpoint = "http://my_gateway_ip_address/openai/deployments/deepseek"
api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Chat model

[model.chat.http]
kind = "openai/chat"
model_name = "deepseek"
api_endpoint = "http://my_gateway_ip_address/openai/deployments/deepseek/chat"
api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

Embedding model

[model.embedding.http]
kind = "openai/embedding"
model_name = "text-embedding-3-small"
api_endpoint = "http://my_gateway_ip_address/openai/deployments/text-embedding-ada-002/embeddings"
api_key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

can you please help to provide your configuration for connecting our local url.