Specifying the model to use for a custom server reached via the OpenAI backend

[jorge-menjivar]

@haydonryan as a follow up to #343 , if you have a custom STT server that speaks the OpenAI API like Open STT Server , but serves custom models other than those listed by the OpenAI backend, how do you specify which model to use?

For example, in order to use the whisper-tiny model from the Open STT Server, which is not listed in the OpenAI backend backend.toml file, I added a new option to the OpenAI backend which accepts any string. The value of this option overwrites the model chosen through the app UI.

Did you find a better approach for this?

[haydonryan]

Honestly at the moment I'm not using my backend - I'm just using parascribe:latest with the PARASCRIBE_MODEL_ID=nemo-parakeet-tdt-0.6b-v2.

This is mainly because I need the gpu vram so I'm not running the larger model. I'm also trying to implement matrix voice call STT -> my own claw derative -> TTS, so needed a model and backend that supports streaming to reduce latency.

I have a new gpu coming tuesday though, so I'll be experimenting more with that (but I need both on off and streaming support so will try voxtral realtime). That said, I'm happy with just the server supporting a single model. The reality is model swapping via docker compose is good enough for me. Especially as the stacks that some of these models use that i'm swapping between are quite different.

The problem with supporting multiple models defined by the client is that it could take a while to load the model, and most people are vram poor so having multiple models loaded at once (unless it's a CPU based model) is less likely.

Edit: I'd be happy to just support the openai protocol for model definition. The issue i do find however (with opencode for example) is you end up having to chase model names across clients / servers.
For example. I changed my llm model from qwen 3.6 27b to deepseek v4 flash, changed the model id on vllm, then had to go and update all my clients to use that model name which was a pain. I could see a server side feature to just serve the default if it doesn't match though.

[jorge-menjivar]

So for now, specifying the custom model name via the backend string option should be enough, do you agree? Can you also elaborate on the STT -> claw -> TTS implementation you are trying to achieve? I am curious about what the final flow will look like and what are the requirements for the STT to work well.

[haydonryan]

Yep specify in the backend is perfect for now i think.

So matrix chat has many well supported apps on android and iPhone - combined the local matrix server with a turn server it allows you to make calls to other users.

I've built out an openclaw clone , and bolted on matrix call support (elementx compatible). Currently I can call it from elementX and have a converstaion. The call goes to a turn server (RTC), then to fastclaw (my clone) which uses stt (streaming preferable so it reduces the time after speaking) . This hits my local llm (deepseek v4 flash) , which will stream into a TTS server, stream back to RTC, and the device. Streaming wherever possible reduces latency / silence between conversations.

Currently I have it running with no tools in the system prompt, and unlike more advanced conversational speech (like openai's chat), you have to take turns.

There's a lof of complexity when you enable "interrupting" when the llm is speaking, as you have to elimiate any echo it hears, and validate that the interruption is a real interruption not just saying "mmhm" or a loud noise. That's what i'm working on now, but ultimately i'd like to "call" my self hosted agent, give it stuff to do, and have it do that.

eg: "i've found a bug in openstt server, can you add a story to the backlog to fix X".

With VPN i can run this from anywhere too. I hate the idea of centralized STT / TTS. Whisperflow just had 4Tb of leaked conversations happen.

[haydonryan]

So for now, specifying the custom model name via the backend string option should be enough, do you agree? Can you also elaborate on the STT -> claw -> TTS implementation you are trying to achieve? I am curious about what the final flow will look like and what are the requirements for the STT to work well.

I now i'm not so sure. I just deployed voxtral realtime with vllm, and it fails with a 404 on the model name, so unfortunately I think we will need to expose that option after all.

I did update it to use --served-model-name whisper-1 but yeah might need a config option for people who have multiple models on an endpoint.

[jorge-menjivar]

If I am not mistaken, vllm does not support serving multiple models under the same port. Were you running v0.1.0 of the backend without the 'Custom model name' option?

When I tested vllm using the openai v0.1.1 backend, both of the following configurations worked correctly. One thing I would like to point out is that the "other" model needs to be active when using a custom model.

with uv run vllm serve --gpu-memory-utilization 0.7 --model="Qwen/Qwen3-ASR-1.7B"

with uv run vllm serve --gpu-memory-utilization 0.7 --model="Qwen/Qwen3-ASR-1.7B" --served-model-name="qwen"

[haydonryan]

Yes I was! Thanks for pointing that out! Glad that got added.

Yes only one model per endpoint - need a router to do anything else on top.

[haydonryan]

I also realized I wasn't tracking Open-stt-server in my internal tracking tool for agents, so have added that back in!

[haydonryan]

@jorge-menjivar another related thought. I've been trying voxtral-realtime, which seems pretty good. Have you thought at all how you'd like to implement streaming wasm backends? (using the openai compatible /v1/transcribe/realtime)

[jorge-menjivar]

The mistral backend has a realtime implementation for voxtral-mini-transcribe-realtime-2602 via WebSockets. The openai protocol uses WebSockets as well, so a similar implementation should fit right in.