[Feature request] Add custom models names and other body parameters

[vlbosch]

In the current implementation, models are fetched from a (custom) OpenAI API. Some clients (like mlx-omni-server) support loading models from the Huggingface cache (which models will also be shown in the list after refreshing), but you can also load custom Model-ID's. Could this feature be added, so that models can be added manually?

Also, some providers support body parameters (like a draft model to use). Could you support the option to add such body parameters?

Thanks in advance and keep up the great work! :-)

[nbonamy]

For custom engines the model field is in fact editable. Agreed that this is not very visible 😀

[nbonamy]

What do you mean by "other body parameters"?

[nbonamy]

@vlbosch any updates? otherwise will close this issue

[vlbosch]

Thanks for your reply! I did find that I can indeed add custom models, just by typing over an already returned name. Indeed not the most intuitive at first, but it works just fine.

With regard to custom body parameters. Some OpenAI API compatible servers have different parameters or custom parameters. See for example the parameters for llama.cpp-server (https://github.com/ggml-org/llama.cpp/blob/master/examples/server/README.md), like dry_multiplier. Now it's impossible to set those values via Witsy.

So my request is about adding custom key-value pairs to the (I think) "data"-part of the API-request. This would also allow to set other parameters which are also by default supported by OpenAI, but cannot be set in the UI now.

I also noticed that top_k is capped at 20, while for example with Gemma 3 a value of 64 is advised. Could this be changed so that top_k can be set to maximum 100?

[nbonamy]

The problem is that Witsy is using the OpenAI SDK which does not support custom parameters. As you can see the custom endpoint that supports all llama.cpp extension is specific to llama.cpp:

Witsy uses the /v1/chat/completions endpoint (through OpenAI SDK) that llama.cpp says llama.cpp /completion-specific features such as mirostat are also supported so it may be worth a try but no guarantee that it works as I am using the SDK (which beyond complaining that these parameters are not supported may filter them out...) and not a direct curl call. Will look into it. Later 😀

[nbonamy]

TopK fixed in 2.4.2 pre-release
https://github.com/nbonamy/witsy/releases/tag/v2.4.2

[nbonamy]

Added something in updated 2.4.2 pre-release. Tested very quickly so please provide feedback. Will probably release 2.4.2 anyway as this is probably a very much niche feature 😀