Optillm unable to serve requests with 'top_k' or 'chat_template_kwargs' parameters

[itsmeknt]

I am using Optillm as a proxy to my locally hosted models (llama.cpp, VLLM, sglang).

These locally hosted models have some additional parameters that the standard OpenAI API doesn't have.

Two of them are top_k which is another sampling parameter, and chat_template_kwargs which dynamically modifies the prompt template (for e.g. setting the reasoning level).

I would like to include these parameters in the request to Optillm to forward to my locally hosted models, since they make a big difference in generation quality, but Optillm gives a 400 Bad Request error.

Here is the test curl command for reference:

curl -X POST http://127.0.0.1:8101/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"qwen3-32b-awq","messages":[{"role":"system","content":"You are a concise assistant."},{"role":"user","content":"I have this mermaid diagram, but its giving me an error. How do I fix it? Can you think step by step and give me the reasoning leading up to the final conclusion. Please include at least 10 thinking steps enclosed in <think></think>, then give me 10 alternate solutions.\n\nMermaid diagram: flowchart TD\n  %% ====================== Build Pipeline ======================\n  subgraph BuildPipeline[\"Build Pipeline\"]\n    Dev[Developer Machine] -->|1. Build Go\nbinary (static, cross-compiled)| GoBin[Go Static Binary]\n    GoBin -->|2. Build Docker image &lpar;scratch&rpar;| DockerImg[Docker Image]\n    DockerImg -->|3. Push to registry| Registry[Container Registry]\n  end\n\nError: Error: Error: Parse error on line 4:\n...|1. Build Go\nbinary (static, cross-compi\n-----------------------^\nExpecting \"SQE\", \"DOUBLECIRCLEEND\", \"PE\", \"-)\", \"STADIUMEND\", \"SUBROUTINEEND\", \"PIPE\", \"CYLINDEREND\", \"DIAMOND_STOP\", \"TAGEND\", \"TRAPEND\", \"INVTRAPEND\", \"UNICODE_TEXT\", \"TEXT\", \"TAGSTART\", got \"PS\""}],"temperature":0.75,"max_tokens":8192,"top_p": 0.85,"top_k": 25,"n": 1,"stream": false, "chat_template_kwargs": {"enable_thinking": true}}'

Would it be possible to forward these parameters?

[codelion]

Can not add them to the extra_body with the OpenAI SDK itself and use with the local inference server?

response = client.chat.completions.create(
  model="meta-llama/Llama-3.2-1B-Instruct",
  messages=messages,
  temperature=0.2,
  extra_body={
        "chat_template_kwargs": {"enable_thinking": true},
        "top_k": 27,
        "min_p": 0.03,
    }
)

[itsmeknt]

I forgot to mention that I am running Optillm locally (cloning the repo @ commit 4e414c46ef19f2ff4fdf7f03ac7fc506976988bf, set up venv, pip install -r requirements.txt).

The OpenAI SDK doesnt work for me either. It still complains about top_k or extra_chat_kwargs:

from openai import OpenAI
    
if __name__ == "__main__":
    client = OpenAI(
        base_url="http://localhost:8102/v1",
        api_key="dummy"
    )
    messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Tell me a fun fact about space."},
    ]
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.2-1B-Instruct",
        messages=messages,
        temperature=0.2,
        extra_body={
              "top_k": 27,
              "min_p": 0.03,
        }
    )
    print(response)

Response:

Traceback (most recent call last):
  File "/home/kevin/workspace/optillm/test_request.py", line 12, in <module>
    response = client.chat.completions.create(
        model="meta-llama/Llama-3.2-1B-Instruct",
    ...<5 lines>...
        }
    )
  File "/home/kevin/workspace/optillm/.venv/lib/python3.13/site-packages/openai/_utils/_utils.py", line 287, in wrapper
    return func(*args, **kwargs)
  File "/home/kevin/workspace/optillm/.venv/lib/python3.13/site-packages/openai/resources/chat/completions/completions.py", line 1147, in create
    return self._post(
           ~~~~~~~~~~^
        "/chat/completions",
        ^^^^^^^^^^^^^^^^^^^^
    ...<46 lines>...
        stream_cls=Stream[ChatCompletionChunk],
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/kevin/workspace/optillm/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1259, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
                           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/kevin/workspace/optillm/.venv/lib/python3.13/site-packages/openai/_base_client.py", line 1047, in request
    raise self._make_status_error_from_response(err.response) from None
openai.InternalServerError: Error code: 500 - {'error': "Completions.create() got an unexpected keyword argument 'top_k'. Did you mean 'top_p'?"}

This is the terminal output from the shell running Optillm:

2025-08-27 12:20:09,779 - INFO - Received request to /v1/chat/completions
2025-08-27 12:20:09,794 - INFO - Using approach(es) ['none'], operation SINGLE, with model meta-llama/Llama-3.2-1B-Instruct
2025-08-27 12:20:09,794 - ERROR - Error in none approach: Completions.create() got an unexpected keyword argument 'top_k'. Did you mean 'top_p'?
2025-08-27 12:20:09,794 - ERROR - Error processing request: Completions.create() got an unexpected keyword argument 'top_k'. Did you mean 'top_p'?
2025-08-27 12:20:09,794 - INFO - 127.0.0.1 - - [27/Aug/2025 12:20:09] "POST /v1/chat/completions HTTP/1.1" 500 -

[codelion]

Ok, will fix it in #234 as well!