HFClientVLLM passes extra parameters to vLLM server causing extra_forbidden error

[DamianB-BitFlipper]

DSPy version: 2.4.9
vLLM version: 0.4.2

When the HFClientVLLM performs an HTTP request to a vLLM server, it passes kwargs. The kwargs has values such as port and url that are not recognized by the vLLM server. This is in this line

The issue is that in the latest vLLM release, extra (unrecognized) parameters, throw an error. This is due to line .

In all, when using the HFClientVLLM, this causes the following error:

Failed to parse JSON response: {"object":"error","message":"[{'type': 'extra_forbidden', 'loc': ('body', 'port'), 'msg': 'Extra inputs are not permitted', 'input': 443}, {'type': 'extra_forbidden', 'loc': ('body', 'url'), 'msg': 'Extra inputs are not permitted', 'input': ['localhost:443']}]","type":"BadRequestError","param":null,"code":400}
Traceback (most recent call last):

I hope that this issue is insightful.

[omkar-sh]

Do we remove only port and url or all the kwargs provided by the user that are not in pydantic variables?

[DamianB-BitFlipper]

So the way vLLM is structured, any field which is not a parameter to this base model will raise: https://github.com/vllm-project/vllm/blob/c7f2cf2b7f67bce5842fedfdba508440fe257375/vllm/entrypoints/openai/protocol.py#L69Now, I believe all you inject extra is the port and url. If you remove those two parameters, it should be good. Please don’t remove any parameters that the user gave. This maximizes the user’s flexibility when interacting with vLLM. I’d be glad to test this once you have a branch up, but ideally it would be better to include a unit test for posterity. Thanks for looking into this. DamianOn 11. May 2024, at 14:07, omkar-sh ***@***.***> wrote: Do we remove only port and url or all the kwargs provided by the user that are not in pydantic variables? —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: ***@***.***>

[omkar-sh]

Made this PR. Please checkout. Link

[DamianB-BitFlipper]

Hey. Thanks for your PR. I do not think that does what you think it does. The keys port and url should be removed from the kwargs. What you are doing is filtering only them. I would suggest using the dict.pop method: https://www.programiz.com/python-programming/methods/dictionary/pop

[omkar-sh]

I missed negating the if statement. Fixed the same now.

[DamianB-BitFlipper]

Moving this conversation to the PR. I left a comment there.

[LorenzoMinto]

Also system_prompt looks like should be removed. Added a PR here.