bug(dspy): removed system_prompt from kwargs sent to vllm pydantic model #1029
Conversation
|
Hello @LorenzoMinto Thank you for your perseverance in trying to iron out this problem. I have run into the same issue and fixed it differently, see here: I then noticed the PR #1012 being in the works, despite having a different approach, thinking it would be sufficient. Also, I am not thinking my "fix" is any better, but I decided to start commenting on this, because I feel there is something wrong that I have to create such a "fix" in the first place. There are sampling parameters of vLLM stored in the same set of key words as some keywords used for only DSPy internal processing. This is asking for trouble down the road. While I don't want to discourage anything about adding "system_prompt" to the elements which should be popped out here, I want to ask for discussion if that is not asking for further trouble down the road and if not a deeper fix is needed to separate DSPy internal parameters from sampling parameters, which will make vLLM choke. It is very strict with regard to the correctness of the requests it will process. What do you think? |
|
I merged this to my local dspy version and have the same issue, seems like another parameter vllm doesn't take |
|
@Wolfsauge I went ahead and created a pull request using your code, hope you don't mind. |
No description provided.