Add vllm chat completion endpoints and fix inspect_history

[noobHappylife]

Related to issue #778 #784

• added vllm chat completion endpoints (support system_prompt)
• fixed an issue with .copy() with vllm client module. (need to include model, url, port into kwargs)
• fix inspect_histrory not respecting n/skip as mentioned in Inspect_history only shows the last history regardless of n/skip #784

[arnavsinghvi11]

model is in fact included in the kwargs already, inherited through HFModel-> LM so we can remove this.

is there a use case for including the port and URL in the LM kwargs? seems like they are only used for setting the URL(s) and not in the model payload? feel free to remove if not.

[noobHappylife]

Noted on the model part. For url and port, basically is when calling lm.copy() (ex: while compiling a program with bootstrap), it is missing the url and port kwargs. As shown as below. Thus, requiring the port and url in the kwargs for the copy() to work properly.

in BootstrapFewShot._bootstrap_one_example(self, example, round_idx)
    148 with dsp.settings.context(trace=[], **self.teacher_settings):
    149     lm = dsp.settings.lm
--> 150     lm = lm.copy(temperature=0.7 + 0.001 * round_idx) if round_idx > 0 else lm
    151     new_settings = dict(lm=lm) if round_idx > 0 else {}
    153     with dsp.settings.context(**new_settings):

File [~/micromamba/envs/yh_hf/lib/python3.10/site-packages/dsp/modules/lm.py:108](http://192.168.77.2:8892/lab/tree/~/micromamba/envs/yh_hf/lib/python3.10/site-packages/dsp/modules/lm.py#line=107), in LM.copy(self, **kwargs)
    105 kwargs = {**self.kwargs, **kwargs}
    106 model = kwargs.pop("model")
--> 108 return self.__class__(model=model, **kwargs)

TypeError: HFClientVLLM.__init__() missing 1 required positional argument: 'port'

[arnavsinghvi11]

let's actually move this to the initialization layer so the user can specify the model_type when configuring the vLLM client, not during generation time.

[noobHappylife]

OK, will include into the initialization layer.

[arnavsinghvi11]

is there a reason for this change? seems to have the same effect originally?

[arnavsinghvi11]

currently failing these tests

[noobHappylife]

I've included some of my testing with the inspect_history in the issue #784. Can you take a look, or I'm misunderstanding how it should works?

[arnavsinghvi11]

Ah thanks for highlighting. Just realized that this is a bug where the printing_value is getting reset everytime in the loop. That should be all the error is and the skip condition is fine as is.

[noobHappylife]

Thank you. Let me know what else is needed to change. I've updated to the vllm initialization part, and fix the error, so the test should pass now.

[arnavsinghvi11]

yeah can you revert the changes made to inspect_history here and just move the printing_value outside the for loop?

[noobHappylife]

ok, updated and tested it works. Thank you. Now inspect_history is able to view n-number of histories with skip.

[ammirsm]

good catch folks!

thanks for it!

[arnavsinghvi11]

Thanks @noobHappylife for the additions to vLLM. left a couple comments and should be good to merge following those comments and running ruff check . --fix-only .

[arnavsinghvi11]

Thanks @noobHappylife !