Comparison between Langchain-VLLM and vllm-project vLLM

[DiwanshuShekhar]

Can somebody please help me understand the differences between Langchain's VLLM and this project's vLLM? From the documentation of Langchain's vLLM, it seems like they use PagedAttention as well, so I am curious about the differences here. Thanks in advance

[simon-mo]

It looks like Langchain just imports vLLM.

[DiwanshuShekhar]

Oh yea, didn't notice that. Thanks for pointing it out. Langchain's VLLM seems to be just the wrapper on vLLM. I was debating whether I should use vLLM or Langchain's VLLM, but since they are the same, I'd probably use Langchain's VLLM - that way I can get other bells and whistles that come with Langchain while benefiting this project's vLLM. Do you see any downside to this aproach?

[simon-mo]

You can also use vLLM's OpenAI compatible server and point Langchain to it. So that it might have more features from Langchain side (e.g. streaming).