You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
vLLM is one of the most popular and effective tool for quick, large-scale LLM inference. Are there any existing examples of incorporating vLLM in Lightning? I have not found any so far.
Pitch
Adding inference via vLLM under the Lightning framework.
YuWang916
changed the title
Are there any existing examples incorporating vLLM in Lightning?
How to incorporate vLLM in Lightning for LLM inference?
Apr 30, 2024
Description & Motivation
vLLM is one of the most popular and effective tool for quick, large-scale LLM inference. Are there any existing examples of incorporating vLLM in Lightning? I have not found any so far.
Pitch
Adding inference via vLLM under the Lightning framework.
Alternatives
No response
Additional context
No response
cc @Borda
The text was updated successfully, but these errors were encountered: