Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add vLLM integration #772

Merged
merged 3 commits into from
Apr 1, 2024
Merged

Add vLLM integration #772

merged 3 commits into from
Apr 1, 2024

Conversation

rlouf
Copy link
Member

@rlouf rlouf commented Mar 27, 2024

In this PR we add an integration for the offline batching mode of vLLM with the following interface:

from outlines import models, generate
from vllm.sampling_params import SamplingParams


model = models.vllm("model_name", **vllm_specific_kwargs)
generator = generate.regex(model, "[0-9]{3}")

model.load_lora("path_to_lora")

params = SamplingParams()
answer = generator("Generate a number", max_tokens=10, sampling_params=params)

While we adopt the overall design of Outlines we let users specify vllm-specific parameters. They can also initialize the model using an instance of vllm.LLM.

TODO

  • Test passing custom parameters to model init
  • Test passing custom parameters to generation
  • Handle sampling parameters
  • Add load_lora method
  • Add docstrings
  • Add documentation
  • Add issue for CFGLogitsProcessor with vLLM
  • Open issues for refactor of llama.cpp integration

@rlouf rlouf added enhancement vLLM Things involving vLLM support labels Mar 27, 2024
@rlouf rlouf force-pushed the add-vllm branch 7 times, most recently from aef1e4c to d8c64d3 Compare April 1, 2024 08:45
@rlouf rlouf force-pushed the add-vllm branch 2 times, most recently from 8c9e6b4 to a5fa0aa Compare April 1, 2024 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement vLLM Things involving vLLM support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant