Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump vLLM version to 0.3.0 #579

Merged
merged 1 commit into from
Feb 5, 2024
Merged

Conversation

lapp0
Copy link
Contributor

@lapp0 lapp0 commented Jan 24, 2024

  • pin new version
  • Fix serving with vLLM, LLMEngine.tokenizer uses a new class now: TokenizerGroup
  • Update docs regarding pydantic

Fixes #576

Fixes #504

vLLM main is incompatible with outlines. Do not merge until vllm==0.2.8 is released, leaving as draft until then.

@lapp0 lapp0 changed the title Fix serving with vLLM, uses new tokenizer Fix serving with vLLM, LLMEngine.tokenizer uses a new class now: TokenizerGroup Jan 25, 2024
@lapp0 lapp0 changed the title Fix serving with vLLM, LLMEngine.tokenizer uses a new class now: TokenizerGroup Staging for vLLM 0.2.8 Jan 25, 2024
@rlouf rlouf added the vLLM Things involving vLLM support label Jan 26, 2024
Copy link
Contributor

@jc-louis jc-louis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vLLM 0.3.0 was just released

(you should also be able to remove this line as vLLM 0.3.0 did migrate to pydantic 2.0)

pyproject.toml Outdated
@@ -59,7 +59,7 @@ test = [
"huggingface_hub"
]
serve = [
"vllm>=0.2.6",
"vllm>=0.2.8",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

vLLM skipped 0.2.8

Suggested change
"vllm>=0.2.8",
"vllm>=0.3.0",

@rlouf
Copy link
Member

rlouf commented Jan 31, 2024

Let's add vllm>=0.3.0 to pyproject.toml as well. I'll cut a new release as soon as all of this is ready.

@rlouf rlouf marked this pull request as ready for review February 1, 2024 10:49
@rlouf rlouf changed the title Staging for vLLM 0.2.8 Bump vLLM version to 0.3.0 Feb 1, 2024
@rlouf
Copy link
Member

rlouf commented Feb 1, 2024

@lapp0 I made a few changes on your branch. Can you confirm that it works on your end, I don't have access to my desktop today.

@lapp0
Copy link
Contributor Author

lapp0 commented Feb 3, 2024

@remic33 yes, I will smoke test in a few hours and report the results.

@lapp0
Copy link
Contributor Author

lapp0 commented Feb 4, 2024

Change-set looks good after a simple smoke test.

On this branch with vllm==0.3.0:

root@C.9153721:~/outlines$ pip freeze | grep -P "(outlines|vllm)"
UNKNOWN @ file:///root/outlines
vllm==0.3.0
root@C.9153721:~/outlines$ git status
On branch fix-vllm-new-tokenizer
Your branch is up to date with 'origin/fix-vllm-new-tokenizer'.
curl http://127.0.0.1:8000/generate \
    -d '{
        "prompt": "What is Pi? Give me the first 15 digits: ",
        "regex": "(-)?(0|[1-9][0-9]*)(\\.[0-9]+)?([eE][+-][0-9]+)?"
        }'

{"text":["What is Pi? Give me the first 15 digits: 3.14159265358979"]}

On main with vllm==0.3.0:

AttributeError: 'TokenizerGroup' object has no attribute 'get_vocab'.

@rlouf
Copy link
Member

rlouf commented Feb 5, 2024

All good then, merging

@rlouf rlouf merged commit 80c0957 into dottxt-ai:main Feb 5, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
vLLM Things involving vLLM support
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Outlines serve Broken if Using vLLM main Install Pydantic>=2.0 after vLLM installation
3 participants