Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Compatibility issue with vllm 0.32 #704

Closed
p-usefulbird opened this issue Feb 23, 2024 · 4 comments · Fixed by #711
Closed

Compatibility issue with vllm 0.32 #704

p-usefulbird opened this issue Feb 23, 2024 · 4 comments · Fixed by #711
Labels
bug

Comments

@p-usefulbird
Copy link

Describe the issue as clearly as possible:

A recent merge into vllm broke the integration with outlines. The deepcopy introduced does not work with outlines due to the dictionaries nested. This throws the following error:

TypeError: cannot pickle 'dict_values' object

vllm-project/vllm#2881

Steps/code to reproduce the bug:

Running vllm with the latest pull of vllm 

python -m outlines.serve.serve --model="mistralai/Mistral-7B-Instruct-v0.2"

Expected result:

vllm running normally

Error message:

INFO:     127.0.0.1:40000 - "POST /generate HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 29, in _raise_exception_on_finish
    task.result()
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 411, in run_engine_loop
    has_requests_in_progress = await self.engine_step()
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 382, in engine_step
    await self.engine.add_request_async(**new_request)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 241, in add_request_async
    return self.add_request(
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 468, in add_request
    sampling_params = copy.deepcopy(sampling_params)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 206, in _deepcopy_list
    append(deepcopy(a, memo))
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 146, in deepcopy
    y = copier(x, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
  File "/home/azureuser/miniconda3/envs/vllm/lib/python3.10/copy.py", line 161, in deepcopy
    rv = reductor(4)
TypeError: cannot pickle 'dict_values' object

Outlines/Python version information:

Version information

``` (command output here) ```

Context for the issue:

No response
@MotzWanted
Copy link

I'm also having this issue. Are you having any concerns downgrading vllm to v0.3.1, and, thus, not serving a model with defensive sampling?

@lapp0
Copy link
Collaborator

lapp0 commented Feb 23, 2024 •
edited
Loading

vLLM is working to integrate outlines vllm-project/vllm#2819. Once they have it integrated hopefully this kind of error won't pop up and the systems will retain compatability.

To resolve this we can simply apply list() to RegexFSM.vocabulary: https://github.com/outlines-dev/outlines/blob/main/outlines/fsm/fsm.py#L124

>>> rfsm = RegexFSM("a", tokenizer)
>>> copy.deepcopy(rfsm)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/lib/python3.11/copy.py", line 172, in deepcopy
    y = _reconstruct(x, memo, *rv)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/lib/python3.11/copy.py", line 271, in _reconstruct
    state = deepcopy(state, memo)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/lib/python3.11/copy.py", line 146, in deepcopy
    y = copier(x, memo)
        ^^^^^^^^^^^^^^^
  File "/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/lib/python3.11/copy.py", line 231, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/y027d3bvlaizbri04c1bzh28hqd6lj01-python3-3.11.7/lib/python3.11/copy.py", line 161, in deepcopy
    rv = reductor(4)
         ^^^^^^^^^^^
TypeError: cannot pickle 'dict_values' object
>>> rfsm.vocabulary = list(rfsm.vocabulary)
>>> copy.deepcopy(rfsm)
<outlines.fsm.fsm.RegexFSM object at 0x7f601eb78a10>

Good first PR for anyone looking to start contributing!

@lapp0
Copy link
Collaborator

lapp0 commented Feb 23, 2024

For CFGFSM, we might choose a hacky solution for now that resolves this issue, otherwise trying to deepcopy a CFGFSM results in

TypeError: cannot pickle 'module' object

The hacky solution would be to construct a brand new CFGFSM from the start state (not a true deepcopy).

    def deepcopy(self) -> "CFGFSM":
        return copy(self)

@saattrupdan saattrupdan mentioned this issue Feb 27, 2024
Merged
@saattrupdan
Copy link
Contributor

saattrupdan commented Feb 27, 2024 •
edited
Loading

I've encountered this issue as well and have attempted a fix here: #711

@p-usefulbird Can you test and see if that fixes it?

rlouf pushed a commit that referenced this issue Feb 27, 2024
@saattrupdan
Fix vLLM integration (#711)
d938678 
When integrating Outlines with vLLM I faced the following issues, which
are fixed in this PR:

1. When calling `vllm.LLM.generate` then within the internals of vLLM a
`copy.deepcopy` of the vLLM `SamplingParams` is made, which includes the
logits processor from Outlines (`RegexLogitsProcessor`, say). This
requires everything to be pickleable, and the
`RegexLogitsProcessor.fsm.vocabulary` is a `dict_values` object, which
doesn't satisfy that. The fix is easy: just convert it to a list. This
doesn't affect how this `vocabulary` variable is being used in the code.
2. The `RegexLogitsProcessor` takes an `llm` argument, which the
docstring states should be a `vllm.LLM` object, but then attempts to
extract the underlying tokenizer via `llm.tokenizer.tokenizer`. The
tokenizer of `vllm.LLM` currently lies in the
`vllm.LLM.llm_engine.tokenizer.tokenizer` attribute, but this is a big
mess and isn't backwards compatible with previous vLLM versions.
Instead, they have a convenience method, `vllm.LLM.get_tokenizer`, which
fetches the tokenizer. To remain backwards compatibility, in case people
have supplied `vllm.LLM.llm_engine` directly into
`RegexLogitsProcessor`, it falls back to a `tokenizer` or
`tokenizer.tokenizer` attribute.

I also updated the vLLM example script, as that was outdated as well
(used the previous `_patched_apply_logits_processors`).

Closes #704
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix vLLM integration
4 participants
@lapp0 @MotzWanted @saattrupdan @p-usefulbird