Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix crash when using vLLM without structured generation #658

Merged
merged 2 commits into from
May 22, 2024

Conversation

cg123
Copy link
Contributor

@cg123 cg123 commented May 21, 2024

When using the vLLM backend, currently structured generation must be set or the task will crash with the following error:

 "/usr/local/lib/python3.10/dist-packages/vllm/model_executor/layers/logits_processor.py", line 106, in _apply_logits_processors
    logits_row = logits_processor(token_ids, logits_row)
TypeError: 'NoneType' object is not callable

This is because [self._logits_processor] is passed to vLLM as the list of logit processors, even when self._logits_processor is None. vLLM isn't picky and happily calls that None.

Just a one-line tweak to make things work again.

@alvarobartt
Copy link
Member

Thanks for the quick fix @cg123! We'll release distilabel v1.1.1 tomorrow with this, thanks!

@alvarobartt alvarobartt changed the base branch from develop to main May 21, 2024 19:44
@alvarobartt alvarobartt merged commit b8fe3dc into argilla-io:main May 22, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

None yet

3 participants