You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Running this as-is in a Colab notebook will result in "ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5." This is discussed in vLLM issues here.
Because everyone who uses Colab would likely use the T4 GPU (if they're on the free Colab at least), they'd face the same error every time if they copy our snippet from the Docs and try to test-run vLLM in their notebook.
I'd want to change the snippet to something like this:
llm=vLLM(
model=LLM(model="argilla/notus-7b-v1", dtype="Bfloat16"), # If using Tesla T4 on Colab, # specify dtype = float16 to prevent compute compatibility errortask=TextGenerationTask(),
and add a brief clarification about it in the text with a link for anyone who wants to understand it further.
The text was updated successfully, but these errors were encountered:
Which page or section is this issue related to?
Currently the code snippet in the vLLM section of the guide (https://distilabel.argilla.io/latest/technical-reference/llms/#vllm) looks like:
Running this as-is in a Colab notebook will result in "ValueError: Bfloat16 is only supported on GPUs with compute capability of at least 8.0. Your Tesla T4 GPU has compute capability 7.5." This is discussed in vLLM issues here.
Because everyone who uses Colab would likely use the T4 GPU (if they're on the free Colab at least), they'd face the same error every time if they copy our snippet from the Docs and try to test-run vLLM in their notebook.
I'd want to change the snippet to something like this:
and add a brief clarification about it in the text with a link for anyone who wants to understand it further.
The text was updated successfully, but these errors were encountered: