Blackwell support/ rtx 50xx series

[Ghost4242]

Would be great if blackwell was supported I know whatever uses cuda 12.8 is still nightly build or whatever but it would be good to get it added anyway since not running at all sucks.

I know LM studio supports blackwell but is that because they use llama.cpp and not the python llama.cpp that this one uses? Or is it some other reason?

[Trojaner]

It runs fine on blackwell as long as you install torch nightly and use cuda 12.8 builds
e.g.

pip install torch==2.8.0.dev20250407+cu128 --extra-index-url https://download.pytorch.org/whl/nightly

You may also have to compile latest flash-attention, xformers, triton and/or llama.cpp from source with export TORCH_CUDA_ARCH_LIST="12.0". See related github repos for instructions on how to build them for Blackwell.

in short; this is unrelated to text-generation-webui, you just need the correct packages

[Ghost4242]

It runs fine on blackwell as long as you install torch nightly and use cuda 12.8 builds e.g.

pip install torch==2.8.0.dev20250407+cu128 --extra-index-url https://download.pytorch.org/whl/nightly

You may also have to compile latest flash-attention, xformers, triton and/or llama.cpp from source with export TORCH_CUDA_ARCH_LIST="12.0". See related github repos for instructions on how to build them for Blackwell.

in short; this is unrelated to text-generation-webui, you just need the correct packages

Not really tech savvy enough to do that

[SoftologyPro]

It runs fine on blackwell as long as you install torch nightly and use cuda 12.8 builds e.g.

pip install torch==2.8.0.dev20250407+cu128 --extra-index-url https://download.pytorch.org/whl/nightly

You may also have to compile latest flash-attention, xformers, triton and/or llama.cpp from source with export TORCH_CUDA_ARCH_LIST="12.0". See related github repos for instructions on how to build them for Blackwell.

in short; this is unrelated to text-generation-webui, you just need the correct packages

Can you give more details on how you install pytroch 2.8.0 nightly once Text Generation Web UI is installed?
Do you have a compatible WHL for flash-attn that will work with it?

[baobabKoodaa]

Related: #6729

[baobabKoodaa]

I was able to get text-generation-webui working with llama.cpp model loader on a 5090 like this:

1. Click cmd_windows.bat to open a terminal in the correct virtual environment.
2. Upgrade PyTorch to nightly 50xx compatible version:
   pip install --pre torch torchvision torchaudio --upgrade --index-url https://download.pytorch.org/whl/nightly/cu128

If you want to use other model loaders, you need to work more than this, but for llama.cpp, this should be enough.

[Katehuuh]

This win-fork worked for me: https://github.com/nan0bug00/text-generation-webui exl2+llama.cpp whl.

• Duplicate of Pytorch 2.7 and CUDA 12.8 for Nvidia Blackwell/RTX 50xx support #6897 – Fixed in main

[oobabooga]

Do CUDA 12.4 wheels (exllamav2/exllamav3/llama.cpp) not work with rtx 5000 series GPUs? I thought CUDA was backwards compatible -- llama.cpp for instance works on a 3090 with the latest driver if you use a version compiled with cuda 11.7.

[baobabKoodaa]

@oobabooga I don't believe 12.4 works with RTX 50xx. If it helps, I can run experiments for you on my 5090, if you tell me specifically what to try.

[oobabooga]

There is a discussion here: #6897

Can you try the cuda 12.4 portable build to see if it runs and the llama.cpp works?

[baobabKoodaa]

I can confirm it works! Portable build, cuda 12.4, llama.cpp, Windows, RTX 5090.

Now I'm wondering what was the issue I had a week ago (just before you published portable builds). Maybe it was related to PyTorch and CUDA 12.4 combination? If PyTorch is not used in the portable build in this configuration?

[Ghost4242]

Blackwell supports only cuda 12.8 If I remember correctly lama.cpp use cpu only if loaded without 12.8 cuda Den tis 29 apr. 2025 21:22Faeleon ***@***.***> skrev:

…

*Faeleon* left a comment (oobabooga/textgen#6828) <#6828 (comment)> I can confirm that the portable 12.4 works with on windows 11 with rtx 5090 but only with llampa.ccp on ExLlamav2_HF Traceback (most recent call last): File "F:\textgen-portable-3.1\text-generation-webui\modules\ui_model_menu.py", line 174, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "F:\textgen-portable-3.1\text-generation-webui\modules\models.py", line 39, in load_model from modules import sampler_hijack File "F:\textgen-portable-3.1\text-generation-webui\modules\sampler_hijack.py", line 6, in import torch ModuleNotFoundError: No module named 'torch' — Reply to this email directly, view it on GitHub <#6828 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUCXY2OZOBHORKOJQL2B5JD237GPTAVCNFSM6AAAAAB2MMFHJ2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMZZHE4DKOBSHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Katehuuh]

@Ghost4242 This issue now fixed in main branch: #6897 (comment)