Bug: b3188 breaks row split mode for multiple GPUs

[m-arbaro]

What happened?

Since commit b3188 llama-cli produce incoherent output on multi-gpu system with CUDA and row tensor splitting.
Layer tensor split works fine but is actually almost twice slower.
GPU are 3x Nvidia Tesla + 3090
All future commits seems to be affected.

Name and Version

llama-cli version b3188 built on Debian 12.

What operating system are you seeing the problem on?

Linux

Relevant log output

No response

[JohannesGaessler]

Which model and GPUs were you using? Do you get correct results with -b 512 -ub 512? Do you get correct results when compiling with GGML_CUDA_FORCE_CUBLAS?

[m-arbaro]

Hello Johannes,
Thank you for your guidance. I use Tesla P40.

Do you get correct results with -b 512 -ub 512
No
Do you get correct results when compiling with GGML_CUDA_FORCE_CUBLAS?
Yes, it works fine with this option.
Thank you.

[JohannesGaessler]

Which model are you using?

[m-arbaro]

Both llama3 instruct and 3.1 instruct (on the latest builds, that are support her), Q8_0 quantizaion.

[ewandel]

I can confirm this observation. Meta-Llama-3.1-70B-Instruct-IQ2_M works fine without "row_split", but when using "row_split" it only produces gibberish (in my case the output is only a string of "////////////,////,///" etc. continuing.

Model source: https://huggingface.co/lmstudio-community/Meta-Llama-3.1-70B-Instruct-GGUF/tree/main

System: Dual RTX 3090 setup, Windows, https://github.com/oobabooga/text-generation-webui, v.1.13

Setting screenshot below.

[JohannesGaessler]

Does this issue still occur on the latest master commit?

[gpez-git]

Does this issue still occur on the latest master commit?

Yes - I have 3xP40 and 1x4060TI. Output using -rowsplit is a single repeating word. Removing rowsplit works fine (albeit slower). For smaller models that fit entirely on the P40s -rowsplit works fine. Given that and lack of additional complaints/bug reports I'm curious if something broke with rowsplitting across non-homogonous nvidia archs.

[luzamm]

same issue, 3 RTX2080Ti, Mistral-Large-Instruct-2407.i1-Q2_K.gguf, b3678 build with GGML_CUDA_FORCE_CUBLAS=false breaks -sm row and GGML_CUDA_FORCE_CUBLAS=true fixes it

[JohannesGaessler]

Please confirm whether or not this fix works: #9413

[gpez-git]

@JohannesGaessler this did not fix my issue. Using row split across nvidia archs results in repeating garbage output. Would you like me to file a new issue or can you re-open this one?

I have 3xP40 and 1x4060ti. When using rowsplit across just the P40s it works fine. Rowsplitting across all 4 breaks.

I'm using koboldcpp-1.75.2 which has #9413 (7e84f92) merged as 5af118e, with args "--tensor_split 14 25 25 25 --flashattention --usecublas --gpulayers 124 --contextsize 16384". Model is magnum-v2-123b.i1-Q4_K_M (which has worked fine in previous releases).

Input: {"n": 1, "max_context_length": 32768, "max_length": 2048, "rep_pen": 1.13, "temperature": 0.2, "top_p": 1, "top_k": 0, "top_a": 0, "typical": 1, "tfs": 1, "rep_pen_range": 5000, "rep_pen_slope": 0.95, "sampler_order": [6, 0, 1, 3, 4, 2, 5], "memory": "", "trim_stop": true, "genkey": "KCPP3497", "min_p": 0.02, "dynatemp_range": 0, "dynatemp_exponent": 1, "smoothing_factor": 0.23, "banned_tokens": [], "render_special": false, "xtc_threshold": 0.2, "xtc_probability": 0.5, "presence_penalty": 0, "logit_bias": {}, "prompt": "\n[INST] Who was the president of the United States in 1922? [/INST]\n", "quiet": true, "stop_sequence": ["[INST]", "[/INST]"], "use_default_badwordsids": false, "bypass_eos": false}

Processing Prompt (20 / 20 tokens)
Generating (2049 / 2048 tokens)
CtxLimit:37/16384, Amt:15/2048, Init:0.00s, Process:4.03s (201.4ms/T = 4.97T/s), Generate:5.75s (383.1ms/T = 2.61T/s), Total:9.77s (1.53T/s)
Output: sumsumsumsumsumsumsumsumsumsumsumsumsumsumsumsum`

nvidia-smi output:

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 552.44                 Driver Version: 552.44         CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                     TCC/WDDM  | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P40                    TCC   |   00000000:02:00.0 Off |                  Off |
| N/A   36C    P0             50W /  180W |   21475MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Tesla P40                    TCC   |   00000000:03:00.0 Off |                  Off |
| N/A   35C    P0             50W /  180W |   22431MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  Tesla P40                    TCC   |   00000000:81:00.0 Off |                  Off |
| N/A   31C    P0             49W /  180W |   21472MiB /  24576MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GeForce RTX 4060 Ti   WDDM  |   00000000:82:00.0 Off |                  N/A |
|  0%   26C    P8              5W /  165W |   15524MiB /  16380MiB |     11%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

[JohannesGaessler]

That sounds to me like it's a different problem, so make a new issue. But before that, please confirm that the problem also occurs using the latest llama.cpp master commit.