Tesla P4 Not Used

[cebtenzzre]

The P4-Card is visible in the devicemanger and i have installed the newest vulkan-drivers and cudnn-lib. (https://developer.nvidia.com/vulkan-driver) - but the P4 is (still) not used.
Which additional drivers are necessary for GPT4all - or evtl. manaul entries into some files ?

It would make sense to show in a field, which card is used or the posibility like in LM-Studio to adjust, how much Ram of the card is used.
(P.S. Mistral-openorca Q6 is much faster than Mistral1.2)

Originally posted by @gtbu in #1843 (comment)

[cebtenzzre]

What is the output of vulkaninfo --summary? You may need to install the Runtime or SDK from here: https://vulkan.lunarg.com/sdk/home

[gtbu]

As i said in another issue, i have of course installed the sdk-runtimes (and published a link there)! I think the problem is that Gtp4all uses either a cpu or a gpu bot not both (why not, if the Ram is small - of course difficult to program such a code) and not 'either of them AND a cuda-card ' without gpu. I

[cebtenzzre]

I need the output of vulkaninfo --summary to have a better idea of what GPT4All sees when it tries to find your GPU. I just mentioned the Vulkan SDK in case that command is not found for some reason.

Btw, CUDA is irrelevant here, as GPT4All does not use it. It uses Vulkan as a compute backend. Tesla card are GPUs and can do graphics, they just don't have any video outputs. But GPT4All does not care whether the card has any video outputs - I use my Tesla P40 with GPT4All and it works just fine.

[gtbu]

I append the vulkaninfo --summary > d:\sum2.txt
sum2.txt
-- with warnings in the powershell appeded. Gpt4all is not on

[cebtenzzre]

So, this is not a GPT4All issue. Your installed NVIDIA driver is not providing the Vulkan API for your Tesla P4. One of these might do it:
https://www.nvidia.com/download/driverResults.aspx/222668/en-us/
https://www.nvidia.com/download/driverResults.aspx/221875/en-us/

I'm using Linux, and the standard nvidia proprietary driver supports Vulkan on my Tesla P40.

[gtbu]

No : I cant install the desktop driver because it is not a gpu-card. I had installed the datacenter-driver and installed now Your proposed new version - with success, then also the newest vulkan runtime (vulkan now 1.3.275 : though it is not documented as necessary with gpt4all).
I still see no card in the dropdown(auto - gpu) . Here Your new vulkaninfo --summary > d:\sum3.txt ( just the same)
sum3.txt
Just in powershell appear now no more warnings. I have now between 2.5 (mixtral instruct.Q6) and 4 tokens(mistral 1.2-Q4)

[sorasoras]

No : I cant install the desktop driver because it is not a gpu-card. I had installed the datacenter-driver and installed now Your proposed new version - with success, then also the newest vulkan runtime (vulkan now 1.3.275 : though it is not documented as necessary with gpt4all). I still see no card in the dropdown(auto - gpu) . Here Your new vulkaninfo --summary > d:\sum3.txt ( just the same) sum3.txt Just in powershell appear now no more warnings. I have now between 2.5 (mixtral instruct.Q6) and 4 tokens(mistral 1.2-Q4)

You have to switch to driver mode from TCC mode to WDDM mode to be able to use graphic APIs
I remember there is a guide somewhere.

[cebtenzzre]

So, there's your answer (thanks sorasoras) - you can only use the Tesla P4 with GPT4All on Windows if you have a GRID license, and use nvidia-smi to switch to WDDM mode: https://docs.nvidia.com/nsight-visual-studio-edition/reference/index.html#setting-tcc-mode-for-tesla-products

This is a limitation of Tesla devices on Windows. Unless GPT4All adds support for llama.cpp's CUDA backend, or you downgrade to an older driver that doesn't require a GRID license, there is no way around this.

[gtbu]

Tesla - Cuda devices like P4 improve gaming-speed on windows without special grid-license .
( I cant install the desktop-driver, because it is for a nvidia gpu-card (and i have only the gpu of the processor) and the installation cant find an nvidia card .)
For what do we need a vGPU ? LM-Studio doesnt need one.
Can You please give me the name of such an older driver that doesn't require a GRID license (and version) ?
I found something at

• https://learn.microsoft.com/en-us/azure/virtual-machines/windows/n-series-driver-setup and
• https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/install-nvidia-driver.html#nvidia-GRID-driver and
• https://developer.download.nvidia.com/compute/cuda/redist/
• https://developer.nvidia.com/cuda-toolkit-archive

• do You plan support for llama.cpp's CUDA backend ?
  ....................................
  P.S. Grid - vGPU - license is a bit more complicated
  vGPUs that require licensing run at full capability even without a license. However, on Windows, users are warned each time a vGPU fails to get a license until a license is acquired1.

[cebtenzzre]

For the last time, the Tesla P4 is a fully featured GPU (minus the physical display connectors), not just a "CUDA card", NVIDIA just treats Tesla GPUs differently from e.g. GeForce and Quadro on Windows. You should not need a vGPU, only WDDM mode so you can use Vulkan for compute. GPT4All does not use CUDA at all at the moment.

I found this guide, maybe it helps.

[gtbu]

Yes - but the desktop-driver doesnt recognize the P4 and similar others as GPU. The P4 and others are no GRID-Cards which require a license.
The above links says : Enable Above 4G memory (my bios doesnt have that point. Evtl. it is default. The windows-device-manager sees the P4 and the inner gpu of the cpu parallel ) ;
Disable CSM : If not disabled, the Nvidia graphics card selection will give an error message (not on my computer). He wants a second geforce-graphics card (...no free second long slot as most boards)
I will go on at that registry basis (make a registry back before with Erunt)
If It doesnt help : Why functions LM-Studio with that . The Cuda-datacenter-driver ist installed. It only needs some code which switches to WDDM mode .

• You said that You have a P40 - can You post Your way here please !
• I do not want to have the P4 as a second graphics - card with just 8 MB Ram as a dropdown-choice but as in LMSTUdio as an extension for more computing power. In LM-Studio i have up to 8 Tokens/s.
• At https://cloud.google.com/compute/docs/gpus/grid-drivers-table?hl=de#windows_drivers is a windows10/11 - grid- driver ( 13.1 | 472.39 )
• At Youtube is a video and another one

**** To finalize this discussion : With grid-driver 4.72.39 (500+ do not function) the nvidia wmi-driver is installed and the card appears in the dropdown. If i choose the P4, then i get an 'out of Vram-error' (as expected). It seems that Gpt4all uses either Cpu or on-chip-Gpu or P4-card or a graphics-card and not all (as LM-Strudio does - which is therefore faster ). This should be the next point in develeopment (from my sight). If LMstudio develops a localdocs-extension, then You will have a problem.

[cebtenzzre]

It seems that Gpt4all uses either Cpu or on-chip-Gpu or P4-card or a graphics-card and not all

I believe that LM Studio uses the llama.cpp CUDA backend. The fastest use of llama.cpp is always to use a single GPU which is the fastest one available. And your integrated Intel GPU certainly isn't supported by the CUDA backend, so LM Studio can't use it. Since the CPU is universally slower than the GPU for LLMs, you should only split computation between CPU and GPU if you would run out of VRAM otherwise - you can do this in GPT4All by adjusting the per-model "GPU layers" setting.

The main reason that LM Studio would be faster than GPT4All when fully offloading is that the kernels in the llama.cpp CUDA backend are better optimized than the kernels in the Nomic Vulkan backend. This is something we intend to work on, but there are higher priorities at the moment.