1-2 Tesla P40 plus a powerful graphics card, does it make sense?

[Vladonai]

I'm not familiar with how llamacpp works, but I've heard that some of the context handling work is done exclusively on GPU0. In this connection there is a question: is there any sense to add one more but powerful video card, for example RTX3090, to 1-2 Tesla P40 video cards? If GPU0 becomes this particular graphics card, won't it improve some properties of the inference? Especially with rowsplit mode.

Alternatively, I could buy another Tesla P40, but in that case I already know what I'm getting. The RTX3090 costs as much as 4 Tesla P40s, but maybe buying it would speed up generation on larger context sizes? That would make sense then.

[simsi-andy]

I got a machine with a rtx 3090 and a server with four p40ties. Both are great. But you can't mix them. This is not supported by their drivers. I tried it out. If you find a way to do it, it would be interesting.

But generally speaking: P40ties work great for llms (do not switch on ecc!). But they are slow for stable diff.

[Vladonai]

If you find a way to do it, it would be interesting.

Try this method:
Download and install the latest drivers for Tesla Datacenter (clean install)
Reboot your system
Download and install the latest version of drivers for RTX3090 (Studio edition). Warning - do not do a "clean install" this time!
Reboot your system and everything should work.

All cards are visible through nvidia-smi, and you can use them (all or some of them) by writing the following in the bat file before running llamacpp:
set CUDA_VISIBLE_DEVICES=0,1,2 etc., where 0,1,2 are the GPU id's of your cards, as I understand it - in the order in which they are installed in the physical slots on your motherboard. Try this method and post the result here.

[sorasoras]

I got a machine with a rtx 3090 and a server with four p40ties. Both are great. But you can't mix them. This is not supported by their drivers. I tried it out. If you find a way to do it, it would be interesting.

But generally speaking: P40ties work great for llms (do not switch on ecc!). But they are slow for stable diff.

You could mix them. I said people mixing 3090 with p40s.
As long as you can offload most of the stuff onto 3090, you would not experience slow down i.e use P40 to extend very long context.

[simsi-andy]

I should have time to test it in about two weeks. Am abroad right now. But I'll post my results here.

[Vladonai]

But I'll post my results here.

That would be a good thing. I currently have the following results on two Tesla P40s (model miqu 70B Q4_K_M):
Context window is 8k, empty prompt - text generation 8 tokens per second.
8к prompt - prompt processing 85 seconds, text generation 3,3 tokens per second.

I'm wondering if adding the RTX3090 to the circuit will help improve these results and if I can get an acceptable generation rate on this model with a window filled with 16k context tokens.

[askmyteapot]

Download and install the latest version of drivers for RTX3090 (Studio edition)

This is the only step you need to do.

I run a 3060ti + P40 using just the studio driver.

I even have the P40 working for gaming (but you need to fiddle with a couple of registry settings to get it to work for that, listed below)

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class{4d36e968-e325-11ce-bfc1-08002be10318},
find where your tesla card is, change “AdapterType” value to “1” and “FeatureScore” value from “cf” to “d1”,
create a new dword key which name is “GridLicensedFeatures”, and set its value to “7”. Disable then enable tesla in the device manager.
This step changs the gefore driver into grid driver，and the tesla is in wddm mode now.
change the “EnableMsHybrid” values to “1” in registry, where the tesla card is. Disable then enable tesla in the device manager again.
Set tesla as default high performance card following the step above.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.