Does it make sense to buy a powerful graphics card to speed up generation?

[Vladonai]

I currently have an RTX 3050 and the latest releases of koboldcpp have really speed up the prompt processing. Obviously, a more powerful graphics card will speed up this process even more. But what about generation? I might buy an RTX 4090 if it would make the token generation rate significantly faster, but I suspect that won't happen. Can't CUDA and RTX cores be plugged in for at least some computation to speed up generation? Because 70B models on a CPU would be very slow...

[LostRuins]

Generation speed will likely be quite fast if you can do a full offload to GPU, so having more vram helps in that aspect. You can compare yourself by trying a full offload on LLAMA 3B, which you card should be able to do. Compare it with offloading half the layers and feel the speed difference.

[Vladonai]

There is an Exllama project that processes a quantized model using only a graphics card. In this project, the entire model is loaded into video memory and the RTX 4090 is enough for the quantized 33B model. But that's the limit. Koboldcpp shares RAM and VRAM; is it possible to use a graphics card for processing in this case? Instead of the processor or in conjunction with it.

[LostRuins]

For CUDA the GPU already does most of the heavy lifting. exllama will probably still be faster.

[Vladonai]

During prompt processing, the graphics card is engaged, but it rests during generation. The Llama2 70B model on my configuration generates ~1 token per second; this is an amazing result, an optimization miracle. But using 16k CUDA cores, the same model generates 12-17 tokens per second, if exllama results are to be believed. However, this requires two graphics cards with 24GB VRAM each. If it would be possible to make do with a single RTX 4090 to achieve at least half of this result, I would probably buy this card.

[LostRuins]

The latest version does support controlling where the tensors go during offloading to multiple GPUs for cuda, with --gpulayers via --tensor_split. You can try that (all GPU must be selected, dont specify single gpu)