How do I use the GPU?

[deversbpv]

I doing something wrong. When I run the model, the GPU is not really used (only 600MB). But RAM isn't uses to. The CPU has load and it seems, that there is a lot of IO-Traffic on the SSD.

I see this in the log:

llm_load_tensors: using CUDA for GPU acceleration
llm_load_tensors: mem required  = 3647,96 MB
llm_load_tensors: offloading 0 repeating layers to GPU
llm_load_tensors: offloaded 0/35 layers to GPU
llm_load_tensors: VRAM used: 0,00 MB

Can someone explain?

[Azeirah]

By default if you compiled with GPU support some calculations will be offloaded to the GPU during inference. If you want the real speedups, you will need to offload layers onto the gpu. This means that you can choose how many layers run on CPU and how many run on GPU.

llm_load_tensors: offloaded 0/35 layers to GPU

This is what I'm talking about.

You can specify how many layers you want to offload to the GPU using the -ngl parameter. The more you offload, the more VRAM is used and generally the faster your inference will become.

Offload all:

./main -m ... -ngl 35

Offload some

./main -m ... -ngl 10

etc

[ichsan2895]

Hello, how to use more than 35/35 GPU layers? It only use 5 GB VRAM. Since I has higher VRAM (>12 GB) and I want maximize all of my VRAM. How to do that?

[Galunid]

You increase 35 to something bigger, e.g. 50. Of course your model needs to have that many layers. If it doesn't you can look into running a bigger one.

[deversbpv]

Thank you. I got it. It works.