what's the relationship between cudart-llama-bin-win-cuda-12.4-x64.zip and llama-b6123-bin-win-cuda-12.4-x64.zip ?

[MaoJianwei]

I want to run llama.cpp on my GPU with CUDA support.

I guess I should use llama-b6123-bin-win-cuda-12.4-x64.zip

But I found there is one additional package: cudart-llama-bin-win-cuda-12.4-x64.zip

What's the relationship between them? How to use cudart-llama-bin-win-cuda-12.4-x64.zip?

Thanks :)

[Rotatingxenomorph]

The cudart zip contains .dll files the cuda version needs. Extract them to join the rest of the files in the llama folder.

[MaoJianwei]

The cudart zip contains .dll files the cuda version needs. Extract them to join the rest of the files in the llama folder.

Thanks, but how to disable CPU backend? @Rotatingxenomorph
I want to run inference with GPU only, but still get something about CPU backend:

load_tensors: loading model tensors, this can take a while... (mmap = false)
load_tensors: offloading 28 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 29/29 layers to GPU
load_tensors:          CPU model buffer size =    83.46 MiB
load_tensors:        CUDA0 model buffer size =   372.65 MiB
..........................................................

llama_context:  CUDA_Host  output buffer size =     0.58 MiB
llama_kv_cache_unified:      CUDA0 KV buffer size =   448.00 MiB
llama_kv_cache_unified: size =  448.00 MiB (  4096 cells,  28 layers,  1/1 seqs), K (f16):  224.00 MiB, V (f16):  224.00 MiB
llama_context:      CUDA0 compute buffer size =   298.75 MiB
llama_context:  CUDA_Host compute buffer size =    14.01 MiB
llama_context: graph nodes  = 1098
llama_context: graph splits = 2

[Rotatingxenomorph]

I believe llamacpp won't run at all without the CPU backend.

[slaren]

You can use -ot =CUDA0 if you really want everything on the GPU, but it won't improve performance and will not work with some quants that are not fully supported by the CUDA backend.

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[MaoJianwei]

You can use -ot =CUDA0 if you really want everything on the GPU, but it won't improve performance and will not work with some quants that are not fully supported by the CUDA backend.

Thanks!

[MaoJianwei]

But this parameter doesn't exist? @slaren

error: invalid argument: -ot=CUDA0

[abc-nix]

But this parameter doesn't exist? @slaren

error: invalid argument: -ot=CUDA0

slaren correctly wrote it with a space after the -ot option, looking like this: -ot =CUDA0.

Or, if you want it with a bit more regex, -ot .*=CUDA0 should also work.

[MaoJianwei]

But this parameter doesn't exist? @slaren
error: invalid argument: -ot=CUDA0

slaren correctly wrote it with a space after the -ot option, looking like this: -ot =CUDA0.

Or, if you want it with a bit more regex, -ot .*=CUDA0 should also work.

I see. Here it is:

--override-tensor, -ot <tensor name pattern>=<buffer type>,...
                                        override tensor buffer type

Many thanks!