ENV var for P2P enabled GPUs - GGML_CUDA_P2P_DEVICES

[vektorprime]

Right now the only env var I see for enabling P2P is GGML_CUDA_P2P, but I have 2 GPUs that do and 2 GPUs that do not support P2P. It would be really nice if we can just specify which GPUs support P2P so that llamacpp does not crash in my scenario.

Example:
GGML_CUDA_P2P_DEVICES=0,1
where GPU 0 and 1 do have P2P enabled
and all my GPUs would be
CUDA_VISIBLE_DEVICES=0,1,2,3

[vektorprime]

My issue was more that the P2P drivers I was using caused my 2x 3080 to attempt P2P and crash whereas the 3090s were working fine. I had an LLM just patch the P2P drivers so that they don't advertise P2P on non-rebar GPUs.

[NumericalAnalysis]

Just asking about GGML_CUDA_P2P_DEVICES=0,1
Is this supposed to be set with the “export” command before running llama-server? Thanks

[vektorprime]

Just asking about GGML_CUDA_P2P_DEVICES=0,1 Is this supposed to be set with the “export” command before running llama-server? Thanks

It's not a real value, this was a suggestion.

[NumericalAnalysis]

Thank you. Two weeks ago, I was wondering whether llama.cpp on my server was using NCCL. Later, I used nsys and could clearly see the time NCCL was being used. Now I'm pretty sure my machine is using NCCL.