Is speed up using multiple GPUs on qsim-mgpu available?

[sungwooahn]

Hi, I'm using NVIDIA docker container 23.03 cuquantum appliance and trying to see the multi-gpu speed up.
I used frontend with cirq and backend with qsim-mgpu and tested pretty much all of the benchmark provided in the github with qubits ranging 10 to 30+.
However, I see performance degradation with multiple GPUs compared to using only single GPU.

I assume this is based on data communication time among GPUs, but I would like to see the performance improvement as stated in NVIDIA cuStatevec blog.

Can anyone help?

[leofang]

Hi @sungwooahn, thank you for reaching out. Would you provide us more information so that we can understand the situation better?

• What's your GPU configuration? This includes
  • The GPU models
  • The interconnect between GPUs, if any
• What's your driver version?
• What are the exact commands that you use for benchmarking? In particular,
  • You mentioned from 10 to "30+", but how large exactly it is?

Thanks!

[leofang]

(ps. I converted your issue to a discussion, since this is a more suitable place to start a Q&A.)

[sungwooahn]

Thanks for the reply.

My host and device settings are: Processor type: AMD EPYC 7543 32-Core Processor, GPU device name: NVIDIA RTX A6000
I have 2 EPYC processors and 4 A6000 devices on the server. I believe devices are just connected through pcie.
I tested qaoa, qft, ghz, simon for qubits from 10 to 32. From 33 qubits, single gpu cannot allocate memory.
Here are some outputs of the experiment.

The top 2 images show a single gpu runtime with 32 and 30 qubits in qft algorithm. I used qsim-mgpu backend.
And third images shows two gpu runtime with 28 qubits in same algorithm and backend. But, it took 16 minutes.
I assume the communication between gpu could be overhead in 28 qubit case since even single gpu can take all the data in the memory.
However, I think this result in two gpu case is so worse, so I want to know how I can get multiple gpus results are better in any condition than a single gpu experiment.

Thanks for the asking and discussion.

[mtjrider]

Can you post the output of nvidia-smi topo -m ?

Separately, can you post a text file containing the output of lspci -tv ?

[leofang]

I believe devices are just connected through pcie.

@sungwooahn If so this could explain the bad perf. Data exchange across pci-e is too slow. We expect multi-GPU interconnected via NVLinks would benefit. Matt's suggestions above would help you confirm the interconnects in use.

That said, using qsim-mgpu can still help you extend the max number of qubits reachable with your system, as you're no longer bound by the single-GPU memory.