Lesser number of qubits in distributed circuit with four GPUs using cuquantum and cusvaer than on single CPU with qiskit

[intelligi123]

I am using a VM with allocation of four A40 GPUs and installed cuquantum appliance. I ran docker with all GPUs and didn't give any memory argument. I ran ghz_cusvaer.py which shows max qubits of 31 for single GPU and when I ran with mpirun giving -n 4 it max generate circuit with 33 qubits and max memory usage was not more than 4.7G even all GPUs were 100 percent utilized.

Same circuit when ran out of the docker, easily generated ghz state with qubits of around 10k till I have tested and didn't stuck at all.

Stats of docker while running circuit to max level

-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.199.02   Driver Version: 470.199.02   CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA A40          Off  | 00000000:03:00.0 Off |                  Off |
|  0%   34C    P0    99W / 300W |  33542MiB / 48685MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA A40          Off  | 00000000:03:01.0 Off |                    0 |
|  0%   34C    P0   104W / 300W |  33541MiB / 45634MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA A40          Off  | 00000000:03:02.0 Off |                    0 |
|  0%   34C    P0   102W / 300W |  33541MiB / 45634MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA A40          Off  | 00000000:03:03.0 Off |                    0 |
|  0%   34C    P0    95W / 300W |  33541MiB / 45634MiB |    100%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

And output of docker stats command showed

CONTAINER ID   NAME           CPU %     MEM USAGE / LIMIT     MEM %     NET I/O       BLOCK I/O         PIDS
3e6d40789e1f   dreamy_ellis   392.72%   4.548GiB / 442.7GiB   1.03%     7.91kB / 0B   11.5MB / 51.7MB   86

My question is why cuquantum cusvaer is not handling more than 33 qubits which I think for this circuit is quite less since its not using much of entanglement.

I need to run circuit with almost 1000 qubits by varying its entanglement, however this much output from cuquantum appliance is not looking encouraging.

Should I allocate more resources to docker manually?

[mtjrider]

Hi @intelligi123

The cuQuantum Appliance with the cusvaer backend implements a numerically dense representation of the state. It does not provide or implement entanglement-based compression, and it doesn't use system memory (RAM). Can you share more about your target workload / circuits?

Below, I detail how the cuQuantum Appliance (cusvaer) determines system size constraints based on available GPU memory.

With complex-valued single precision, an $n$ qubit state vector will require $2^{n+3} \mathrm{B}$. Converting to MiB is done by dividing the result by $1024^{2} = (2^{10})^{2} = 2^{20}$ -- this yields a clean expression for required memory in Mebibytes: $2^{n - 17} \mathrm{MiB}$.

For double precision, the expression becomes $2^{n - 16} \mathrm{MiB}$.

num. of qubits	single precision	double precision
30	8,192 MiB (8,589.93 MB)	16,384 MiB (17,179.87 MB)
31	16,384 MiB (17,179.87 MB)	32,786 MiB (34,359.74 MB)
32	32,786 MiB (34,359.74 MB)	65,536 MiB (68,719.48 MB)
33	65,536 MiB (68,719.48 MB)	131,072 MiB (137,438.95 MB)
34	131,072 MiB (137,438.95 MB)	262,144 MiB (274,877.91 MB)

A system with 4x A40 will have 194,740 MiB (204,199.69 MB) when ECC is off. When ECC is on, the same system will have 183,105.6 MiB (192,000 MB).

I'll make two observations:

1. The largest state your GPU system can store corresponds with a $n = 34$ qubits (single precision).
2. Your GPUs appear to be in different ECC configuration states (off for GPU 0, on for all others).

(2) may confuse the cusvaer backend -- it assumes a homogeneous pool of GPUs. You mention you can't simulate more than 33 qubits. Can you clarify the operating precision for the state (single, double)?

[mtjrider]

@intelligi123
I've marked my response as the answer.
It clarifies the maximum number of qubits you can simulate on your system under a numerically dense state vector ansatz.

If this wasn't your question, let us know.