AMD (8GB) vs NVIDIA (6GB) - direct comparison - VRAM Problems

[Chilluminati91]

I've been getting questionable results from my AMD card, so I decided to also test my older NVIDIA card to compare the performance.

Test system:

Ubuntu 22.04 LTS
Intel i5-12600k
32GB DDR4
AMD Radeon RX 6650 XT (8GB)
NVidia Geforce GTX 1060 (6GB)
Pytorch 2.0.1

The repo is installed once and I use two different venvs to run either pytorch cuda or pytorch rocm.

Command line (AMD): --opt-sdp-attention
Command line (NVidia): --xformers

---

Inference Speeds:

512x512 - Euler a - 30 steps - 3 runs - a woman riding a horse through a forest

AMD RX 6650 XT:

100%|███████████████████████████████████████████| 30/30 [00:06<00:00,  4.73it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:06<00:00,  4.83it/s]
100%|███████████████████████████████████████████| 30/30 [00:06<00:00,  4.67it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:06<00:00,  4.68it/s]
100%|███████████████████████████████████████████| 30/30 [00:06<00:00,  4.64it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:06<00:00,  4.63it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:06<00:00,  4.58it/s]

NVIDIA GTX 1060

100%|███████████████████████████████████████████| 30/30 [00:20<00:00,  1.43it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:20<00:00,  1.47it/s]
100%|███████████████████████████████████████████| 30/30 [00:21<00:00,  1.42it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:21<00:00,  1.41it/s]
100%|███████████████████████████████████████████| 30/30 [00:22<00:00,  1.32it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:22<00:00,  1.31it/s]
Total progress: 100%|███████████████████████████| 30/30 [00:22<00:00,  1.29it/s]

-> No surprise here, the newer card is running at 3x speed. On par with expectations of AMD + rocm.

---

Upscaling:

Same settings as above - Hi-Res Fix - Latent - Steps 0 - Strength 0.5

AMD RX 6650 XT

1.2x:
100%|█████████████████████████████████████████████████████████████████| 30/30 [00:06<00:00,  4.68it/s]
100%|█████████████████████████████████████████████████████████████████| 30/30 [00:11<00:00,  2.72it/s]
Total progress: 100%|█████████████████████████████████████████████████| 60/60 [00:17<00:00,  3.37it/s]
Total progress: 100%|█████████████████████████████████████████████████| 60/60 [00:17<00:00,  2.71it/s]

1.3x:
100%|█████████████████████████████████████████████████████████████████| 30/30 [00:06<00:00,  4.81it/s]
100%|█████████████████████████████████████████████████████████████████| 30/30 [00:13<00:00,  2.20it/s]
Total progress: 100%|█████████████████████████████████████████████████| 60/60 [00:20<00:00,  2.96it/s]
Total progress: 100%|█████████████████████████████████████████████████| 60/60 [00:20<00:00,  2.19it/s]

1.4x:
torch.cuda.OutOfMemoryError: HIP out of memory. Tried to allocate 1.87 GiB (GPU 0; 7.98 GiB total capacity; 4.02 GiB already allocated; 1.98 GiB free; 5.88 GiB reserved in total by PyTorch) 
If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.

NVIDIA GTX 1060:

1.2x:
100%|███████████████████████████████████████████| 30/30 [00:20<00:00,  1.49it/s]
100%|███████████████████████████████████████████| 30/30 [00:32<00:00,  1.07s/it]
Total progress: 100%|███████████████████████████| 60/60 [01:06<00:00,  1.11s/it]
Total progress: 100%|███████████████████████████| 60/60 [01:06<00:00,  1.08s/it]

1.3x:
100%|███████████████████████████████████████████| 30/30 [00:20<00:00,  1.47it/s]
100%|███████████████████████████████████████████| 30/30 [00:44<00:00,  1.50s/it]
Total progress: 100%|███████████████████████████| 60/60 [01:06<00:00,  1.12s/it]
Total progress: 100%|███████████████████████████| 60/60 [01:06<00:00,  1.55s/it]

2.0x:
100%|███████████████████████████████████████████| 30/30 [00:20<00:00,  1.48it/s]
100%|███████████████████████████████████████████| 30/30 [02:34<00:00,  5.15s/it]
Total progress: 100%|███████████████████████████| 60/60 [02:58<00:00,  2.98s/it]
Total progress: 100%|███████████████████████████| 60/60 [02:58<00:00,  5.22s/it]

2.5x:
100%|███████████████████████████████████████████| 30/30 [00:21<00:00,  1.42it/s]
100%|███████████████████████████████████████████| 30/30 [05:31<00:00, 11.03s/it]
Total progress: 100%|███████████████████████████| 60/60 [05:59<00:00,  5.98s/it]
Total progress: 100%|███████████████████████████| 60/60 [05:59<00:00, 11.06s/it]

3.0x:
100%|███████████████████████████████████████████| 30/30 [00:20<00:00,  1.48it/s]
100%|███████████████████████████████████████████| 30/30 [10:21<00:00, 20.73s/it]
Total progress: 100%|███████████████████████████| 60/60 [10:53<00:00, 10.89s/it]
Total progress: 100%|███████████████████████████| 60/60 [10:53<00:00, 20.73s/it]

No further tests due to time consumption.

-> With +2GB VRAM on the AMD gpu this should not happen. We can see that the NVIDIA lacks speed in comparison but is able to pump out higher resolutions without a problem. This points to a huge problem with AMD VRAM management.

I did a few more tests with --medvram on the AMD card and you can push the limit up to 1.4x with slower speeds, but even then it is capped there.

---

Do you guys on AMD experience the same behavior?

[Soulreaver90]

Yup, always had weird memory allocation issues with AMD. I added the following command to my launch args and noticed vram usage imporoved significantly.
export PYTORCH_CUDA_ALLOC_CONF=garbage_collection_threshold:0.9,max_split_size_mb:32
You can tinker with max split size, some will use 128.
Retest and let me know, I am curious!

[elen07zz]

what is the difference between using 32 or 128?

[tablescarola]

not real!

[Sakura-Luna]

AMD GPUs usually consume more VRAM is a well-known thing, especially for gamers, can only expect AMD to improve in the future.

[elen07zz]

its just because xformers optimization, its way better