cifar10 - multi GPU training

[sergeimonakhov]

hi, i have 5 card amd x470, i run python3 ./cifar10_multi_gpu_train.py --num_gpus=5, i get visible one card in x16 pci. How work other cards in x1 pci?

[whchung]

@D1abloRUS please refer to https://github.com/RadeonOpenCompute/ROCm#supported-cpus . RX470 is of GFX8 family so we don't support them on x1 PCIe yet.

[sergeimonakhov]

@whchung hmm ok. What about gfx7xx? 2018-09-12 17:16:59.854955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1532] Ignoring visible gpu device (device: 2, name: Hawaii XT [Radeon R9 290X], pci bus id: 0000:03:00.0) with AMDGPU ISA gfx701. The minimum required AMDGPU ISA is gfx803. how run him?

[whchung]

Unfortunately Hawaii (GFX7) family is not in the roadmap. Quite a few DNN algorithms in MIOpen are implemented in GFX-specific assembly, so we are only focused on GFX8/GFX9 and upcoming architectures.

[sergeimonakhov]

@whchung

RX470 is of GFX8 family so we don't support them on x1 PCIe yet.

what about x8?

[whchung]

x8 should work. please check:
https://rocm.github.io/hardware.html

[dagamayank]

/cc @jlgreathouse to confirm supported hw list.

[jlgreathouse]

Hi @D1abloRUS

When you say "x8", "x1", etc. the major thing to ask is how are these GPUs connected to your CPU? In particular, gfx8 GPUs require PCIe Gen 3 atomics at every step between the CPU and the GPUs. Many people running multiple GPUs through x1 lanes are using PCIe switches to split off multiple ports from a single port. One of the major impediments here is that your PCIe switches must know how to properly forward PCIe atomic commands.

Note that this is true for "x1" or "x8". So if your "PCIe x8" solution also has a switch in between your CPU and your GPU(s), you will also need to make sure this switch properly handles atomics.

Towards that end, I'll ask:

• What CPU are you using?
• What motherboard are you using?
• How are you connecting your GPUs to that motherboard?
  • In other words, if your motherboard has X PCIe slots, which slot is each of your GPUs connected to, and how?

Thanks.

[sunway513]

Closing this ticket as there're no more feedbacks.
@D1abloRUS feel free to reopen it if you have further questions.