Multiple AMD GPUs with Tensorflow and OpenCL on Ubuntu 16.04

[gianks]

Hi, i'm using TF with openCL with amdgpu 17.50 (headless, opencl=legacy,pal), built today from your repository, eiger_sycl brach, on Ubuntu 16.04.
TF runs but sees and uses just videocard 0, while computecpp_info and clinfo report all the 5 GPUs i have in the system, all the same model (amd rx580).

Why is TF/SYCL failing in using all of them? How can i debug this problem?
I also asked a question about this on stackoverflow, you can look here for more details:
https://stackoverflow.com/questions/52864399/multiple-amd-gpus-with-tensorflow-and-opencl-on-ubuntu-16-04

Thanks

[Rbiessy]

Good to know that it built.
Unfortunately we don't support multiple GPUs at the moment. It would be a really good feature to have but it is not a priority for now. If all your devices reported by clinfo are the same and you want to use them all, I think we would just need to register all SYCL-compatible devices instead of just the first one.
In the more general case, if the SYCL devices are different, the user has no way to know which SYCL device is which on the python side. As far as I know the user would only see "SYCL:0" and "SYCL:1" so we're limited by that.
I'll have a closer look at the simpler case to see if it really is as easy as I think it is.

[gianks]

@Rbiessy, Sorry i didn't notice you were already looking into it :)

I could test with no problems on my machines if you want to check how it turns out.

In the end identifying the device would be a minor problem since as far as i recorded SYCL0 corresponds to the first available bus number GPU, so i would guess SYCL 1,2, etc would follow the same ORDER.

Can you point me to the right part of the code which should be edited to give it a try?
(As i said assuming all equal video cards are present on the machine).

Thanks a lot!

[Rbiessy]

Sure so you want to look at https://github.com/codeplaysoftware/tensorflow/blob/eigen_sycl/tensorflow/core/common_runtime/sycl/sycl_device.h.
This is related to the TODO line 210 where we basically want a for loop and some attributes should probably become vectors. It is useful to see how CUDA does it (in the common_runtime/gpu folder) but they have a lot more going on in here than we do.
You'll also need to change one compile flag so that get_sycl_supported_devices() returns a list of all the devices. You can either remove the line -DUSE_DEFAULT_SELECTOR=1 in tools/bazel.rc or add --copt=-UUSE_DEFAULT_SELECTOR to your command line.
I won't have time to work on it right now but you're more than welcome to do a pull request even if it's a WIP!

[gianks]

Hi, i have no background on this code but i'll give it a try, maybe tomorrow already.
Anyway, can you give a look meanwhile if there are other TODOs to be completed elsewhere related to this, when is possible for you?

Thanks a lot, see you later!

Edit:
Would be nice if you could help me figure out where i shall take the gpu_id (currently fixed to 0), since i don't know what that has to indicate (I guess that is the index will be used with devices:SYCL:X, right?) and if it has to follow some rules (as maybe assumed somewhere in other classes), like continuity.
I sadly don't have a lot of time to inspect the code tree and a little of help to get started would help a lot if possible, at least to avoid to mess things up and have then to spend a lot of time in debugging stupid logical mistakes i made, given the project knowledge i frankly don't have.

Thanks

[Rbiessy]

Hi, we don't have any other TODO related to this.
Yes the gpu_id is what you described. It should just be an increasing index for all SYCL devices.
I will need some time to look a bit more at the CUDA code too. I'll keep you updated if I have something interesting to try.

[gianks]

Hi, now SYCL 1,2 and 3 are available!
Remains to understand why the graph still relies on just SYCL0 running the testing code provided by the installation guide of the convolutional image (MNIST: models/tutorials/images/mnist/convolutional.py), see below:

2018-10-20 19:53:13.051077: I ./tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2018-10-20 19:53:13.051222: W ./tensorflow/core/common_runtime/device_factory.cc:110] PROCESSING
2018-10-20 19:53:13.380001: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:90] Found following OpenCL devices:
2018-10-20 19:53:13.380036: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:92] id: 0, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
2018-10-20 19:53:13.380045: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:92] id: 1, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
2018-10-20 19:53:13.380052: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:92] id: 2, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
2018-10-20 19:53:13.380058: I ./tensorflow/core/common_runtime/sycl/sycl_device.h:92] id: 3, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
2018-10-20 19:53:13.380092: W ./tensorflow/core/common_runtime/device_factory.cc:110] PROCESSING
2018-10-20 19:53:13.380103: W ./tensorflow/core/common_runtime/device_mgr.cc:29] 5
Device mapping:
/job:localhost/replica:0/task:0/device:SYCL:0 -> id: 0, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:1 -> id: 1, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:2 -> id: 2, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:3 -> id: 3, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
2018-10-20 19:53:13.380294: I ./tensorflow/core/common_runtime/direct_session.cc:288] Device mapping:
/job:localhost/replica:0/task:0/device:SYCL:0 -> id: 0, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:1 -> id: 1, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:2 -> id: 2, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE
/job:localhost/replica:0/task:0/device:SYCL:3 -> id: 3, type: GPU, name: Ellesmere, vendor: Advanced Micro Devices, Inc., profile: FULL_PROFILE

truncated_normal/TruncatedNormal: (TruncatedNormal): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384000: I ./tensorflow/core/common_runtime/placer.cc:886] truncated_normal/TruncatedNormal: (TruncatedNormal)/job:localhost/replica:0/task:0/device:SYCL:0
truncated_normal/mul: (Mul): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384024: I ./tensorflow/core/common_runtime/placer.cc:886] truncated_normal/mul: (Mul)/job:localhost/replica:0/task:0/device:SYCL:0
truncated_normal: (Add): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384034: I ./tensorflow/core/common_runtime/placer.cc:886] truncated_normal: (Add)/job:localhost/replica:0/task:0/device:SYCL:0
Variable: (VariableV2): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384042: I ./tensorflow/core/common_runtime/placer.cc:886] Variable: (VariableV2)/job:localhost/replica:0/task:0/device:SYCL:0
Variable/Assign: (Assign): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384049: I ./tensorflow/core/common_runtime/placer.cc:886] Variable/Assign: (Assign)/job:localhost/replica:0/task:0/device:SYCL:0
Variable/read: (Identity): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384056: I ./tensorflow/core/common_runtime/placer.cc:886] Variable/read: (Identity)/job:localhost/replica:0/task:0/device:SYCL:0
Variable_1: (VariableV2): /job:localhost/replica:0/task:0/device:SYCL:0
2018-10-20 19:53:13.384065: I ./tensorflow/core/common_runtime/placer.cc:886] Variable_1: (VariableV2)/job:localhost/replica:0/task:0/device:SYCL:0
Variable_1/Assign: (Assign): /job:localhost/replica:0/task:0/device:SYCL:0

I will now trying writing a snippet to test if i can manually run some op on all the cards but i think we shall verify also placer.cc (but i don't know if i shall expect convolutional.py to really employ all the cards by default at this point).

Please let me know.
Thanks

[Rbiessy]

Hello, that's great to hear!
I just had a look at the python script you mentioned and indeed it doesn't look like it can use multiple devices. According to this guide you have to manually select which operations are run on which device. I understand this is not very practical but I believe this could be implemented with rather little changes, maybe you are already close to having this working. I don't think anything needs to be changed in placer.cc but it may be useful to look into it when debugging.

What you really want though is the stream executor. There is surprisingly very little official documentation about this. The implementation is in the stream_executor folder which we haven't touched at all. Unfortunately supporting this would be a major task that is not a priority right now.

In the end, if you plan to write your own model or adapt existing models for your machine specifically, the first feature is enough.

EDIT: After reading a bit more it seems the stream executor is only available in the C++ API which would explain the lack of documentation.
Here are more links on the subject of manually placing operations:

• More detailed guide on using multiple GPUs: https://jhui.github.io/2017/03/07/TensorFlow-GPU/
• Complete example training CIFAR-10 on multiple GPUs: https://www.tensorflow.org/tutorials/images/deep_cnn

[gianks]

Hi, sorry i didn't post you additional details but i can tell you that with a little hack in keras code i have been able to use multiple gpus (see multiple_gpu_model)!
Sadly as far as i got i cannot really take advantage of all of them for real, in opposition using one gpu performs way better than two, which in turn is better than 4 and so on.

Anyway all the SYCL devices CAN be engaged but performance is poor.
I'll involve the keras team with a ticket about the performances to understand where is the bottleneck here, i can't believe two are worse than one with a large NN and large batches, i don't know if the problem relies in the available bandwidth for the flow of data from/to the GPUs memory to load each batch.

I can notice high kernel times while training, is that expected?
Thank you!

[Rbiessy]

Oh I see you are going through Keras.
I can imagine that moving the data around can be a bottleneck but that's not specific to SYCL.
We are still optimising our kernels, kernels time are expected to be higher than using CUDA but still faster than using CPU. If you can share with us any profiling data that would show the kernel times that would be really helpful! We could find out if a kernel is unexpectedly slow.

Also if you think your changes to TensorFlow could be useful to anyone else using SYCL could you consider doing a pull request to the experimental branch?
Thank you.

[gianks]

Hi, sure i will, i wish i could arrive to something really working before that :)

I was observing that it takes a long time to start the first epoch and a massive amount of memory, expecially when you use multiple gpus with replicated models.
Is this normal? I've also noticed that while Dense and Dropout work, as soon as i involve a convolutional layer you need at least a TB of RAM on the host and the GPU to make it ( I overall consumed 28 GB for a single 1D layer preparation with 10000 features and a filter of 60 elements and it didn't fit in the 4 Gb of the RX580) i'm using. Moreover it has been stuck for more than 40 minutes with an i3 2.8Ghz,, while with multiple huge Dense layers the start time is around 8 minutes, multiplied the number of cards).

The same code on CUDA works employing less than 3 Gb on the host and starts almost immediately.

Is SYCL compiling or transcoding stuff at the beginning? If true, is possible the process is repeated per each card even if the model is exactly the same?
In case i guess it would be possible to optimize the timings reusing the prepared code to run on the GPUs.
About the general performance when training, I read someone on stackoverflow talking about huge amount of data going back and forth from/to CPU and GPU caused by the lack of optimized opencl kernles and others, is this correct? Is possibile there are still some bottleneck in SYCL due to the need to reinterpret things or to do operations in an non effective way due to not yet specialized code has been released?

Thanks!

[Rbiessy]

Hi,
Hmm I don't see why SYCL would need more memory than CUDA on the host nor on the device. As far I know, the way to deal with huge dataset in TensorFlow is to split the data before hand so that would work the same for any backend.
SYCL, as OpenCL, is indeed compiling the kernels at an early stage. Also as you pointed out, with the way multiple devices are handled in your version, the kernels will be compiled multiple times. This is the only stage where I would expect SYCL to need more RAM.
About the general performance, there is a copy back and forth to the CPU only if there is no SYCL implementation for an operation that is placed between two other operations running on a SYCL device. This is the same for CUDA and we already implement most of the operations CUDA has. You can check if this is an issue by enabling the log device placement in TensorFlow.

[mirh]

Is whatever the new "accelerator_selector" is, related to this?

[DuncanMcBain]

The accelerator selector will match against OpenCL devices that correspond to CL_DEVICE_TYPE_ACCELERATOR from this page.