Thanks so much to Nagadit for hosting this repository. Here is my problem and hopefully it helps someone else:
Expected behaviour
Training SAEHD or XSeg on DFL with RTX 3090, tensorflow 2.4.0
Actual behaviour
Python throws Error code of "illegal instruction, core dumped" on last line of DFL script (which says "train")
This is despite Tensorflow 2.4.0 correctly recognising the RTX 3090, and despite cuda 11.0 or 11.1 and compatible nvidia drivers (455.28) all working correctly.
You can check if Tensorflow is recognising your GPU by typing
python
from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())
Steps to reproduce
Install DFL on Linux as per Nagadit repository but use python 3.8 instead, and cudnn 8.0.5 and cudatoolkit 11.0 from Conda or 11.1 from Nvidia direct. Use latest requirements-cuda.txt since some newer versions of some modules are required (like h5py==2.10.0 or pyqt5==5.15.2)
Tensorflow 2.4.0
Use latest files from iperov Github.
Solution:
This will only apply to some people out there with older CPUs, but here is what I eventually found:
This is a Tensorflow 2.4.0 problem. Even if RTX 3090 works with TF 2.4.0, older CPUs do not. TF requires AVX or AVX2 support. TF 2.3 and 2.2 support AVX and AVX2. The tensorflow guys forgot to include AVX support in 2.4.0, despite it being compatible! Newer CPUs with AVX2 support will be ok with TF 2.4.0.
I therefore compiled my own tensorflow for my machine, and this produced TF 2.5.0 which had AVX support. I can now fully train DFL using RTX 3090!
To compile your own TF you can use: https://www.tensorflow.org/install/source
I compiled with cudnn 8.0.5 dev files (filename libcudnn8-dev_8.0.5.39-1+cuda11.1) and cudatoolkit 11.1 installed.
Compiling TF requires Bazel. To install bazelisk, can use:
$ wget https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64
$ chmod +x bazelisk-linux-amd64
$ sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel
$ bazel version
Issuing the command:
$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
This produced tensorflow 2.5.0 and this works great with RTX3090 and current DFL build. I can train SAEHD now.
Don't think this problem is common but hopefully of some use to someone out there
Thanks so much to Nagadit for hosting this repository. Here is my problem and hopefully it helps someone else:
Expected behaviour
Training SAEHD or XSeg on DFL with RTX 3090, tensorflow 2.4.0
Actual behaviour
Python throws Error code of "illegal instruction, core dumped" on last line of DFL script (which says "train")
This is despite Tensorflow 2.4.0 correctly recognising the RTX 3090, and despite cuda 11.0 or 11.1 and compatible nvidia drivers (455.28) all working correctly.
You can check if Tensorflow is recognising your GPU by typing
python
Steps to reproduce
Install DFL on Linux as per Nagadit repository but use python 3.8 instead, and cudnn 8.0.5 and cudatoolkit 11.0 from Conda or 11.1 from Nvidia direct. Use latest requirements-cuda.txt since some newer versions of some modules are required (like h5py==2.10.0 or pyqt5==5.15.2)
Tensorflow 2.4.0
Use latest files from iperov Github.
Solution:
This will only apply to some people out there with older CPUs, but here is what I eventually found:
This is a Tensorflow 2.4.0 problem. Even if RTX 3090 works with TF 2.4.0, older CPUs do not. TF requires AVX or AVX2 support. TF 2.3 and 2.2 support AVX and AVX2. The tensorflow guys forgot to include AVX support in 2.4.0, despite it being compatible! Newer CPUs with AVX2 support will be ok with TF 2.4.0.
I therefore compiled my own tensorflow for my machine, and this produced TF 2.5.0 which had AVX support. I can now fully train DFL using RTX 3090!
To compile your own TF you can use: https://www.tensorflow.org/install/source
I compiled with cudnn 8.0.5 dev files (filename libcudnn8-dev_8.0.5.39-1+cuda11.1) and cudatoolkit 11.1 installed.
Compiling TF requires Bazel. To install bazelisk, can use:
$ wget https://github.com/bazelbuild/bazelisk/releases/download/v1.7.4/bazelisk-linux-amd64
$ chmod +x bazelisk-linux-amd64
$ sudo mv bazelisk-linux-amd64 /usr/local/bin/bazel
$ bazel version
Issuing the command:
$ ./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg
This produced tensorflow 2.5.0 and this works great with RTX3090 and current DFL build. I can train SAEHD now.
Don't think this problem is common but hopefully of some use to someone out there