TF 1.13.1 GPU (CUDA 10.0 cuDNN 7.4) without AVX, Ubuntu 16.04, Python 3.5 [from official docker image]

[saippuakauppias]

Builded from official docker devel-gpu-py3 image:

# pull docker image
docker pull tensorflow/tensorflow:devel-gpu-py3

# run docker
docker run --runtime=nvidia -it  -v ~/projects/tf:/my-devel tensorflow/tensorflow:devel-gpu-py3 bash

# fix bazel version (need 0.21)
rm -rf /usr/local/bin/bazel
wget -O /bazel/installer.sh "https://github.com/bazelbuild/bazel/releases/download/0.21.0/bazel-0.21.0-installer-linux-x86_64.sh"
chmod +x /bazel/installer.sh
/bazel/installer.sh

# go to TF folder
cd /tensorflow_src
git checkout v1.13.1

# set some variables
# see full in Dockerfile: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/dockerfiles/dockerfiles/devel-gpu.Dockerfile
export TF_ENABLE_XLA=1
export CC_OPT_FLAGS="-march=native -mno-avx"


./configure

# run compilation
bazel build --config=opt --config=cuda --noincompatible_strict_action_env //tensorflow/tools/pip_package:build_pip_package

# get wheel
bazel-bin/tensorflow/tools/pip_package/build_pip_package /my-devel/tensorflow_pkg1

Builded wheel: https://drive.google.com/file/d/198py006N2mH635O99-lnDPWeDbr8LjTt/view?usp=sharing

[edumucelli]

Just for information, it requires CUDA 10.0.130, as TF matches CUDA versions straightforwardly it won't work on CUDA 10.1, e.g., the one on Debian Bullseye (Testing). Error will be the famous mismatch between versions:

ImportError: libcublas.so.10.0: cannot open shared object file: No such file or directory

[saippuakauppias]

@edumucelli can you check?

docker pull tensorflow/tensorflow:1.13.1-gpu-py3
docker run --gpus all -it  -v ~/projects/tf:/tf tensorflow/tensorflow:1.13.1-gpu-py3 bash

# ~/projects/tf in local machine (not docker) need contains my tf build

pip uninstall tensorflow-gpu tensorflow

cd /tf
pip install tensorflow-1.13.1-cp35-cp35m-linux_x86_64.whl

Its worked without ImportError?

[edumucelli]

@saippuakauppias it works without ImportError if you have the exact same CUDA version than the one compiled in this docker tag (lines 9 and 10) for the docker tag you mentioned:

ENV CUDA_VERSION=10.0.130
ENV CUDA_PKG_VERSION=10-0=10.0.130-1

Otherwise if, as me, you are using Debian Bullseye (Testing) it won't work as its nvidia-cuda-toolkit package comes with CUDA 10.1. Given the state of the whole Tensorflow/CUDA/CuDnn compatibility mess it is hard it is to put together Tensorflow as we have to have the precise version of CUDA compilation or it won't work, e.g., #26289, the building from source rabbit hole #26155. Even the corresponding compatibility tables do not give the precise minor version, but it won't work if you do not match up to the patch version for CUDA and CuDnn.