tensorflow-gpu

[sathiez]

By default s2i wrapper uses tensorflow cpu, how to use tensorflow gpu?

[ukclivecox]

We don't have a particular example is showing how you can include the appropriate GPU version of the tensorflow libraries. However, did you not manage to get it working as discussed in #215

[sathiez]

Nope. I used seldonio/core-python-wrapper:0.7 wrapper before.

[ukclivecox]

You should be able to use the steps described in #215 to create a custom assemble script in s2i which will allow you to do any installations needed to install the appropriate libraries.

[sathiez]

how to set environment variable in s2i. If I set in .s2i/environment or in assemble script with "export" it is not reflecting in built image.

[sathiez]

Now environment variable is working fine but When I run tensorflow inside s2i image I get no NVIDIA GPU device is present: /dev/nvidia0 does not exist. while I can successfully able to build and run tensorflow in normal docker image using nvidia as runtime

[ukclivecox]

Are you sure you have access to the NVIDIA drivers from the container, e.g. for GKE: https://cloud.google.com/kubernetes-engine/docs/how-to/gpus#installing_drivers

[ukclivecox]

or for example: https://github.com/NVIDIA/nvidia-docker

[sathiez]

Actually when I run s2i built docker image with "- - runtime nvidia", nvidia drivers are not getting installed, but I can able to run other docker image with "runtime nvidia" successfully.
I hope seldon core wrapper built with debian stretch and my host os is ubuntu 18.04 with nvidia 390.87. Does this cause any error?

[sathiez]

Does seldonio/seldon-core-s2i-python3:0.4 image supports nvidia-390 driver? If not which version does it support, compatible with cuda 9.0?

[ukclivecox]

You may need to replace the cpu the tensorflow library with the gpu one.
What issues areyou having exactly?

[sathiez]

I already have tensorflow gpu in my seldon image. But, my issue is that
Tensorflow gpu depends on cuda9.0 which in turn depends on nvidia-390 driver since my host system has nvidia-390 driver. When I build my tensorflow model with this seldonio/seldon-core-s2i-python3:0.4 using s2i, and run the image, I get the following error while importing tensorflow:

2019-01-14 10:58:31.828997: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-01-14 10:58:31.834895: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: UNKNOWN ERROR (-1)
2019-01-14 10:58:31.834941: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] retrieving CUDA diagnostic information for host: 7a52b07f170d
2019-01-14 10:58:31.834949: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:170] hostname: 7a52b07f170d
2019-01-14 10:58:31.835018: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:194] libcuda reported version is: Not found: was unable to find libcuda.so DSO loaded into this program
2019-01-14 10:58:31.835050: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:198] kernel reported version is: 390.87.0

This error is probably due to nvidia driver version mismatch. So as mentioned in #380 (comment) which version of nvidia driver does seldonio/seldon-core-s2i-python3:0.4 support?

[ukclivecox]

There should be not restrictions on the version of the nvidia driver.

[sathiez]

Thank you, your comment was useful to know atleast this may not be issue