failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error

[atulapra]

System information

• Ubuntu 16.04 LTS
• TensorFlow version: 1.12
• Python version: 3.6
• CUDA: 9.0
• CUDNN: 7.3
• GPU model and memory: NVIDIA GeForce GTX 1050 Ti

Code
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))

Output
2018-12-08 22:20:54.515356: E tensorflow/stream_executor/cuda/cuda_driver.cc:300] failed call to cuInit: CUDA_ERROR_UNKNOWN: unknown error
Device mapping: /job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device /job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device

I have tried nvidia-smi and it works as expected:

Sat Dec 8 22:24:22 2018
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130 Driver Version: 384.130 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 105... Off | 00000000:01:00.0 Off | N/A |
| N/A 51C P0 N/A / N/A | 500MiB / 4038MiB | 14% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 1357 G /usr/lib/xorg/Xorg 300MiB |
| 0 2937 G compiz 58MiB |
| 0 9866 G ...quest-channel-token=7879820401391808802 92MiB |
| 0 26531 G ...-token=5979D08CF023D5A85228AD97A282F087 47MiB |
+-----------------------------------------------------------------------------+

Also, nvidia-modprobe is installed.
Even rebooting has not solved the issue.

[SlyStrategist]

did you find the solution? I am facing similar problem...

[atulapra]

@SlyStrategist

My version of cuDNN was not compatible with tensorflow-gpu 1.12.
The correct version for tensorflow-gpu 1.12 can be found here.

If it still doesn't work, try rebooting.

[rsingh2083]

System reboot worked for me

[Yossarian0916]

Reboot worked for me, too

[DiyaLi]

reboot worked for me, too. But still don't know how to avoid this happening