You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
tensorflow-core-api:0.2.0-SNAPSHOT is failing on AWS EC2 instances with Tesla V100 GPUs.
all the core-api tests failed with error code:
CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid
I have root caused the issue to be CUDA compute 7.0 is not enabled during compilation, running the following command and build from source again fixed the issue. export TF_CUDA_COMPUTE_CAPABILITIES=7.0
Somehow in the release build of tensorflow-core-api, compute 7.0 is not enabled, but compute 3.5 and 7.0 should be the default capability of TF 2.3 according to here. The python packages and main repo built from source works fine without any modification.
Hi,
tensorflow-core-api:0.2.0-SNAPSHOT
is failing on AWS EC2 instances with Tesla V100 GPUs.all the core-api tests failed with error code:
I have root caused the issue to be CUDA compute 7.0 is not enabled during compilation, running the following command and build from source again fixed the issue.
export TF_CUDA_COMPUTE_CAPABILITIES=7.0
Somehow in the release build of tensorflow-core-api, compute 7.0 is not enabled, but compute 3.5 and 7.0 should be the default capability of TF 2.3 according to here. The python packages and main repo built from source works fine without any modification.
References:
tensorflow/tensorflow#41132
tensorflow/tensorflow@cf1b6b3
The text was updated successfully, but these errors were encountered: