Error with GPU: CUBLAS_STATUS_EXECUTION_FAILED

[skull615d]

Unable to run test training on GPU. Here is the error stack. (Win 10, RTX 3060Ti, CUDA 10.0 cudnn 7.6.3) Please tell me what i am doing wrong.

2023-02-05 10:33:42.061163: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library cublas64_100.dll
2023-02-05 10:33:42.317036: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
org.tensorflow.TensorFlowException: 2 root error(s) found.
  (0) Internal: Blas GEMM launch failed : a.shape=(100, 300), b.shape=(300, 100), m=100, n=100, k=300
	 [[{{node MatMul_1}}]]
  (1) Internal: Blas GEMM launch failed : a.shape=(100, 300), b.shape=(300, 100), m=100, n=100, k=300
	 [[{{node MatMul_1}}]]
	 [[Mean_1/_13]]
0 successful operations.
0 derived errors ignored.

[zaleslaw]

Hmm, I've tested it on the following configuration . Could you miss the C++ redistributable parts?

[zaleslaw]

Also, there could be a problem with the drivers for the GPU card or memory allocation https://forums.developer.nvidia.com/t/error-failed-to-run-cublas-routine-cublas-status-execution-failed/164278
matterport/Mask_RCNN#2510

[zaleslaw]

Also, if you experimented with many CUDA versions, probably, you need to check twice the environment variables; the LIBRARY_PATH or PATH could contain the path to another CUDA version (it happens to me)

[skull615d]

C++ redistributable parts and PATH is OK.
How can this density be applied in KotlinDL?

import tensorflow as tf
config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

[zaleslaw]

We could provide such kind of options in the future release

This should be solved in the #540