An error occurred when using Nvidia 50 series GPU.

[TimerFirst]

I encountered a critical error while running TensorFlow-based DeepMD-kit (v2.0.3) on an ​NVIDIA RTX 5080 GPU (Compute Capability 12.0).The GPU is detected successfully and CUDA libraries load without errors,but the failure occurs during kernel execution.

I used "conda install deepmd-kit==GPU libdeepmd==gpu" to install deepmd-kit.

This error occurred when I using command "dp train input.json",and the file of input.json is from Quick Start Tutorial.

Your guidance on resolving this problem would be greatly appreciated.

Best regards.

This is the error message:
2025-04-10 23:51:56.602411: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /home/kang/anaconda3/envs/deepmd/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
2025-04-10 23:52:05.516622: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-04-10 23:52:05.518234: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2025-04-10 23:52:05.564458: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2025-04-10 23:52:05.565415: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA GeForce RTX 5080 computeCapability: 12.0
coreClock: 2.64GHz coreCount: 84 deviceMemorySize: 15.47GiB deviceMemoryBandwidth: 894.13GiB/s
2025-04-10 23:52:05.565434: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2025-04-10 23:52:05.567986: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2025-04-10 23:52:05.568008: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2025-04-10 23:52:05.568618: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2025-04-10 23:52:05.568771: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2025-04-10 23:52:05.569094: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2025-04-10 23:52:05.569592: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2025-04-10 23:52:05.569682: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2025-04-10 23:52:05.569723: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2025-04-10 23:52:05.570638: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:937] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUM

[njzjz]

I don't see where the error message is. Did you post the entire message?