tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found

[lauthirteen]

I trained 2 models using dpgen, following the example of CH4, In the 00.trian stage, I got the following error:

RuntimeError: Job ecbbfb3f-001e-4b19-8ee2-2abb65a182fc failed for more than 3 times

I checked the train.log file. Found the following error:
{
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
(0) Internal: Blas xGEMM launch failed : a.shape=[1,16,1], b.shape=[1,1,10], m=16, n=10, k=1
[[node filter_type_1/MatMul (defined at /lib/python3.9/site-packages/deepmd/utils/network.py:176) ]]
[[add_14/_35]]
(1) Internal: Blas xGEMM launch failed : a.shape=[1,16,1], b.shape=[1,1,10], m=16, n=10, k=1
[[node filter_type_1/MatMul (defined at /lib/python3.9/site-packages/deepmd/utils/network.py:176) ]]
0 successful operations.
0 derived errors ignored.
}

How should i solve this problem?
Best regards!

[njzjz]

Maybe same as deepmodeling/deepmd-kit#742. Did you use CUDA Toolkit 10.1 on a 30x card?

[lauthirteen]

Maybe same as deepmodeling/deepmd-kit#742. Did you use CUDA Toolkit 10.1 on a 30x card?

I use CUDA Toolkit 10.3 on GTX 2080 Ti card whith deepmd-kit 2.0.0. Do I need to lower the cuda version(11.1)?

[njzjz]

There is never a CUDA version 10.3. Do you mean 11.3 or others?

[lauthirteen]

There is never a CUDA version 10.3. Do you mean 11.3 or others?
Sorry, there's a typo. I used 11.3

[njzjz]

CUDA 11.3 requires driver version >=450.80.02. What's your driver version?

[lauthirteen]

CUDA 11.3 requires driver version >=450.80.02. What's your driver version?

version = 470.57.02. But, when I have installed the deepmd-kit-2.0.0.b2-cuda11.0_gpu-Linux-x86_64.sh package, it it works well.

[njzjz]

Can you provide the full log and input file? I'll try to reproduce the error.
Besides: What's version you are using?

[lauthirteen]

Can you provide the full log and input file? I'll try to reproduce the error.
Besides: What's version you are using?
The deepmd-kit verison is latest releases 2.0.0 with dpgen latest releases 0.10.2.
The train.log file information:
2021-08-31 08:43:26.112782: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
WARNING:deepmd.train.run_options:Switch to serial execution due to lack of horovod module.
/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/compat.py:316: UserWarning: It seems that you are using a deepmd-kit input of version 1.x.x, which is deprecated. we have converted the input to >2.0.0 compatible, and output it to file input_v2_compat.json
warnings.warn(msg)
2021-08-31 08:43:29.589483: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-08-31 08:43:29.595866: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-08-31 08:43:29.663294: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:b3:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.635GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2021-08-31 08:43:29.663391: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-08-31 08:43:29.673890: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-08-31 08:43:29.673980: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-08-31 08:43:29.676125: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-08-31 08:43:29.676521: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-08-31 08:43:29.677520: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-08-31 08:43:29.678962: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-08-31 08:43:29.679175: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-08-31 08:43:29.687849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-08-31 08:43:29.687905: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-08-31 08:43:30.339849: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-31 08:43:30.339889: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-08-31 08:43:30.339897: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-08-31 08:43:30.341527: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8150 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-08-31 08:43:30.341878: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-08-31 08:43:30.345022: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2100000000 Hz
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 0-31
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 32 available OS procs
OMP: Info #158: KMP_AFFINITY: Uniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 1 socket x 16 cores/socket x 2 threads/core (16 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 16 maps to socket 0 core 0 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 1 maps to socket 0 core 1 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 17 maps to socket 0 core 1 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 2 maps to socket 0 core 2 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 18 maps to socket 0 core 2 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 3 maps to socket 0 core 3 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 19 maps to socket 0 core 3 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 4 maps to socket 0 core 4 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 20 maps to socket 0 core 4 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 5 maps to socket 0 core 5 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 21 maps to socket 0 core 5 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 6 maps to socket 0 core 6 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 22 maps to socket 0 core 6 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 7 maps to socket 0 core 7 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 23 maps to socket 0 core 7 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 8 maps to socket 0 core 8 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 24 maps to socket 0 core 8 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 9 maps to socket 0 core 9 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 25 maps to socket 0 core 9 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 10 maps to socket 0 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 26 maps to socket 0 core 10 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 11 maps to socket 0 core 11 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 27 maps to socket 0 core 11 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 12 maps to socket 0 core 12 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 28 maps to socket 0 core 12 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 13 maps to socket 0 core 13 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 29 maps to socket 0 core 13 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 14 maps to socket 0 core 14 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 30 maps to socket 0 core 14 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 15 maps to socket 0 core 15 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 31 maps to socket 0 core 15 thread 1
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28818 thread 1 bound to OS proc set 1
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28888 thread 2 bound to OS proc set 2
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28889 thread 3 bound to OS proc set 3
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28891 thread 5 bound to OS proc set 5
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28890 thread 4 bound to OS proc set 4
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28892 thread 6 bound to OS proc set 6
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28894 thread 8 bound to OS proc set 8
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28893 thread 7 bound to OS proc set 7
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28895 thread 9 bound to OS proc set 9
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28898 thread 10 bound to OS proc set 10
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28900 thread 11 bound to OS proc set 11
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28902 thread 12 bound to OS proc set 12
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28903 thread 13 bound to OS proc set 13
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28905 thread 14 bound to OS proc set 14
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28907 thread 15 bound to OS proc set 15
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28910 thread 16 bound to OS proc set 16
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28912 thread 17 bound to OS proc set 17
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28913 thread 18 bound to OS proc set 18
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28915 thread 19 bound to OS proc set 19
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28916 thread 20 bound to OS proc set 20
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28918 thread 21 bound to OS proc set 21
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28921 thread 22 bound to OS proc set 22
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28922 thread 23 bound to OS proc set 23
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28924 thread 24 bound to OS proc set 24
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28926 thread 25 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28928 thread 26 bound to OS proc set 26
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28930 thread 27 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28932 thread 28 bound to OS proc set 28
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28934 thread 29 bound to OS proc set 29
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28936 thread 30 bound to OS proc set 30
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28939 thread 31 bound to OS proc set 31
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28941 thread 32 bound to OS proc set 0
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28817 thread 33 bound to OS proc set 1
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28950 thread 34 bound to OS proc set 2
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28951 thread 35 bound to OS proc set 3
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28953 thread 37 bound to OS proc set 5
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28952 thread 36 bound to OS proc set 4
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28954 thread 38 bound to OS proc set 6
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28955 thread 39 bound to OS proc set 7
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28956 thread 40 bound to OS proc set 8
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28957 thread 41 bound to OS proc set 9
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28958 thread 42 bound to OS proc set 10
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28960 thread 44 bound to OS proc set 12
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28959 thread 43 bound to OS proc set 11
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28961 thread 45 bound to OS proc set 13
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28962 thread 46 bound to OS proc set 14
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28963 thread 47 bound to OS proc set 15
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28964 thread 48 bound to OS proc set 16
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28965 thread 49 bound to OS proc set 17
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28966 thread 50 bound to OS proc set 18
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28967 thread 51 bound to OS proc set 19
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28968 thread 52 bound to OS proc set 20
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28969 thread 53 bound to OS proc set 21
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28970 thread 54 bound to OS proc set 22
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28971 thread 55 bound to OS proc set 23
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28972 thread 56 bound to OS proc set 24
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28973 thread 57 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28974 thread 58 bound to OS proc set 26
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28975 thread 59 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28976 thread 60 bound to OS proc set 28
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28977 thread 61 bound to OS proc set 29
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28978 thread 62 bound to OS proc set 30
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28979 thread 63 bound to OS proc set 31
OMP: Info #254: KMP_AFFINITY: pid 28435 tid 28980 thread 64 bound to OS proc set 0
DEEPMD INFO training data with min nbor dist: 1.042950111442311
DEEPMD INFO training data with max nbor size: [4, 1]
DEEPMD INFO _____ _____ __ __ _____ _ _ _
DEEPMD INFO | __ \ | __ \ | / || __ \ | | ()| |
DEEPMD INFO | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |
DEEPMD INFO | | | | / _ \ / _ | / | |/| || | | |||| |/ /| || |
DEEPMD INFO | || || /| /| | | | | || || | | < | || |
DEEPMD INFO |/ _| _||| || |_||____/ ||_|| __|
DEEPMD INFO Please read and cite:
DEEPMD INFO Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
DEEPMD INFO installed to: /tmp/pip-req-build-8xtytcha/_skbuild/linux-x86_64-3.9/cmake-install
DEEPMD INFO source : v2.0.0
DEEPMD INFO source brach: HEAD
DEEPMD INFO source commit: 1a25414
DEEPMD INFO source commit at: 2021-08-28 08:15:38 +0800
DEEPMD INFO build float prec: double
DEEPMD INFO build with tf inc: /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/include
DEEPMD INFO build with tf lib:
DEEPMD INFO ---Summary of the training---------------------------------------
DEEPMD INFO running on: dell-Precision-7920-Tower
DEEPMD INFO computing device: gpu:0
DEEPMD INFO CUDA_VISIBLE_DEVICES: unset
DEEPMD INFO Count of visible GPU: 1
DEEPMD INFO num_intra_threads: 0
DEEPMD INFO num_inter_threads: 0
DEEPMD INFO -----------------------------------------------------------------
2021-08-31 08:43:31.556613: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:b3:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.635GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2021-08-31 08:43:31.557916: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-08-31 08:43:31.557964: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-31 08:43:31.557980: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-08-31 08:43:31.557991: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-08-31 08:43:31.558955: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8150 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
DEEPMD INFO ---Summary of DataSystem: training -----------------------------------------------
DEEPMD INFO found 1 system(s):
DEEPMD INFO system natoms bch_sz n_bch prob pbc
DEEPMD INFO -- CAR.01x01x01/02.md/sys-0004-0001/deepmd 5 8 25 1.000 T
DEEPMD INFO --------------------------------------------------------------------------------------
DEEPMD INFO training without frame parameter
2021-08-31 08:43:31.586302: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 7.96G (8546484224 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.586975: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 7.16G (7691835392 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.587934: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 6.45G (6922651648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.588797: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 5.80G (6230386176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.589363: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 5.22G (5607347200 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.589974: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 4.70G (5046612480 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.590864: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 4.23G (4541950976 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.591671: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 3.81G (4087755776 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.599253: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 3.43G (3678980096 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.599843: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 3.08G (3311081984 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.600575: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 2.77G (2979973632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.601385: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 2.50G (2681976320 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.601912: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 2.25G (2413778688 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.606865: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 2.02G (2172400640 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.608086: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.82G (1955160576 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.609064: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.64G (1759644416 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.610106: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.47G (1583680000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.612003: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.33G (1425312000 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.612927: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.19G (1282780928 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.618917: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 1.08G (1154502912 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.620508: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 990.92M (1039052800 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.621974: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 891.83M (935147520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.631103: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 802.64M (841632768 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.633019: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 722.38M (757469440 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.634418: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 650.14M (681722624 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.635879: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 585.13M (613550336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.638084: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 526.61M (552195328 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.640111: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 473.95M (496975872 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.642486: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 426.56M (447278336 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.645508: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 383.90M (402550528 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.647665: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 345.51M (362295552 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.649850: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 310.96M (326066176 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.652066: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 279.86M (293459712 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.654642: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 251.88M (264113920 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.657175: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 226.69M (237702656 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.659289: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 204.02M (213932544 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.667406: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 183.62M (192539392 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.670401: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 165.26M (173285632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.673285: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 148.73M (155957248 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.675783: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 133.86M (140361728 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.678175: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 120.47M (126325760 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.680826: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 108.43M (113693184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.683715: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 97.58M (102323968 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.686199: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 87.83M (92091648 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.689297: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 79.04M (82882560 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.691922: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 71.14M (74594304 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.694284: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 64.02M (67134976 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.696014: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 57.62M (60421632 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.698566: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 51.86M (54379520 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.700678: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 46.67M (48941568 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.703234: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 42.01M (44047616 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.705742: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 37.81M (39642880 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.707421: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 34.03M (35678720 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.709873: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 30.62M (32110848 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.711981: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 27.56M (28899840 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.714466: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 24.80M (26009856 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.716543: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 22.32M (23408896 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.718091: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 20.09M (21068032 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.720527: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 18.08M (18961408 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.722989: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 16.27M (17065472 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.725422: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 14.65M (15358976 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.727497: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 13.18M (13823232 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.729092: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 11.86M (12441088 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.731468: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 10.68M (11197184 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.733571: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 9.61M (10077696 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.735574: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 8.65M (9070080 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.738459: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 7.78M (8163072 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
2021-08-31 08:43:31.740384: I tensorflow/stream_executor/cuda/cuda_driver.cc:789] failed to allocate 7.01M (7346944 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY: out of memory
cuda assert: out of memory /tmp/pip-req-build-8xtytcha/source/lib/include/gpu_cuda.h 123
Your memory is not enough, thus an error has been raised above. You need to take the following actions:

1. Check if the network size of the model is too large.
2. Check if the batch size of training or testing is too large. You can set the training batch size to auto.
3. Check if the number of atoms is too large.
4. Check if another program is using the same GPU by execuating nvidia-smi. The usage of GPUs is controlled by CUDA_VISIBLE_DEVICES environment variable.
   2021-08-31 08:43:31.755090: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at custom_op.cc:12 : Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
   Traceback (most recent call last):
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
   return fn(*args)
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
   return self._call_tf_sessionrun(options, feed_dict, fetch_list,
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
   return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
   tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
   (0) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
   [[{{node ProdEnvMatA}}]]
   Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[{{node ProdEnvMatA}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[ProdEnvMatA/_11]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/sess.py", line 20, in run_sess
return sess.run(*args, **kwargs)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[node ProdEnvMatA (defined at :1307) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[node ProdEnvMatA (defined at :1307) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[ProdEnvMatA/_11]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node ProdEnvMatA:
d_sea_t_type (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:172)
d_sea_t_box (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:171)
d_sea_t_natoms (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:173)
Const_1 (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:182)
Const (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:181)
d_sea_t_mesh (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:174)

Input Source operations connected to node ProdEnvMatA:
d_sea_t_type (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:172)
d_sea_t_box (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:171)
d_sea_t_natoms (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:173)
Const_1 (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:182)
Const (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:181)
d_sea_t_mesh (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:174)

Original stack trace for 'ProdEnvMatA':
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/bin/dp", line 10, in
sys.exit(main())
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 102, in train
_do_work(jdata, run_opt, is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 126, in _do_work
model = DPTrainer(jdata, run_opt=run_opt, is_compress = is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 88, in init
self._init_param(jdata)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 107, in _init_param
self.descrpt = _generate_descrpt_from_param_dict(descrpt_param)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 66, in _generate_descrpt_from_param_dict
descrpt = DescrptSeA(**descrpt_param)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 176, in init
= op_module.prod_env_mat_a(self.place_holders['coord'],
File "", line 1307, in prod_env_mat_a
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3557, in _create_op_internal
ret = Operation(
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2045, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/bin/dp", line 10, in
sys.exit(main())
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 102, in train
_do_work(jdata, run_opt, is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 158, in _do_work
model.build(train_data, stop_batch)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 311, in build
self.model.data_stat(data)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/model/ener.py", line 93, in data_stat
self._compute_input_stat(m_all_stat, protection = self.data_stat_protect)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/model/ener.py", line 98, in _compute_input_stat
self.descrpt.compute_input_stats(all_stat['coord'],
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 266, in compute_input_stats
= self._compute_dstats_sys_smth(cc,bb,tt,nn,mm)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 617, in _compute_dstats_sys_smth
= run_sess(self.sub_sess, self.stat_descrpt,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/sess.py", line 38, in run_sess
raise RuntimeError(MESSAGE) from e
RuntimeError: Your memory may be not enough, thus an error has been raised above. You need to take the following actions:

1. Check if the network size of the model is too large.
2. Check if the batch size of training or testing is too large. You can set the training batch size to auto.
3. Check if the number of atoms is too large.
4. Check if another program is using the same GPU by execuating nvidia-smi. The usage of GPUs is controlled by CUDA_VISIBLE_DEVICES environment variable (current value: None).

2021-08-31 08:44:26.574334: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
WARNING:tensorflow:From /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/compat/v2_compat.py:96: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term
WARNING:root:Environment variable KMP_BLOCKTIME is empty. Use the default value 0
WARNING:root:Environment variable KMP_AFFINITY is empty. Use the default value granularity=fine,verbose,compact,1,0
WARNING:deepmd.train.run_options:Switch to serial execution due to lack of horovod module.
/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/compat.py:316: UserWarning: It seems that you are using a deepmd-kit input of version 1.x.x, which is deprecated. we have converted the input to >2.0.0 compatible, and output it to file input_v2_compat.json
warnings.warn(msg)
2021-08-31 08:44:30.006053: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-08-31 08:44:30.013430: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcuda.so.1
2021-08-31 08:44:30.028359: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:b3:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.635GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2021-08-31 08:44:30.028428: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-08-31 08:44:30.033870: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublas.so.11
2021-08-31 08:44:30.033933: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcublasLt.so.11
2021-08-31 08:44:30.035140: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcufft.so.10
2021-08-31 08:44:30.035457: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcurand.so.10
2021-08-31 08:44:30.036218: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusolver.so.11
2021-08-31 08:44:30.037283: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcusparse.so.11
2021-08-31 08:44:30.037456: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudnn.so.8
2021-08-31 08:44:30.038180: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-08-31 08:44:30.038224: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0
2021-08-31 08:44:30.697524: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-31 08:44:30.697570: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-08-31 08:44:30.697578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-08-31 08:44:30.698398: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 844 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
2021-08-31 08:44:30.698922: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
2021-08-31 08:44:30.702186: I tensorflow/core/platform/profile_utils/cpu_utils.cc:114] CPU Frequency: 2100000000 Hz
OMP: Info #155: KMP_AFFINITY: Initial OS proc set respected: 0-31
OMP: Info #216: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #157: KMP_AFFINITY: 32 available OS procs
OMP: Info #158: KMP_AFFINITY: Uniform topology
OMP: Info #287: KMP_AFFINITY: topology layer "LL cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L3 cache" is equivalent to "socket".
OMP: Info #287: KMP_AFFINITY: topology layer "L2 cache" is equivalent to "core".
OMP: Info #287: KMP_AFFINITY: topology layer "L1 cache" is equivalent to "core".
OMP: Info #192: KMP_AFFINITY: 1 socket x 16 cores/socket x 2 threads/core (16 total cores)
OMP: Info #218: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #172: KMP_AFFINITY: OS proc 0 maps to socket 0 core 0 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 16 maps to socket 0 core 0 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 1 maps to socket 0 core 1 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 17 maps to socket 0 core 1 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 2 maps to socket 0 core 2 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 18 maps to socket 0 core 2 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 3 maps to socket 0 core 3 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 19 maps to socket 0 core 3 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 4 maps to socket 0 core 4 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 20 maps to socket 0 core 4 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 5 maps to socket 0 core 5 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 21 maps to socket 0 core 5 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 6 maps to socket 0 core 6 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 22 maps to socket 0 core 6 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 7 maps to socket 0 core 7 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 23 maps to socket 0 core 7 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 8 maps to socket 0 core 8 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 24 maps to socket 0 core 8 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 9 maps to socket 0 core 9 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 25 maps to socket 0 core 9 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 10 maps to socket 0 core 10 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 26 maps to socket 0 core 10 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 11 maps to socket 0 core 11 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 27 maps to socket 0 core 11 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 12 maps to socket 0 core 12 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 28 maps to socket 0 core 12 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 13 maps to socket 0 core 13 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 29 maps to socket 0 core 13 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 14 maps to socket 0 core 14 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 30 maps to socket 0 core 14 thread 1
OMP: Info #172: KMP_AFFINITY: OS proc 15 maps to socket 0 core 15 thread 0
OMP: Info #172: KMP_AFFINITY: OS proc 31 maps to socket 0 core 15 thread 1
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29363 thread 1 bound to OS proc set 1
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29366 thread 2 bound to OS proc set 2
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29367 thread 3 bound to OS proc set 3
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29369 thread 5 bound to OS proc set 5
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29368 thread 4 bound to OS proc set 4
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29370 thread 6 bound to OS proc set 6
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29371 thread 7 bound to OS proc set 7
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29372 thread 8 bound to OS proc set 8
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29373 thread 9 bound to OS proc set 9
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29374 thread 10 bound to OS proc set 10
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29375 thread 11 bound to OS proc set 11
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29376 thread 12 bound to OS proc set 12
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29377 thread 13 bound to OS proc set 13
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29378 thread 14 bound to OS proc set 14
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29379 thread 15 bound to OS proc set 15
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29380 thread 16 bound to OS proc set 16
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29381 thread 17 bound to OS proc set 17
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29383 thread 19 bound to OS proc set 19
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29382 thread 18 bound to OS proc set 18
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29384 thread 20 bound to OS proc set 20
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29385 thread 21 bound to OS proc set 21
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29386 thread 22 bound to OS proc set 22
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29387 thread 23 bound to OS proc set 23
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29388 thread 24 bound to OS proc set 24
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29389 thread 25 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29390 thread 26 bound to OS proc set 26
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29391 thread 27 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29392 thread 28 bound to OS proc set 28
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29393 thread 29 bound to OS proc set 29
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29394 thread 30 bound to OS proc set 30
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29395 thread 31 bound to OS proc set 31
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29396 thread 32 bound to OS proc set 0
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29364 thread 33 bound to OS proc set 1
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29397 thread 34 bound to OS proc set 2
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29398 thread 35 bound to OS proc set 3
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29399 thread 36 bound to OS proc set 4
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29400 thread 37 bound to OS proc set 5
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29401 thread 38 bound to OS proc set 6
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29402 thread 39 bound to OS proc set 7
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29403 thread 40 bound to OS proc set 8
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29404 thread 41 bound to OS proc set 9
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29405 thread 42 bound to OS proc set 10
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29406 thread 43 bound to OS proc set 11
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29407 thread 44 bound to OS proc set 12
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29408 thread 45 bound to OS proc set 13
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29409 thread 46 bound to OS proc set 14
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29410 thread 47 bound to OS proc set 15
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29411 thread 48 bound to OS proc set 16
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29412 thread 49 bound to OS proc set 17
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29413 thread 50 bound to OS proc set 18
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29414 thread 51 bound to OS proc set 19
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29415 thread 52 bound to OS proc set 20
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29416 thread 53 bound to OS proc set 21
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29417 thread 54 bound to OS proc set 22
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29418 thread 55 bound to OS proc set 23
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29420 thread 57 bound to OS proc set 25
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29419 thread 56 bound to OS proc set 24
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29421 thread 58 bound to OS proc set 26
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29422 thread 59 bound to OS proc set 27
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29423 thread 60 bound to OS proc set 28
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29424 thread 61 bound to OS proc set 29
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29425 thread 62 bound to OS proc set 30
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29426 thread 63 bound to OS proc set 31
OMP: Info #254: KMP_AFFINITY: pid 29106 tid 29427 thread 64 bound to OS proc set 0
DEEPMD INFO training data with min nbor dist: 1.042950111442311
DEEPMD INFO training data with max nbor size: [4, 1]
DEEPMD INFO _____ _____ __ __ _____ _ _ _
DEEPMD INFO | __ \ | __ \ | / || __ \ | | ()| |
DEEPMD INFO | | | | ___ ___ | |__) || \ / || | | | ______ | | __ _ | |
DEEPMD INFO | | | | / _ \ / _ | / | |/| || | | |||| |/ /| || |
DEEPMD INFO | || || /| /| | | | | || || | | < | || |
DEEPMD INFO |/ _| _||| || |_||____/ ||_|| __|
DEEPMD INFO Please read and cite:
DEEPMD INFO Wang, Zhang, Han and E, Comput.Phys.Comm. 228, 178-184 (2018)
DEEPMD INFO installed to: /tmp/pip-req-build-8xtytcha/_skbuild/linux-x86_64-3.9/cmake-install
DEEPMD INFO source : v2.0.0
DEEPMD INFO source brach: HEAD
DEEPMD INFO source commit: 1a25414
DEEPMD INFO source commit at: 2021-08-28 08:15:38 +0800
DEEPMD INFO build float prec: double
DEEPMD INFO build with tf inc: /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/include
DEEPMD INFO build with tf lib:
DEEPMD INFO ---Summary of the training---------------------------------------
DEEPMD INFO running on: dell-Precision-7920-Tower
DEEPMD INFO computing device: gpu:0
DEEPMD INFO CUDA_VISIBLE_DEVICES: unset
DEEPMD INFO Count of visible GPU: 1
DEEPMD INFO num_intra_threads: 0
DEEPMD INFO num_inter_threads: 0
DEEPMD INFO -----------------------------------------------------------------
2021-08-31 08:44:32.261894: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1733] Found device 0 with properties:
pciBusID: 0000:b3:00.0 name: NVIDIA GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.635GHz coreCount: 68 deviceMemorySize: 10.75GiB deviceMemoryBandwidth: 573.69GiB/s
2021-08-31 08:44:32.263442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1871] Adding visible gpu devices: 0
2021-08-31 08:44:32.263489: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-08-31 08:44:32.263503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0
2021-08-31 08:44:32.263514: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N
2021-08-31 08:44:32.264324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 844 MB memory) -> physical GPU (device: 0, name: NVIDIA GeForce RTX 2080 Ti, pci bus id: 0000:b3:00.0, compute capability: 7.5)
DEEPMD INFO ---Summary of DataSystem: training -----------------------------------------------
DEEPMD INFO found 1 system(s):
DEEPMD INFO system natoms bch_sz n_bch prob pbc
DEEPMD INFO -- CAR.01x01x01/02.md/sys-0004-0001/deepmd 5 8 25 1.000 T
DEEPMD INFO --------------------------------------------------------------------------------------
DEEPMD INFO training without frame parameter
cuda assert: out of memory /tmp/pip-req-build-8xtytcha/source/lib/include/gpu_cuda.h 123
Your memory is not enough, thus an error has been raised above. You need to take the following actions:

1. Check if the network size of the model is too large.
2. Check if the batch size of training or testing is too large. You can set the training batch size to auto.
3. Check if the number of atoms is too large.
4. Check if another program is using the same GPU by execuating nvidia-smi. The usage of GPUs is controlled by CUDA_VISIBLE_DEVICES environment variable.
   2021-08-31 08:44:32.462001: W tensorflow/core/framework/op_kernel.cc:1767] OP_REQUIRES failed at custom_op.cc:12 : Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
   Traceback (most recent call last):
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1375, in _do_call
   return fn(*args)
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1359, in _run_fn
   return self._call_tf_sessionrun(options, feed_dict, fetch_list,
   File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1451, in _call_tf_sessionrun
   return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
   tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
   (0) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
   [[{{node ProdEnvMatA}}]]
   Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[{{node ProdEnvMatA}}]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[ProdEnvMatA/_11]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/sess.py", line 20, in run_sess
return sess.run(*args, **kwargs)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 967, in run
result = self._run(None, fetches, feed_dict, options_ptr,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1190, in _run
results = self._do_run(handle, final_targets, final_fetches,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1368, in _do_run
return self._do_call(_run_fn, feeds, fetches, targets, options,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/client/session.py", line 1394, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: 2 root error(s) found.
(0) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[node ProdEnvMatA (defined at :1307) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

(1) Resource exhausted: Operation received an exception: DeePMD-kit OOM: CUDA Assert, in file /tmp/pip-req-build-8xtytcha/source/op/custom_op.cc:12
[[node ProdEnvMatA (defined at :1307) ]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

 [[ProdEnvMatA/_11]]

Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.

0 successful operations.
0 derived errors ignored.

Errors may have originated from an input operation.
Input Source operations connected to node ProdEnvMatA:
d_sea_t_type (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:172)
d_sea_t_box (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:171)
d_sea_t_natoms (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:173)
Const_1 (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:182)
Const (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:181)
d_sea_t_mesh (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:174)

Input Source operations connected to node ProdEnvMatA:
d_sea_t_type (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:172)
d_sea_t_box (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:171)
d_sea_t_natoms (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:173)
Const_1 (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:182)
Const (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:181)
d_sea_t_mesh (defined at /home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py:174)

Original stack trace for 'ProdEnvMatA':
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/bin/dp", line 10, in
sys.exit(main())
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 102, in train
_do_work(jdata, run_opt, is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 126, in _do_work
model = DPTrainer(jdata, run_opt=run_opt, is_compress = is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 88, in init
self._init_param(jdata)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 107, in _init_param
self.descrpt = _generate_descrpt_from_param_dict(descrpt_param)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 66, in _generate_descrpt_from_param_dict
descrpt = DescrptSeA(**descrpt_param)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 176, in init
= op_module.prod_env_mat_a(self.place_holders['coord'],
File "", line 1307, in prod_env_mat_a
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/op_def_library.py", line 748, in _apply_op_helper
op = g._create_op_internal(op_type_name, inputs, dtypes=None,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 3557, in _create_op_internal
ret = Operation(
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/tensorflow/python/framework/ops.py", line 2045, in init
self._traceback = tf_stack.extract_stack_for_node(self._c_op)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/bin/dp", line 10, in
sys.exit(main())
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/main.py", line 437, in main
train_dp(**dict_args)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 102, in train
_do_work(jdata, run_opt, is_compress)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/entrypoints/train.py", line 158, in _do_work
model.build(train_data, stop_batch)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/train/trainer.py", line 311, in build
self.model.data_stat(data)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/model/ener.py", line 93, in data_stat
self._compute_input_stat(m_all_stat, protection = self.data_stat_protect)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/model/ener.py", line 98, in _compute_input_stat
self.descrpt.compute_input_stats(all_stat['coord'],
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 266, in compute_input_stats
= self._compute_dstats_sys_smth(cc,bb,tt,nn,mm)
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/descriptor/se_a.py", line 617, in _compute_dstats_sys_smth
= run_sess(self.sub_sess, self.stat_descrpt,
File "/home/dell/soft_install/deepmd-kit/envs/deepmd_gpu/lib/python3.9/site-packages/deepmd/utils/sess.py", line 38, in run_sess
raise RuntimeError(MESSAGE) from e
RuntimeError: Your memory may be not enough, thus an error has been raised above. You need to take the following actions:

1. Check if the network size of the model is too large.
2. Check if the batch size of training or testing is too large. You can set the training batch size to auto.
3. Check if the number of atoms is too large.
4. Check if another program is using the same GPU by execuating nvidia-smi. The usage of GPUs is controlled by CUDA_VISIBLE_DEVICES environment variable (current value: None).

The input file is about CH4 example.

[njzjz]

As mentioned above, this is an out-of-memory error. Some guides have been given.

[lauthirteen]

As mentioned above, this is an out-of-memory error. Some guides have been given.

Thank you very much.