I am using DPA-1 for the 'OC_10M.pb' large model 'pre-training + fine-tuning' and I am experiencing the following error reported by tensorflow and CUDA #3065

ZKNB · 2023-12-15T13:45:20Z

ZKNB
Dec 15, 2023

DEEPMD INFO          ./training_data/training_data9994/25      25       2       1  0.000    T
DEEPMD INFO          ./training_data/training_data9995/31      31       2       1  0.000    T
DEEPMD INFO          ./training_data/training_data9996/16      16       2       1  0.000    T
DEEPMD INFO          ./training_data/training_data9997/44      44       1       1  0.000    T
DEEPMD INFO          ./training_data/training_data9998/19      19       2       1  0.000    T
DEEPMD INFO          ./training_data/training_data9999/32      32       1       1  0.000    T
DEEPMD INFO    --------------------------------------------------------------------------------------
DEEPMD INFO    training without frame parameter
DEEPMD INFO    Changing energy bias in pretrained model for types ['H', 'C', 'O', 'Fe']... (this step may take long time)
WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2023-12-16 20:18:07.220340: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at custom_op.cc:18 : INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Runtime library throws an error: an illegal memory access was encountered, in file /home/conda/feedstock_root/build_artifacts/libdeepmd_1698439191117/work/source/lib/src/gpu/prod_env_mat.cu: 625, in file /home/conda/feedstock_root/build_artifacts/deepmd-kit_1698439299855/work/source/op/custom_op.cc:18
Traceback (most recent call last):
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1377, in _do_call
    return fn(*args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1360, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1453, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
tensorflow.python.framework.errors_impl.InternalError: 2 root error(s) found.
  (0) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Runtime library throws an error: an illegal memory access was encountered, in file /home/conda/feedstock_root/build_artifacts/libdeepmd_1698439191117/work/source/lib/src/gpu/prod_env_mat.cu: 625, in file /home/conda/feedstock_root/build_artifacts/deepmd-kit_1698439299855/work/source/op/custom_op.cc:18
	 [[{{node load/ProdEnvMatAMix}}]]
	 [[load/o_energy/_37]]
  (1) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Runtime library throws an error: an illegal memory access was encountered, in file /home/conda/feedstock_root/build_artifacts/libdeepmd_1698439191117/work/source/lib/src/gpu/prod_env_mat.cu: 625, in file /home/conda/feedstock_root/build_artifacts/deepmd-kit_1698439299855/work/source/op/custom_op.cc:18
	 [[{{node load/ProdEnvMatAMix}}]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
    sys.exit(main())
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
    deepmd_main(args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
    train_dp(**dict_args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
    _do_work(jdata, run_opt, is_compress)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
    model.build(train_data, stop_batch, origin_type_map=origin_type_map)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
    self._init_from_pretrained_model(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
    self._change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
    self.model.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
    self.fitting.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 848, in change_energy_bias
    ret = dp.eval(coord, box, atype, mixed_type=mixed_type)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 380, in eval
    output = self._eval_func(self._eval_inner, numb_test, natoms)(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 292, in eval_func
    return self.auto_batch_size.execute_all(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 194, in execute_all
    n_batch, result = self.execute(execute_with_batch_size, index, natoms)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 108, in execute
    n_batch, result = callable(max(batch_nframes, 1), start_index)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 172, in execute_with_batch_size
    return (end_index - start_index), callable(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 533, in _eval_inner
    v_out = run_sess(self.sess, t_out, feed_dict=feed_dict_test)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/sess.py", line 31, in run_sess
    return sess.run(*args, **kwargs)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 967, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1190, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1370, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/session.py", line 1396, in _do_call
    raise type(e)(node_def, op, message)  # pylint: disable=no-value-for-parameter
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'load/ProdEnvMatAMix' defined at (most recent call last):
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
      sys.exit(main())
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
      deepmd_main(args)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
      train_dp(**dict_args)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
      _do_work(jdata, run_opt, is_compress)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
      model.build(train_data, stop_batch, origin_type_map=origin_type_map)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
      self._init_from_pretrained_model(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
      self._change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
      self.model.change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
      self.fitting.change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 813, in change_energy_bias
      dp = DeepPotential(frozen_model)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/__init__.py", line 95, in DeepPotential
      dp = DeepPot(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 108, in __init__
      DeepEval.__init__(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 60, in __init__
      self.graph = self._load_graph(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 198, in _load_graph
      tf.import_graph_def(
Node: 'load/ProdEnvMatAMix'
Detected at node 'load/ProdEnvMatAMix' defined at (most recent call last):
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
      sys.exit(main())
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
      deepmd_main(args)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
      train_dp(**dict_args)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
      _do_work(jdata, run_opt, is_compress)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
      model.build(train_data, stop_batch, origin_type_map=origin_type_map)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
      self._init_from_pretrained_model(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
      self._change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
      self.model.change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
      self.fitting.change_energy_bias(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 813, in change_energy_bias
      dp = DeepPotential(frozen_model)
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/__init__.py", line 95, in DeepPotential
      dp = DeepPot(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 108, in __init__
      DeepEval.__init__(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 60, in __init__
      self.graph = self._load_graph(
    File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 198, in _load_graph
      tf.import_graph_def(
Node: 'load/ProdEnvMatAMix'
2 root error(s) found.
  (0) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Runtime library throws an error: an illegal memory access was encountered, in file /home/conda/feedstock_root/build_artifacts/libdeepmd_1698439191117/work/source/lib/src/gpu/prod_env_mat.cu: 625, in file /home/conda/feedstock_root/build_artifacts/deepmd-kit_1698439299855/work/source/op/custom_op.cc:18
	 [[{{node load/ProdEnvMatAMix}}]]
	 [[load/o_energy/_37]]
  (1) INTERNAL: Operation received an exception: DeePMD-kit Error: CUDA Runtime library throws an error: an illegal memory access was encountered, in file /home/conda/feedstock_root/build_artifacts/libdeepmd_1698439191117/work/source/lib/src/gpu/prod_env_mat.cu: 625, in file /home/conda/feedstock_root/build_artifacts/deepmd-kit_1698439299855/work/source/op/custom_op.cc:18
	 [[{{node load/ProdEnvMatAMix}}]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'load/ProdEnvMatAMix':
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
    sys.exit(main())
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
    deepmd_main(args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
    train_dp(**dict_args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
    _do_work(jdata, run_opt, is_compress)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
    model.build(train_data, stop_batch, origin_type_map=origin_type_map)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
    self._init_from_pretrained_model(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
    self._change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
    self.model.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
    self.fitting.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 813, in change_energy_bias
    dp = DeepPotential(frozen_model)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/__init__.py", line 95, in DeepPotential
    dp = DeepPot(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 108, in __init__
    DeepEval.__init__(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 60, in __init__
    self.graph = self._load_graph(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 198, in _load_graph
    tf.import_graph_def(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py", line 561, in new_func
    return func(*args, **kwargs)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 403, in import_graph_def
    return _import_graph_def_internal(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 516, in _import_graph_def_internal
    _ProcessNewOps(graph)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/importer.py", line 247, in _ProcessNewOps
    for new_op in graph._add_new_tf_operations(compute_devices=False):  # pylint: disable=protected-access
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3904, in _add_new_tf_operations
    new_ops = [
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3905, in <listcomp>
    self._create_op_from_tf_operation(c_op, compute_device=compute_devices)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 3787, in _create_op_from_tf_operation
    ret = Operation(c_op, self)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/ops.py", line 2133, in __init__
    self._traceback = tf_stack.extract_stack_for_node(self._c_op)

I just installed the latest version of deepmd kit and used the command 'conda create - n deepmd deepmd kit== gpu libdeepmd== gpu lammps horovod - c' https://conda.deepmodeling.com -C defaults, the dataset I used for testing is [Pt C-O-H]

I have 30,000 pieces of data.My test dateset as follows:

drwxrwxr-x 2 liluotong liluotong 146 Dec 15  2023 set.000000
-rw-rw-r-- 1 liluotong liluotong  10 Dec 15  2023 type_map.raw
-rw-rw-r-- 1 liluotong liluotong 288 Dec 15  2023 type.raw
ls set.000000
-rw-rw-r-- 1 liluotong liluotong  200 Dec 15  2023 box.npy
-rw-rw-r-- 1 liluotong liluotong 2432 Dec 15  2023 coord.npy
-rw-rw-r-- 1 liluotong liluotong  136 Dec 15  2023 energy.npy
-rw-rw-r-- 1 liluotong liluotong 2432 Dec 15  2023 force.npy
-rw-rw-r-- 1 liluotong liluotong  896 Dec 15  2023 real_atom_types.npy
-rw-rw-r-- 1 liluotong liluotong  200 Dec 15  2023 virial.npy

njzjz · 2023-12-16T20:55:08Z

njzjz
Dec 16, 2023
Maintainer

Same as #3034. I haven't successfully reproduced the issue. Could you share the data with me?

6 replies

njzjz Dec 17, 2023
Maintainer

The Horovod issue has been fixed by #3048. I'll take a look at other errors.

ZKNB Dec 18, 2023
Author

I check my run_option.py is same as #3048.
Thank you.I am looking forward to your reply.

ZKNB Dec 18, 2023
Author

When I retry 'dp train input.json --finetune OC_10M.pb',the #3034 error raise again.
input.json

ZKNB Dec 18, 2023
Author

I found that when I use one of the 10,000 pieces of data to run on one GPU, he doesn't have 'INTEL MKL ERROR', and when I use four GPUs in parallel, he has just 'Runtimeerror'

1] DEEPMD rank:1  INFO          ./training_data/training_data9996/16      16       2       1  0.000    T
[1] DEEPMD rank:1  INFO          ./training_data/training_data9997/44      44       1       1  0.000    T
[1] DEEPMD rank:1  INFO          ./training_data/training_data9998/19      19       2       1  0.000    T
[1] DEEPMD rank:1  INFO          ./training_data/training_data9999/32      32       1       1  0.000    T
[1] DEEPMD rank:1  INFO    --------------------------------------------------------------------------------------
[1] DEEPMD rank:1  INFO    training without frame parameter
[1] DEEPMD rank:1  INFO    Changing energy bias in pretrained model for types ['H', 'C', 'O', 'Fe']... (this step may take long time)
[0] DEEPMD rank:0  INFO    Changing energy bias in pretrained model for types ['H', 'C', 'O', 'Fe']... (this step may take long time)
[0] WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
[0] Instructions for updating:
[0] Use `tf.config.list_physical_devices('GPU')` instead.
[0] WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
[0] Instructions for updating:
[0] Use `tf.config.list_physical_devices('GPU')` instead.
[1] WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
[1] Instructions for updating:
[1] Use `tf.config.list_physical_devices('GPU')` instead.
[1] WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
[1] Instructions for updating:
[1] Use `tf.config.list_physical_devices('GPU')` instead.
[1] Traceback (most recent call last):
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
[1]     sys.exit(main())
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
[1]     deepmd_main(args)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
[1]     train_dp(**dict_args)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
[1]     _do_work(jdata, run_opt, is_compress)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
[1]     model.build(train_data, stop_batch, origin_type_map=origin_type_map)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
[1]     self._init_from_pretrained_model(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
[1]     self._change_energy_bias(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
[1]     self.model.change_energy_bias(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
[1]     self.fitting.change_energy_bias(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 813, in change_energy_bias
[1]     dp = DeepPotential(frozen_model)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/__init__.py", line 95, in DeepPotential
[1]     dp = DeepPot(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_pot.py", line 108, in __init__
[1]     DeepEval.__init__(
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/infer/deep_eval.py", line 79, in __init__
[1]     self.auto_batch_size = AutoBatchSize()
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py", line 62, in __init__
[1]     if tf.test.is_gpu_available():
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func
[1]     return func(*args, **kwargs)
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/framework/test_util.py", line 1923, in is_gpu_available
[1]     for local_device in device_lib.list_local_devices():
[1]   File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/tensorflow/python/client/device_lib.py", line 41, in list_local_devices
[1]     _convert(s) for s in _pywrap_device_lib.list_devices(serialized_config)
[1] RuntimeError: TensorFlow device (GPU:0) is being mapped to multiple devices (0 now, and 1 previously), which is not supported. This may be the result of providing different GPU configurations (ConfigProto.gpu_options, for example different visible_device_list) when creating multiple Sessions in the same process. This is not currently supported, see https://github.com/tensorflow/tensorflow/issues/19083

ZKNB Dec 19, 2023
Author

I found that when I train with a GPU card with only one set of data (data attached), he also gets 'INTEL MKL ERROR'. I performed the following test:

10000 data (including this set {O,H}), triggering 'INTEL MKL ERROR'
10000 data (not including this group {O,H}), successfully run
alone this data, trigger 'INTEL MKL ERROR'

DEEPMD INFO    ---Summary of DataSystem: training     -----------------------------------------------
DEEPMD INFO    found 1 system(s):
DEEPMD INFO                                        system  natoms  bch_sz   n_bch   prob  pbc
DEEPMD INFO                       ./training_data15368/24      24       2       1  1.000    T
DEEPMD INFO    --------------------------------------------------------------------------------------
DEEPMD INFO    training without frame parameter
DEEPMD INFO    Changing energy bias in pretrained model for types ['H', 'C', 'O', 'Fe']... (this step may take long time)
WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
WARNING:tensorflow:From /export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/utils/batch_size.py:62: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.

Intel MKL ERROR: Parameter 6 was incorrect on entry to DGELSD.
Traceback (most recent call last):
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/bin/dp", line 10, in <module>
    sys.exit(main())
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd_cli/main.py", line 635, in main
    deepmd_main(args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/main.py", line 74, in main
    train_dp(**dict_args)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 168, in train
    _do_work(jdata, run_opt, is_compress)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/entrypoints/train.py", line 280, in _do_work
    model.build(train_data, stop_batch, origin_type_map=origin_type_map)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 290, in build
    self._init_from_pretrained_model(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1137, in _init_from_pretrained_model
    self._change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/train/trainer.py", line 1145, in _change_energy_bias
    self.model.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/model/ener.py", line 507, in change_energy_bias
    self.fitting.change_energy_bias(
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/deepmd/fit/ener.py", line 856, in change_energy_bias
    delta_bias = np.linalg.lstsq(type_numbs, bias_diff, rcond=None)[0]
  File "<__array_function__ internals>", line 180, in lstsq
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 2292, in lstsq
    x, resids, rank, s = gufunc(a, b, rcond, signature=signature, extobj=extobj)
  File "/export/home/liluotonggpu2/anaconda3/envs/dpmd/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 100, in _raise_linalgerror_lstsq
    raise LinAlgError("SVD did not converge in Linear Least Squares")
numpy.linalg.LinAlgError: SVD did not converge in Linear Least Squares

training_data15368.zip

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

I am using DPA-1 for the 'OC_10M.pb' large model 'pre-training + fine-tuning' and I am experiencing the following error reported by tensorflow and CUDA #3065

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 6 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

I am using DPA-1 for the 'OC_10M.pb' large model 'pre-training + fine-tuning' and I am experiencing the following error reported by tensorflow and CUDA #3065

ZKNB Dec 15, 2023

Replies: 1 comment · 6 replies

njzjz Dec 16, 2023 Maintainer

njzjz Dec 17, 2023 Maintainer

ZKNB Dec 18, 2023 Author

ZKNB Dec 18, 2023 Author

ZKNB Dec 18, 2023 Author

ZKNB Dec 19, 2023 Author

ZKNB
Dec 15, 2023

Replies: 1 comment 6 replies

njzjz
Dec 16, 2023
Maintainer

njzjz Dec 17, 2023
Maintainer

ZKNB Dec 18, 2023
Author

ZKNB Dec 18, 2023
Author

ZKNB Dec 18, 2023
Author

ZKNB Dec 19, 2023
Author