error running dpgen cp2k methane example #1237

kyuan2019 · 2023-05-30T16:03:07Z

kyuan2019
May 30, 2023

Hi all,
I am a beginner on dpgen and am testing the dpgen using the cp2k methane example . However, the program run to the second iteration and stoped because the accuracy is very low (0.08%). See error message below. What could be the reason for this? and which file should I look into? I used most parameters from the example file. Please see my machine.jason and parameters.jason files attached after the error message. Thanks!

error message:
nohup: ignoring input
INFO:dpgen:-------------------------iter.000000 task 01--------------------------
INFO:dpgen:-------------------------iter.000000 task 02--------------------------
INFO:dpgen:-------------------------iter.000000 task 03--------------------------
INFO:dpgen:-------------------------iter.000000 task 04--------------------------
INFO:dpgen:-------------------------iter.000000 task 05--------------------------
INFO:dpgen:-------------------------iter.000000 task 06--------------------------
INFO:dpgen:system 000 candidate : 624 in 2000 31.20 %
INFO:dpgen:system 000 failed : 0 in 2000 0.00 %
INFO:dpgen:system 000 accurate : 1376 in 2000 68.80 %
INFO:dpgen:system 000 accurate_ratio: 0.6880 thresholds: 1.0000 and 1.0000 eff. task min and max -1 30 number of fp tasks:
30
INFO:dpgen:-------------------------iter.000000 task 07--------------------------
INFO:dpgen:-------------------------iter.000000 task 08--------------------------
INFO:dpgen:failed frame: 0 in 30 0.00 %
INFO:dpgen:failed tasks: 0 in 30 0.00 %
INFO:dpgen:=============================iter.000001==============================
INFO:dpgen:-------------------------iter.000001 task 00--------------------------
INFO:dpgen:-------------------------iter.000001 task 01--------------------------
INFO:dpgen:-------------------------iter.000001 task 02--------------------------
INFO:dpgen:-------------------------iter.000001 task 03--------------------------
INFO:dpgen:-------------------------iter.000001 task 04--------------------------
INFO:dpgen:-------------------------iter.000001 task 05--------------------------
INFO:dpgen:-------------------------iter.000001 task 06--------------------------
INFO:dpgen:system 001 candidate : 2306 in 5000 46.12 %
INFO:dpgen:system 001 failed : 2690 in 5000 53.80 %
INFO:dpgen:system 001 accurate : 4 in 5000 0.08 %
INFO:dpgen:system 001 accurate_ratio: 0.0008 thresholds: 1.0000 and 1.0000 eff. task min and max -1 30 number of fp tasks: 30
INFO:dpgen:-------------------------iter.000001 task 07--------------------------
INFO:dpgen:-------------------------iter.000001 task 08--------------------------
Traceback (most recent call last):
File "/home/cloud/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/main.py", line 233, in main
args.func(args)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 5109, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4464, in run_iter
post_fp(ii, jdata)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4330, in post_fp
post_fp_cp2k(iter_index, jdata)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4213, in post_fp_cp2k
all_sys.to_deepmd_raw(sys_data_path)
File "/home/cloud/.local/lib/python3.8/site-packages/dpdata/system.py", line 1697, in to_format
return self.to_fmt_obj(ff(), *args, **kwargs)
File "/home/cloud/.local/lib/python3.8/site-packages/dpdata/system.py", line 1176, in to_fmt_obj
return fmtobj.to_labeled_system(self.data, *args, **kwargs)
File "/home/cloud/.local/lib/python3.8/site-packages/dpdata/format.py", line 238, in to_labeled_system
return self.to_system(data, *args, **kwargs)
File "/home/cloud/.local/lib/python3.8/site-packages/dpdata/plugins/deepmd.py", line 26, in to_system
dpdata.deepmd.raw.dump(file_name, data)
File "/home/cloud/.local/lib/python3.8/site-packages/dpdata/deepmd/raw.py", line 67, in dump
nframes = data["cells"].shape[0]
AttributeError: 'list' object has no attribute 'shape'

Machine Jason:
{
"api_version": "1.0",
"deepmd_version": "2.2.1",
"train" :[
{
"command": "dp",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"remote_root": "/home/cloud/dpgen_example/run/work"
},
"resources": {
"number_node": 1,
"cpu_per_node": 32,
"gpu_per_node": 0,
"source_list": ["export OMP_NUM_THREADS=32; export TF_INTRA_OP_PARALLELISM_THREADS=16; export TF_INTER_OP_P
ARALLELISM_THREADS=2"],
"group_size": 4
}
}
],
"model_devi":[
{
"command": "/home/cloud/deepmd-kit/bin/lmp -i input.lammps -v restart 0",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"remote_root": "/home/cloud/dpgen_example/run/work"
},
"resources": {
"number_node": 1,
"cpu_per_node": 32,
"gpu_per_node": 0,
"group_size": 4
}
}
],
"fp":[
{
"command": "/home/cloud/cp2k-7.1/exe/local/cp2k.psmp -i input.inp",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"remote_root": "/home/cloud/dpgen_example/run/work"
},
"resources": {
"number_node": 1,
"cpu_per_node": 32,
"gpu_per_node": 0,
"group_size": 4
}
}
]
}

parameter.json
{
"type_map": ["H","C"],
"mass_map": [1,12],
"init_data_prefix": "../",
"init_data_sys": ["init/CH4.POSCAR.01x01x01/02.md/sys-0004-0001/deepmd"],
"sys_configs_prefix": "../",
"sys_configs": [
["init/CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale-1.000/00000*/POSCAR"],
["init/CH4.POSCAR.01x01x01/01.scale_pert/sys-0004-0001/scale-1.000/00001*/POSCAR"]
],
"_comment": " that's all ",
"numb_models": 4,
"default_training_param": {
"model": {
"type_map": ["H","C"],
"descriptor": {
"type": "se_a",
"sel": [16,4],
"rcut_smth": 0.5,
"rcut": 5.0,
"neuron": [25,50,100],
"resnet_dt": true,
"axis_neuron": 12,
"seed": 1
},
"fitting_net": {
"neuron": [120,120,120],
"resnet_dt": false,
"seed": 1
}
},
"learning_rate": {
"type": "exp",
"start_lr": 0.001,
"decay_steps": 200
},
"loss": {
"start_pref_e": 0.02,
"limit_pref_e": 1,
"start_pref_f": 1000,
"limit_pref_f": 1,
"start_pref_v": 0.0,
"limit_pref_v": 0.0
},
"training": {
"stop_batch": 40000,
"disp_file": "lcurve.out",
"disp_freq": 1000,
"numb_test": 4,
"save_freq": 1000,
"save_ckpt": "model.ckpt",
"disp_training": true,
"time_training": true,
"profiling": false,
"profiling_file": "timeline.json",
"_comment": "that's all"
}
},
"model_devi_dt": 0.002,
"model_devi_skip": 5,
"model_devi_f_trust_lo": 0.02,
"model_devi_f_trust_hi": 0.15,
"model_devi_e_trust_lo": 10000000000.0,
"model_devi_e_trust_hi": 10000000000.0,
"model_devi_clean_traj": false,
"model_devi_jobs": [
{"sys_idx": [0],"temps": [100],"press": [1.0],"trj_freq": 10,"nsteps": 2000,"ensemble": "nvt","_idx": "00"},
{"sys_idx": [1],"temps": [100],"press": [1.0],"trj_freq": 10,"nsteps": 5000,"ensemble": "nvt","idx": "01"}
],
"ratio_failed": 0.20,
"fp_style": "cp2k",
"shuffle_poscar": false,
"fp_task_max": 30,
"fp_task_min": 1,
"fp_pp_path": "/home/cloud/dpgen_example/run/",
"fp_pp_files": ["GTH_POTENTIALS","BASIS_MOLOPT"],
"fp_params": {
"FORCE_EVAL":{
"DFT":{
"BASIS_SET_FILE_NAME": "BASIS_MOLOPT",
"POTENTIAL_FILE_NAME": "GTH_POTENTIALS"
},
"SUBSYS":{
"KIND":{
"": ["C", "H"],
"POTENTIAL": ["GTH-PBE-q4", "GTH-PBE-q1"],
"BASIS_SET": ["DZVP-MOLOPT-GTH", "DZVP-MOLOPT-GTH"]
}
}
}
}
}

Answered by kyuan2019

Jul 27, 2023

Thanks for all your reply. The error is due to that the initial training data is maybe calculated by VASP and when I try to use cp2k basis set, the energy is different which caused the CH4 to explode.

View full answer

njzjz · 2023-05-30T16:25:01Z

njzjz
May 30, 2023
Maintainer

The error message is the same as deepmodeling/dpdata#458. CC @robinzyb

17 replies

kyuan2019 Jun 7, 2023
Author

Ok. Does that mean this is a bug? How to fix it? Any suggestions! Thanks!

robinzyb Jun 8, 2023
Collaborator

Ok. Does that mean this is a bug? How to fix it? Any suggestions! Thanks!

len(all_sys) is zero, which means all the calculations is not converged. Is it true ?

kyuan2019 Jun 8, 2023
Author

according to the error message below it seems there are 4 runs which worked:
INFO:dpgen:system 001 accurate : 4 in 5000 0.08 %

robinzyb Jun 9, 2023
Collaborator

hello, could you send me the 02.iter directory to robinzhuang@outlook.com

kyuan2019 Jun 9, 2023
Author

sure. Thanks!

robinzyb · 2023-06-13T09:02:44Z

robinzyb
Jun 13, 2023
Collaborator

After examing the files you sent, I found that no cp2k scf converges in the 02.fp.

(base) [iMac](02.fp)$ grep "run NOT converged" task.001.0000*/output
task.001.000000/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000001/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000002/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000003/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000004/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000005/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000006/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000007/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000008/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000009/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000010/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000011/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000012/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000013/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000014/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000015/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000016/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000017/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000018/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000019/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000020/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000021/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000022/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000023/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000024/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000025/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000026/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000027/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000028/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***
task.001.000029/output: *** WARNING in qs_scf.F:542 :: SCF run NOT converged ***

As a result, all_sys is

Data Summary
Labeled System
-------------------
Frame Numbers      : 0
Atom Numbers       : 0
Including Virials  : No
Element List       :
-------------------

You may check the setting of trust_hi and trust_lo to ensure the appropriate selection of candidates
I agree @njzjz on the bug of dealing with zero length, which is fixed by this pr #1245.

1 reply

kyuan2019 Jun 13, 2023
Author

Thanks for checking the file. I followed pr #1245 to modify the run.py code and now I got the follow error:

nohup: ignoring input
INFO:dpgen:failed frame: 30 in 30 100.00 %
Traceback (most recent call last):
File "/home/cloud/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/main.py", line 233, in main
args.func(args)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 5110, in gen_run
run_iter(args.PARAM, args.MACHINE)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4465, in run_iter
post_fp(ii, jdata)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4331, in post_fp
post_fp_cp2k(iter_index, jdata)
File "/home/cloud/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 4228, in post_fp_cp2k
raise RuntimeError(
RuntimeError: find too many unsuccessfully terminated jobs. Too many FP tasks are not converged. Please check your files in directories 'iter../02.fp/task../.'

Indeed, all tasks in 02.fp failed. I checked, for example, the coord.xyz file in task.001.000000 and fund an unrealistic input methane structure (see figure attached). Is this because the trust_hi is too high? It was set at trust_hi = 0.15. I am not sure how can lammps run give such a strange structure from the last interation. Thanks!

robinzyb · 2023-06-14T08:35:27Z

robinzyb
Jun 14, 2023
Collaborator

It may be caused by

the trust_hi too high
over large MD steps at very begining of DP-GEN since the DP potential is broken at first stage.
the element symbol errorly exchanges in the lammps input

6 replies

kyuan2019 Jun 16, 2023
Author

Thanks for your suggestions. I found, however, something is wrong in the training of the iteration.000001.
Here is the an example of one input.json file found in iteration.000001/00.train/000/. It used data.000 from previous iteration.000000 to train the model together the old data.init.

    "training_data": {
        "systems": [
            "../data.init/init/CH4.POSCAR.01x01x01/02.md/sys-0004-0001/deepmd",
            "../data.iters/iter.000000/02.fp/data.000"

however, the error after adding new data in data.000 is very high in the lcurve.out file:

step rmse_trn rmse_e_trn rmse_f_trn lr

0      9.09e+00      1.76e+01      2.27e-01    1.0e-03

1000 4.73e+00 4.08e+00 2.02e-02 7.5e-04
...
40000 7.77e+00 3.47e+00 8.88e-02 1.0e-08

The error is much lower in the iteration 000:
39000 3.05e-03 2.69e-05 3.03e-03 1.3e-08
40000 5.99e-03 2.71e-05 5.96e-03 1.0e-08

I also check most fp runs in data.000 is converged and they look correct to me.

Basically the model generated after training in interation.001 is bad (graph.000.pd, graph.001.pd, and so on). That's why the lammps calculation outputted unrealistic structures, leading to all fp calculations to be non-converging.

Not sure how to fix this.
...

robinzyb Jun 22, 2023
Collaborator

maybe the type_map.raw in "../data.iters/iter.000000/02.fp/data.000" is not consistent with the one in "../data.init/init/CH4.POSCAR.01x01x01/02.md/sys-0004-0001/deepmd"

kyuan2019 Jul 27, 2023
Author

Thanks for all your reply. The error is due to that the initial training data is maybe calculated by VASP and when I try to use cp2k basis set, the energy is different which caused the CH4 to explode.

Answer selected by kyuan2019

Baopidragon1 Dec 25, 2023

Hi, I'm having the same problem running this calculus, do you mean just replace the initial training data with the data converted with cp2k and it will work fine?

robinzyb Dec 25, 2023
Collaborator

Yes. All data should come from the same software.

PhelanShao Jan 9, 2024

I've recently encountered a similar issue and have been considering using init_reaction to generate initial data. However, I'm somewhat perplexed by the instructions and descriptions in the GUI, and despite several attempts, I've been unable to successfully submit an init reaction task. Could you kindly advise if there are particular aspects I should be mindful of when preparing the input files for init_reaction and using the machine command? Your guidance would be greatly appreciated.

Description

Traceback (most recent call last):
File "/home/dell/anaconda3/envs/deepmd/bin/dpgen", line 8, in
sys.exit(main())
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/data/reaction.py", line 233, in gen_init_reaction
run_reaxff(jdata, mdata)
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/data/reaction.py", line 84, in run_reaxff
reaxff_command = "{} -in {}".format(mdata["reaxff_command"], lmp_path)
KeyError: 'reaxff_command'

Description

Traceback (most recent call last):
File "/home/dell/anaconda3/envs/deepmd/bin/dpgen", line 8, in
sys.exit(main())
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/main.py", line 255, in main
args.func(args)
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/data/reaction.py", line 233, in gen_init_reaction
run_reaxff(jdata, mdata)
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/data/reaction.py", line 89, in run_reaxff
make_submission_compat(
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpgen/dispatcher/Dispatcher.py", line 158, in make_submission_compat
submission.run_submission()
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 219, in run_submission
self.generate_jobs()
File "/home/dell/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdispatcher/submission.py", line 468, in generate_jobs
raise RuntimeError("submission must have at least 1 task")
RuntimeError: submission must have at least 1 task

ps: machine.json
{
"api_version": "1.0",
"deepmd_version": "2.0.1",
"reaxff": {
"command": "lmp -in in.CHO",
"machine": {
"batch_type": "Shell",
"local_root": "./",
"remote_root": "/home/dell/dpgen_work",
"context_type": "LocalContext"
},
"resources": {
"number_node": 1,
"cpu_per_node": 4,
"gpu_per_node": 0,
"group_size": 1
}
},
"build": {
"command": "datasetbuilder",
"machine": {
"batch_type": "Shell",
"context_type": "LocalContext",
"local_root": "./",
"remote_root": "/home/dell/dpgen_work",
"clean_asynchronously": false
},
"resources": {
"number_node": 1,
"cpu_per_node": 4,
"gpu_per_node": 0,
"group_size": 1
}
},
"fp": {
"command": "g16 < input || :",
"machine": {
"batch_type": "Shell",
"context_type": "LocalContext",
"local_root": "./",
"remote_root": "/home/dell/dpgen_work",
"clean_asynchronously": false
},
"resources": {
"number_node": 1,
"cpu_per_node": 4,
"gpu_per_node": 0,
"group_size": 1
}
}
}

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error running dpgen cp2k methane example #1237

{{title}}

Replies: 3 comments 24 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

error running dpgen cp2k methane example #1237

kyuan2019 May 30, 2023

Replies: 3 comments · 24 replies

njzjz May 30, 2023 Maintainer

kyuan2019 Jun 7, 2023 Author

robinzyb Jun 8, 2023 Collaborator

kyuan2019 Jun 8, 2023 Author

robinzyb Jun 9, 2023 Collaborator

kyuan2019 Jun 9, 2023 Author

robinzyb Jun 13, 2023 Collaborator

kyuan2019 Jun 13, 2023 Author

robinzyb Jun 14, 2023 Collaborator

kyuan2019 Jun 16, 2023 Author

step rmse_trn rmse_e_trn rmse_f_trn lr

robinzyb Jun 22, 2023 Collaborator

kyuan2019 Jul 27, 2023 Author

Baopidragon1 Dec 25, 2023

robinzyb Dec 25, 2023 Collaborator

PhelanShao Jan 9, 2024

Description

Description

kyuan2019
May 30, 2023

Replies: 3 comments 24 replies

njzjz
May 30, 2023
Maintainer

kyuan2019 Jun 7, 2023
Author

robinzyb Jun 8, 2023
Collaborator

kyuan2019 Jun 8, 2023
Author

robinzyb Jun 9, 2023
Collaborator

kyuan2019 Jun 9, 2023
Author

robinzyb
Jun 13, 2023
Collaborator

kyuan2019 Jun 13, 2023
Author

robinzyb
Jun 14, 2023
Collaborator

kyuan2019 Jun 16, 2023
Author

robinzyb Jun 22, 2023
Collaborator

kyuan2019 Jul 27, 2023
Author

robinzyb Dec 25, 2023
Collaborator