how to check that init-model or init-frz-model are working correctly

[damonturney]

I'm attempting to use init-model or init-frz-model to restart training, but it doesn't seem to work.
In my initial folder, I have my initial input.json file (pasted below) and the training files:
{
"model": {
"type_map": [ "O", "H" ],
"descriptor": { "type": "se_e2_a", "sel": [4,9],
"rcut_smth": 1.0, "rcut": 6.0,
"neuron": [ 25, 50, 100 ],
"resnet_dt": false, "axis_neuron": 12, "seed": 327201647
},
"fitting_net": {
"neuron": [ 240, 240, 240 ], "resnet_dt": true, "seed": 734495905
}
},
"learning_rate": {
"type": "exp", "start_lr": 0.001, "decay_steps": 400, "stop_lr": 4e-08
},
"loss": {
"start_pref_e": 0.01, "limit_pref_e": 1, "start_pref_f": 1000, "limit_pref_f": 1,
"start_pref_v": 0, "limit_pref_v": 0
},
"training": {
"disp_file": "lcurve.out", "disp_freq": 100, "save_freq": 1000,
"disp_training": true, "time_training": true, "profiling": false,
"numb_steps": 100000, "seed": 635789808,
"training_data": {
"set_prefix": "set",
"batch_size": "auto",
"systems": [
"1200_K_500steps_config1/data",
"1200_K_500steps_config2/data",
"1200_K_500steps_config3/data",
"1200_K_500steps_config4/data",
"1200_K_500steps_config5/data",
"1200_K_500steps_config6/data",
"250_K_500steps_config1/data",
"250_K_500steps_config2/data",
"250_K_500steps_config3/data",
"250_K_500steps_config4/data",
"250_K_500steps_config5/data",
"250_K_500steps_config6/data",
"300_K_500steps_config1/data",
"300_K_500steps_config2/data",
"300_K_500steps_config3/data",
"300_K_500steps_config4/data",
"300_K_500steps_config5/data",
"300_K_500steps_config6/data",
"375_K_500steps_config1/data",
"375_K_500steps_config2/data",
"375_K_500steps_config3/data",
"375_K_500steps_config4/data",
"375_K_500steps_config5/data",
"375_K_500steps_config6/data",
"500_K_500steps_config1/data",
"500_K_500steps_config2/data",
"500_K_500steps_config3/data",
"500_K_500steps_config4/data",
"500_K_500steps_config5/data",
"500_K_500steps_config6/data",
"800_K_500steps_config1/data",
"800_K_500steps_config2/data",
"800_K_500steps_config3/data",
"800_K_500steps_config4/data",
"800_K_500steps_config5/data",
"800_K_500steps_config6/data"
]
}
}
}

The 1st folder (initial) training goes well for 100000 training steps, and produces a frozen_model.pb that works nicely in LAMMPS. But I want to improve the model so I produced additional (secondary) training files to add.

To restart training, after training the first folder (above) I copied its frozen_model.pb (and also its model.chkpt* files) into a separate folder (my 2nd folder) which contained it's own input.json file (pasted below) and also the secondary (additional) training files:
{
"model": {
"type_map": [ "O", "H" ],
"descriptor": { "type": "se_e2_a", "sel": [4,9],
"rcut_smth": 1.0, "rcut": 6.0,
"neuron": [ 25, 50, 100 ],
"resnet_dt": false, "axis_neuron": 12, "seed": 327201647
},
"fitting_net": {
"neuron": [ 240, 240, 240 ], "resnet_dt": true, "seed": 734495905
}
},
"learning_rate": {
"type": "exp", "start_lr": 0.001, "decay_steps": 400, "stop_lr": 4e-08
},
"loss": {
"start_pref_e": 0.01, "limit_pref_e": 1, "start_pref_f": 1000, "limit_pref_f": 1,
"start_pref_v": 0, "limit_pref_v": 0
},
"training": {
"disp_file": "lcurve.out", "disp_freq": 100, "save_freq": 1000,
"disp_training": true, "time_training": true, "profiling": false,
"numb_steps": 100000, "seed": 635789808,
"training_data": {
"set_prefix": "set",
"batch_size": "auto",
"systems": [
"250_K_ConfigFromLAMMPS_1200_K_config1_step100/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step130/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step160/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step190/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step220/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step250/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step280/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step310/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step340/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step370/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step400/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step430/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step460/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step520/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step580/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step610/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step640/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step670/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step700/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step730/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step760/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step820/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step850/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step880/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step910/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step940/data",
"250_K_ConfigFromLAMMPS_1200_K_config1_step970/data",
"250_K_ConfigFromLAMMPS_1200_K_config2_step100/data",
"250_K_ConfigFromLAMMPS_1200_K_config2_step1000/data"
]
}
}
}

But if I execute
dp train training_config.json --init-frz-model frozen_model.pb
or
dp train training_config.json --init-model model.chkpt
in the 2nd folder it generates a lcurve.out file (and model.chkpt-***** files) that start back at 000000 as if there was never any initial training of the model. I was expecting lcurve.out and model.chkpt-***** to start at 100000 because my 1st training had 100000 steps. Also I get a nonsense frozen_model.pb from the 2nd training that creates erroneous LAMMPS simulations. Am I making a mistake? Should the 2nd training produce lcurve.out file and model.chkpt-***** files?

[njzjz]

You should use restart to restart training.

[damonturney]

But in this post (#4056) you say that if the input.json file has new source files then it's better to use init-model.
Which is correct? init-model or restart?
And if I'm using a new folder in which to execute the second round of training, which files (model.chkpt* and others?) do I need to copy over from the first folder?

Also, when I create my second input.json file with the new source files, should I remove the initial source files from the input.json file list? Or keep them there along with the new source files?

[njzjz]

init-model starts a new round of training, and restart restarts training. If new data is added, a new round of training is suggested. Otherwise, new data is not trained with a high learning rate.
Each round of training should be executed in a separate directory.

[damonturney]

Thanks, I'll use init-model! In my second training folder's input.json file (in which I list the new data source files), should I remove the initial source files from the .json list of source files?

[njzjz]

Usually no, if old data shouldn't be forgotten by the model.