Issue of resubmitting after dpgen task interruption #1878
Replies: 1 comment 30 replies
|
When a dpgen task is interrupted and you want to resume, you typically do NOT need to modify dpgen run param.json machine.jsondpgen automatically reads A few notes:
To reply, just mention @dosu. Docs are dead. Just use Dosu.
|
|
@li-shining The accuracy didn't actually drop — you're now testing the model more stringently. Here's what happened: When you changed from 0.30/0.60 to 0.15/0.35, frames with force deviations between 0.15–0.30 eV/Å that were previously counted as "accurate" are now reclassified as "candidates." The model itself didn't get worse — you just raised the bar for what counts as accurate. How to choose appropriate values for α–Bi₂Mo₃O₁₂: The key guideline is that
For context: Most dpgen examples use 0.05/0.15 for metals and molecules [1] [2]. Your original 0.30/0.60 was quite permissive — it was accepting predictions with large uncertainties as "accurate," which means early iterations may not have been selecting the most informative structures for labeling. My recommendation for your system: Don't switch abruptly. Use a gradual tightening approach:
Jumping straight to 0.15/0.35 from 0.30/0.60 floods the fp step with many candidates at once, which is computationally expensive but will ultimately produce a better model. To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
|
@dosu, Is there something wrong with my settings and results: "learning_rate": { |
|
@li-shining Yes, there are issues. Your force RMSE is not converging properly — it's fluctuating between 0.15–0.70 eV/Å at the end of 600,000 steps across all iterations, with large spikes (e.g., 0.719 at step 594000 in iter.000000, 0.538 at step 600000 in iter.000002). This is problematic for two reasons: Problem 1: Force RMSE is too high relative to your trust levels. Your Problem 2: Training hasn't converged — large fluctuations persist. The wild oscillations (0.15 → 0.70 → 0.21 within a few thousand steps) suggest the model is struggling to fit all your data simultaneously. With 22 Recommended fixes:
The core issue is likely that your model architecture is undersized for 22 diverse oxide systems, or there are data quality issues (outlier frames or DFT inconsistencies) causing the training instability. To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
|
@dosu, I just replaced the methane POSCAR in the example on the official website with a propane POSCAR. I'm worried that after running the init and then the run, the accurate_ratio will always be 0. Why is this happening? Do I need to adjust some other parameters? I checked the lcurve.out file and the results seem fine. step rmse_trn rmse_e_trn rmse_f_trn lrIf there is no available reference data, rmse_*_{val,trn} will print nan1000 3.90e+00 4.24e-02 1.23e-01 1.0e-03 step rmse_trn rmse_e_trn rmse_f_trn lrIf there is no available reference data, rmse_*_{val,trn} will print nan1000 3.90e+00 4.24e-02 1.23e-01 1.0e-03 |
|
@li-shining I can see two likely issues: 1. Your The
For propane (C₃H₈), each atom has at most 2–3 C neighbors but up to 6–8 H neighbors within a 5.0 Å cutoff. You have it inverted — allowing far too many C neighbors and too few H neighbors. This likely degrades your descriptor quality and causes high model deviation during exploration. Fix: Change to 2. Your trust levels may be too strict. The methane example you adapted from uses To debug: Check your Recommended actions:
The To reply, just mention @dosu. Share context across your team and agents. Try Dosu.
|
Uh oh!
There was an error while loading. Please reload this page.
When the dpgen task is interrupted and resubmitted, do I only need to change the record.dpgen file, or do I also need to delete other unfinished files, such as 02.fp, and the files under the work folder?
All reactions