How to restart ECHAM6 standalone simulation?

[QinggangGao]

Dear developers

May I ask how to restart a ECHAM6 standalone simulation? I added following lines to my yaml scripts here /work/ollie/qigao001/a_basic_analysis/d_model_simulations/2_echam6/1.1_pi/pi_echam6_1d_restart.yaml:

lresume: True
ini_parent_exp_id: "pi_echam6_1d_315_4.1"
ini_parent_date:  "2000-01-31"
ini_parent_dir: "/work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1d_315_4.1/restart/echam/"

But the simulation met problems as shown here /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1d_317_4.1. The main error message in the log file is copied below:

0: IO_initial : read tagmap
0: inquire_rerun_file: restart_pi_echam6_1d_317_4.1_jsbach.nc - No such file or directory
0: IO_open failed on: restart_pi_echam6_1d_317_4.1_jsbach.nc
0: IO_open : error status = 2
0: IO_INQ_DIMID: lat
0: IO_INQ_DIMID: NetCDF: Not a valid ID
0:
0: ================================================================================
0:
0: FATAL ERROR in IO_INQ_DIMID: Run terminated.
0: FINISH called from PE: 0

I am using esm_tools version 6.4.0. It also does not work when I add lresume: False in the jsbach section. Why does ECHAM6 still try to find the restart files *.1_jsbach.nc.

Regards, Qinggang

[xshi-awi]

This can be solved by renaming the
/work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1d_315_4.1/restart/jsbach/restart_pi_echam6_1d_315_4.1_20000131235230_.nc files to /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1d_315_4.1/restart/jsbach/restart_pi_echam6_1d_315_4.1_20000131000000_.nc

Note that the time tag 20000131235230=> 20000131000000

[QinggangGao]

Hi Xiaoxu

Thanks. Renaming the file names works.

[pgierz]

Hi @l975421700: we have by now a much newer version; v6.11.4 is the current newest as of this morning. Can you please update and let me know which problems still exist?

[denizural]

Exactly, this bug should have been resolved by now. esm_tools --version will display the installed version.

[QinggangGao]

Hi

I am using the esm_tools, version 6.11.4 now. But it still can not recognize the right file names. One of the warning is like below:

WARNING: File not found: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1y_309_4.3/restart/jsbach/restart_pi_echam6_1y_309_4.3_20001231000000_jsbach.nc

And the error message are the same as above.

My esm_tools is here: /home/ollie/qigao001/esm_tools

[QinggangGao]

WARNING: File not found: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1y_309_4.3/restart/jsbach/restart_pi_echam6_1y_309_4.3_20001231000000_jsbach.nc

Correction: sorry, the warning message is from the terminal rather than the log file.

[denizural]

Hi @l975421700 ,
could you please tell me where you are getting this warning message? In a log file or in your terminal (or its redirection). We will solve this issue today. Thanks.

[QinggangGao]

Hi Deniz

Thanks. I got this in the terminal. The experiment is installed here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test1.
The yaml scripts used is here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/test1/scripts/pi_echam6_1d_restart.yaml.
The experiment is restarted from this experiment: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1y_309_4.3.

After the failure I created symbolic links to have correct file names here: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1y_309_4.3/restart/jsbach. And it works in another restarting experiment /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_echam6_1d_321_4.3.

[denizural]

Hi Qinggang,
Does you simulations run now? We recommend to make a check run before submitting an actual run. Eg.

esm_runscripts -e test myrunscript.yaml --check &>check.out

You can search for the missing files in the check.out and fix them before submitting a job.
For more detailed information you can also add a --verbose option.

[QinggangGao]

Hi Deniz

Thanks a lot for your reply. Yes and no. This problem can be easily solved by manually changing the JSBACH restart file name. But it might be better if esm_tools can detect the correct restart file name.

I did a check run by issuing the command: esm_runscripts /work/ollie/qigao001/a_basic_analysis/d_model_simulations/2_echam6/1.1_pi/pi_echam6_1d_restart.yaml -e test -c.

It still reports missing files like

WARNING: File not found: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_402_4.7/restart/jsbach/restart_pi_m_402_4.7_20001231000000_jsbach.nc

The restart file indeed exists with a different file name: /work/ollie/qigao001/output/echam-6.3.05p2-wiso/pi/pi_m_402_4.7/restart/jsbach/restart_pi_m_402_4.7_20001231**235230**_jsbach.nc.

I am using esm_tools version 6.11.4.

[mandresm]

Hi Qinggang,

The solution for this (at least for the jsbach.nc file) is to include the time T23:52:30(not only the date) in the ini_parent_date in your runscript:

echam:
    [ ... ]
    ini_parent_date:  "2000-12-31T23:52:30"
    [ ... ]

Please, let us know if that solved the problem.

[QinggangGao]

Thanks, I think this solves the problem. When I run in check mode. It does not report missing jsbach restart files as before.