short term archiving fails on maint-1.0

[tangq]

Short term archiving script fails to move the history files, but the restart files are moved correctly. I used the code of maint-1.0 for some additional DECK runs on Edison.

The $rundir is /global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_A2.ne30_oEC.edison/run

$casedir: /global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_A2.ne30_oEC.edison/case_scripts

[rljacob]

Can you give more information on how to reproduce the problem? Did you run for some number of years and then executed "case.st_archive" after the run was finished? Any error messages?

[tangq]

I ran ./case.st_archive --last-date 1905-01-01 --force-move --no-incomplete-logs > st.txt & in the case directory when the model completed 1905-01-01.

The log file is at /global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_A2.ne30_oEC.edison/case_scripts/st.txt. You can see some warnings and errors in the st.txt file.

[tangq]

@rljacob , I need the short term archiving function to examine whether some ongoing runs are working as expected for the DECK overview paper @golaz is writing. Is there an alternative way to run the short term archiving script before it is fixed?

[rljacob]

Try waiting until the run is finished and just running "./case.st_archive" with no arguments.
If that doesn't work, run it again and add the --debug option. This will make a case.st_archive.log file with more info.

[tangq]

By "finished" do you mean the completion of one submission or the whole simulation? The simulations are for a few hundred years, I would prefer to run ./case_st_archive --debug after this submission is completed.

[rljacob]

completion of one submission.

[tangq]

@rljacob , I created a 5-day test at Edison:/global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_tst_st_archive.ne30_oEC.edison

I saved daily atmospheric output and daily restart files. After the 5-day run, I ran ./case.st_archive --debug > st.txt. It reproduces the issue: restart files were moved to archive directory, but the history files were left in the run directory.

The debug log file is at /global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_tst_st_archive.ne30_oEC.edison/case_scripts/case.st_archive.log

Hopefully, this test log is helpful.

[jgfouca]

@tangq , could you describe how you created this case? I'd like to try to duplicate this myself.

[tangq]

@jgfouca, You can find the run script I used to create this case at ~/tang30/ACME_scripts/tst_st_archive.csh

You should be able to duplicate it by modifying a few lines on the top session of this script. Let me know if you have any problems.

[jedwards4b]

If you are up to date with the cime version you can run ./case.st_archive --test-case and it should show you all of the files it's trying to handle and the disposition of each

[tangq]

The simulation cannot use the current master, as it is an additional DECK run.

[jedwards4b]

I can't read any of your directories on edison. I didn't mean to imply that you should update cime, only if it was a new enough version to have that feature. I can't tell you if it is since I can't read the directory.

[tangq]

@jedwards4b, are you in group acme on edison. My directory is group readable.

[tangq]

I changed my test run's permission (/global/cscratch1/sd/tang30/ACME_simulations/20180622.DECKv1b_tst_st_archive.ne30_oEC.edison). You should be able to read it now.

[rljacob]

This is the maint-1.0 branch of E3SM which is using cime5.5.0

[jedwards4b]

So the entries in env_archive are wrong - can you try replacing the hist_file_extension entries for clm and cam with these and see if it solves the problem:
For cam

 <hist_file_extension>h\d*.*\.nc$</hist_file_extension>
    <hist_file_extension>e</hist_file_extension>

for clm:

[jedwards4b]

For clm:

  <hist_file_extension>h\d*.*\.nc$</hist_file_extension>
    <hist_file_extension>e</hist_file_extension>

[jedwards4b]

I'm just using the values from the latest config_archive.xml file (but you can't just use the latest file since it isn't backward compatible)

[tangq]

It worked after I changed the env_archive.xml file as you suggested. I will use this manual fix for now.

[rljacob]

@jedwards4b thanks for finding the cause and the workaround. @golaz, didn't you use case.st_archive in your DECK runs without any problems?

[tangq]

Can we have a PR to fix this issue on maint-1.0? There will be more DECK runs that will be done with maint-1.0.

I don't know which file to modify to change the settings in env_archive.xml. Otherwise, I will create the PR. Thanks.

[rljacob]

I'm working on this. But also trying to figure out what happened.

The branch used for most of the v1.0 runs had for the hist_file_extension:
<hist_file_extension>\.h.*.nc$</hist_file_extension>

But right before actually making the v1.0 tag, we brought in cime5.5.0 which has this:
<hist_file_extension>[eh]</hist_file_extension>

And the correct version given by JimE above is a third version.

[jedwards4b]

There was insufficient testing of the archiving capability and changes in the xml file were made with little conformation of their validity. The final version was added after testing for st_archive functionality was added to cime.