default way of doing long CTSM simulation does not work with CMEPS/CDEPS

[mvertens]

The default way of running a standard CTSM long simulation is the following:

1. Run from 1850-1901 and cycle over the 1901-1920 GSWP3V1 forcing data
/glade/p/cgd/tss/people/oleson/atm_forcing.datm7.GSWP3.0.5d.v1.c200929/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.1901-01.nc to 
/glade/p/cgd/tss/people/oleson/atm_forcing.datm7.GSWP3.0.5d.v1.c200929/TPHWL/clmforc.GSWP3.c2011.0.5x0.5.TPQWL.1920-12.nc
/glade/p/cgd/tss/people/oleson/atm_forcing.datm7.GSWP3.0.5d.v1.c200929/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.1901-01.nc

2. **Restart** (1) and run from 1901-2014 using the 1901-2014 atmospheric data
/glade/p/cgd/tss/people/oleson/atm_forcing.datm7.GSWP3.0.5d.v1.c200929/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.1901-01.nc
to 
/glade/p/cgd/tss/people/oleson/atm_forcing.datm7.GSWP3.0.5d.v1.c200929/Solar/clmforc.GSWP3.c2011.0.5x0.5.Solr.2014-12.nc
and issuing the following xmlchange commands
./xmlchange DATM_YR_ALIGN=1901
./xmlchange DATM_YR_START=1901
./xmlchange DATM_YR_END=2014

The problem is that for cdeps this is really not a restart and the following error occurs when during the read of the datm restart file at model date 19010101:

ERROR: something is wrong
The traceback is:

729:Image              PC                Routine            Line        Source
1729:cesm.exe           00000000013CEAF6  Unknown               Unknown  Unknown
1729:cesm.exe           0000000000EDC600  shr_abort_mod_mp_         114  shr_abort_mod.F90
1729:cesm.exe           0000000000EC3577  dshr_stream_mod_m        1732  dshr_stream_mod.F90
1729:cesm.exe           0000000000E8BE65  dshr_mod_mp_dshr_         941  dshr_mod.F90
1729:cesm.exe           0000000000527B94  datm_datamode_clm         576  datm_datamode_clmncep_mod.F90
1729:cesm.exe           000000000051DAE1  atm_comp_nuopc_mp         583  atm_comp_nuopc.F90
1729:cesm.exe           000000000051FD02  atm_comp_nuopc_mp         402  atm_comp_nuopc.F90

This is due the fact that the restart file expects 240 files in each stream (20 years X 12 months), whereas we are now using 1368 files (114 years X 12 months).

The quick solution was to comment out the following line in dshr_mod.F90
!call shr_stream_restIO(pioid, sdat%stream, 'read')

Another longer term solution needs to be implemented. One idea is to introduce a namelist variable that would skip the reading of the datm restart data if it was turned on. In fact - the simplest thing would have been to comment out the restart read in
datm_datamode_clmncep_mod.F90
based on the namelist variable.

At any rate - this needs to be fixed in a general manner as a high priority in order for CTSM to use cmeps/cdeps as their default data model and coupling architecture.

I am labeling this as a bug since this capability does work in mct.

[ekluzek]

We talked about this in our CTSM software meeting. This is a roadblock in moving away from MCT.

[mvertens]

@ekluzek - thanks for pointing out the criticality of this. I'll work with @jedwards4b to come up with a flexible solution.

[mvertens]

@jedwards4b - I think that adding a namelist variable in cdeps for datm that specifies that restarts should not be read would be simple to do and would solve this problem. Lets talk if you have other ideas or concerns.

[jedwards4b]

What should that variable be called?

[jedwards4b]

@ekluzek ctsm_spinup_mode sound okay?

[mvertens]

What I am suggesting is that a new datm variable - skip_restart_read - be add. The default value would be .false.
When you start step 2 - you could simply add to user_nl_datm skip_restart_read = .true. for the first run phase after 2. and then reset it back to .false.
Or we could have a DATM_SKIP_RESTART_READ xml variable that could be set - like the other ones and you could simply change it with xmlchange at the beginning of step 2.

[ekluzek]

I like @mvertens name better. It's not intrinsic to the CTSM spinup itself. The thing that I need from DATM is to skip reading restarts. That is something that's sometimes useful to do. It may be in other contexts as well.

I think we just need this in the namelist. This is obscure enough that I don't think it needs to be lifted to the XML level.

[mvertens]

@ekluzek - so you would be happy to manually just change this namelist variable in user_nl_datm when you start step2 and then change it back again? This would have to be done manually - but if that's fine its very easy to do this.

[ekluzek]

Yes, I think that's fine. But, I'll let @olyson chime in.

This is an important thing to have in place, but it's still obscure enough that we don't need a super slick way of doing it. It's required for the first step, but turning it back to false is only needed to save some time on the continue run. So good to do, but not strictly required.

[olyson]

If it's not too much more work, I would lean toward an xml variable. It's just a bit more awkward to change the user_nl_datm during the run and maybe not as visible and clear as an xml change. Just my opinion, thanks for doing this.

[mvertens]

@olyson - making it an xml variable is very easy. I agree that it will be clearer with that.