Newer restart file (generated using ctsm5.1.dev026) doesn't work with older version of code (release-clm5.0.34)

[olyson]

The discussion here is initiated from a user post on the CESM Forum:

https://bb.cgd.ucar.edu/cesm/threads/restart-file-error-error-reading-variable-more-than-max-value-size.7616/

One thing I'd like to focus on here is that the simulation is attempting to use an initial file generated from a version of CLM (ctsm5.1.dev026 according to git describe) that is newer than the version that is being used to run the simulation (release-clm5.0.34).

One aspect of the errors that are appearing in the log files of this simulation is large snow balance errors at nstep=0 (there are other errors in the simulation, including an FPE that crashes the model, but it's possible those are related to the snow balance error). I've generated some additional output from the balance check for illustration:

133: WARNING: water balance error nstep= 0 local indexc= 144261
133: errh2o= 2.829681888086977E-003
133: WARNING: snow balance error
133: nstep= 0 local indexc= 144587 col%itype= 410
133: lun%itype= 4 errh2osno= 9900.00000000000
133: snl = -10
133: snow_depth = 25.1053313791244
133: frac_sno_eff = 1.00000000000000
133: h2osno = 10000.0011261617
133: h2osno_old = 100.000000000000

Here, note that h2osno_old is 100 but h2osno is 10,000.
In the newer version of CLM (used to generate the initial file being used), H2OSNO is no longer on the restart file. Since the older code doesn't find H2OSNO, it seems to set h2osno_old to 100 (in clm_instInit I believe). However, the restart file contains information about the snow layers (e.g., snl = -10, snow_depth = 25.1, etc) from the restart file and hence the new H2OSNO appears to be calculated based on that (10,000) (a guess, I haven't looked in the code yet to see where that might be happening).

I'm going to look further to verify this but wanted to post what I have so far to see if anyone thinks this is a reasonable hypothesis.

[billsacks]

I haven't looked at or thought about this carefully, but I do remember reworking how h2osno is stored on restart files. I wouldn't call the situation you describe a backwards compatibility issue: normally we think of that as meaning that newer versions of the code work with older restart files. We don't typically try to ensure that newer restart files will work with older versions of the code, so I'm not surprised that there would be problems with doing that.

[olyson]

Ok, I changed the title to make it more clear, hopefully.
I did verify that the finidat_interp_dest.nc file that is generated has values for H2OSNO of either 100 or 0, which is what causes the snow balance errors, because it is inconsistent with the other snow information in the file.
I tried overwriting that H2OSNO with the sum of h2osoi_liq and h2osoi_ice over the snow layers using the information in finidat_interp_dest.nc. That solved the snow balance errors. However, there are some water balance errors that stop the model (largest seems to be around E-02).
I'm not sure what to recommend to the user, would appreciate any suggestions.

[billsacks]

From ctsm-software discussion: We feel that, in general, we can't commit to supporting people trying to use a newer restart file in older versions of the code: this could be very time consuming. Is there a reason they can't use a compatible restart file, or use a newer code version? If that's not possible, we should probably suggest they just do their own spinup.