Failing ne0CONUSne30x8_ne0CONUSne30x8_mt12 in CESM testing

[ekluzek]

Brief summary of bug

@fischer-ncar found the following test to be failing in cesm2_3_alpha17f testing:

PEND SMS_D_Ln9_P1280x1.ne0CONUSne30x8_ne0CONUSne30x8_mt12.FCnudged.derecho_intel.cam-outfrq9s SHAREDLIB_BUILD

General bug information

CTSM version you are using: ctsm5.2.005

Does this bug cause significantly incorrect results in the model's science? No
Configurations affected: CONUS VR grid for FCnudged compset

Details of bug

Important output or errors that show the problem

err=ERROR : CLM build-namelist::CLMBuildNamelist::setup_logic_initial_conditions() : use_init_interp is NOT synchronized with init_interp_attributes in the namelist_defaults file, this should be corrected there'

Definition of done:

• Test SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam6LndTuningMode_2013Start to show it fails
• Change CONUS test in build-namelist test to do a transient use-case
• Make sure we have f19 tests in build-namelist tester for transient
• Make sure the namelist testing is doing transient tests with the correct start dates for the datasets we have for 1979-PD.
• Run build-namelist tests to show those new tests fail
• Fix use_init_interp for CLM50 so it works
• Fix use_init_interp for f19
• Show that build-namelist tests now pass
• Test SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam6LndTuningMode_2013Start to show it now works
• @ekluzek test these to show that they work
  SMS_D_Ln9.f19_f19_mg17.FXHIST.derecho_intel.cam-outfrq9s_amie
  SMS_D_Ln9_P1280x1.ne0CONUSne30x8_ne0CONUSne30x8_mt12.FCnudged.derecho_intel.cam-outfrq9s

[ekluzek]

We talked about this in CSEG and decided since this is only for a less used VR grid (and there is a namelist fix for it) that we would do this after the remove externals beta tag. @cacraigucar @adamrher

The namelist fix is to just add the following to user_nl_clm:

use_init_interp = .true.

I have NOT explicitly tested the above, but do believe it will work. If someone actually tries this -- let us know if it works.

[adamrher]

We talked about this in CSEG and decided since this is only for a less used VR grid (and there is a namelist fix for it) that we would do this after the remove externals beta tag. @cacraigucar @adamrher

Fine by me. But for the record, this is probably the most used VR grid. But lately we have been more focused on the 1deg workhorse and so we haven't really been running the VR grids.

[ekluzek]

This is the same problem as #2520 and again is a result of the fragility issues identified in #2169.

FCnudged is a HIST compset currently with CLM50%SP that starts in 2013. The following test was run and PASSED in ctsm5.2.005 and looks to me like it's almost identical (other than using CLM60 rather than CLM50).

SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm60Sp.derecho_intel.clm-clm60cam6LndTuningMode_2013Start

The build-namelist testing does NOT cover it because it's not doing a transient case for is only 1850 and 2000 control. So we should add a transient test in the namelist testing as well.

[slevis-lmwg]

As expected (see checklist at top of this issue):

• ./create_test SMS_Ln9.ne0CONUSne30x8_ne0CONUSne30x8_mt12.IHistClm50Sp.derecho_intel.clm-clm50cam6LndTuningMode_2013Start fails with
  CLMBuildNamelist::setup_logic_initial_conditions() : use_init_interp is NOT synchronized with init_interp_attributes in the namelist_defaults file
• ./build-namelist_test.pl fails in

916/3997 < FAIL> <Test Id: 916> <Desc: -res ne0np4CONUS.ne30x8 -bgc sp -use_case 20thC_transient -namelist '&a start_ymd=20130101/' -lnd_tuning_mode clm4_5_cam7.0 -phys clm4_5: lnd_in file exists>
949/3997 < FAIL> <Test Id: 949> <Desc: -res ne0np4CONUS.ne30x8 -bgc sp -use_case 20thC_transient -namelist '&a start_ymd=20130101/' -lnd_tuning_mode clm5_0_cam7.0 -phys clm5_0: lnd_in file exists>

[slevis-lmwg]

Further updates appear in the PR.

[brian-eaton]

I have just updated cam6_4_038 to use the same externals as cesm3_0_alpha03d (uses ctsm5.3.002). In addition to the test mentioned in this issue, I now get the same failure from test SMS_D_Ln9.f19_f19_mg17.FXHIST.derecho_intel.cam-outfrq9s_amie

[ekluzek]

@brian-eaton thanks for the update on the f19 test. Another part on this is that we are bringing in a 16-pft finidat file f19 for the PPE work, that will help with that particular problem.

[ekluzek]

@slevis-lmwg I tried the F compset as well as the basic I compset and they both failed for me on the cesm3_0_beta04_changes branch for the CONUS grid. Can you look at the branch again and see what happened? There are many things that could have happened here.

Thanks.