b.e23_alpha16g.BLT1850.ne30_t232.076

[cecilehannay]

Purpose:
Retuned version of the warm branch with taus code.

---

Description:
Same as #461 but with

micro_mg_vtrmi_factor = 1.5
micro_mg_dcs = 0.0005

---

Case directory:

• Locally (if still available):
  /glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16g.BLT1850.ne30_t232.076

• On github:
  https://github.com/NCAR/amwg_dev/tree/b.e23_alpha16g.BLT1850.ne30_t232.076

---

Sandbox:

• Locally (if still available): /glade/work/hannay/cesm_tags/cesm2_3_alpha16g

• On github: https://github.com/ESCOMP/cesm/tree/cesm2_3_alpha16g

---

Diagnostics:

• Diags (if available):
  https://webext.cgd.ucar.edu/BLT1850/b.e23_alpha16g.BLT1850.ne30_t232.076/

---

Output:

• Raw output on derecho (if still available): /glade/derecho/scratch/hannay/archive/b.e23_alpha16g.BLT1850.ne30_t232.076/
• CAM Climos: /glade/campaign/cgd/amp/amwg/climo/b.e23_alpha16g.BLT1850.ne30_t232.076

---

Contacts:
@JulioTBacmeister @cecilehannay @adamrher @PeterHjortLauritzen @vlarson @zarzycki

---

Extra details:

[cecilehannay]

I plot the diags for that run although it didn't reach the 20-year yet. It is better balanced than "74" (#461) but the SWCF and LWCF seems on the very low side.

[cecilehannay]

This run is crashing. It seems to be in the radiation:

dec0066.hsn.de.hpc.ucar.edu 249:  Negative conc. in ch4tran. c,j,deficit (mol):          63           2                                                                                                                   
dec0066.hsn.de.hpc.ucar.edu 249:   1.131059372120394E-003                                                                                                                                                                 
dec0073.hsn.de.hpc.ucar.edu 1115: forrtl: severe (174): SIGSEGV, segmentation fault occurred                                                                                                                              
dec0073.hsn.de.hpc.ucar.edu 1115: Image              PC                Routine            Line        Source                                                                                                              
dec0073.hsn.de.hpc.ucar.edu 1115: libpthread-2.31.s  0000146E405C58C0  Unknown               Unknown  Unknown                                                                                                             
dec0073.hsn.de.hpc.ucar.edu 1115: cesm.exe           0000000000FA39E0  rrtmg_sw_spcvmc_m         570  rrtmg_sw_spcvmc.f90                                                                                                 
dec0073.hsn.de.hpc.ucar.edu 1115: cesm.exe           0000000000F9817E  rrtmg_sw_rad_mp_r         497  rrtmg_sw_rad.f90                                                                                                    
dec0073.hsn.de.hpc.ucar.edu 1115: cesm.exe           00000000008367FE  radsw_mp_rad_rrtm         515  radsw.F90                                                                                                           
dec0073.hsn.de.hpc.ucar.edu 1115: cesm.exe           0000000000816763  radiation_mp_radi        1212  radiation.F90  

I could perturb and see if we pass the crashing point.

[cecilehannay]

I got another in 76 around year 40:

dec2353.hsn.de.hpc.ucar.edu 1138: SHR_REPROSUM_CALC: Input contains  0.27000E+02 NaNs and  0.00000E+00 INFs on process    1138
dec2463.hsn.de.hpc.ucar.edu 1535: SHR_REPROSUM_CALC: Input contains  0.27000E+02 NaNs and  0.00000E+00 INFs on process    1535
dec2464.hsn.de.hpc.ucar.edu 1632: SHR_REPROSUM_CALC: Input contains  0.18000E+02 NaNs and  0.00000E+00 INFs on process    1632
dec2319.hsn.de.hpc.ucar.edu 610: SHR_REPROSUM_CALC: Input contains  0.27000E+02 NaNs and  0.00000E+00 INFs on process     610
dec2353.hsn.de.hpc.ucar.edu 1138:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
dec2463.hsn.de.hpc.ucar.edu 1535:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
dec2464.hsn.de.hpc.ucar.edu 1632:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
dec2319.hsn.de.hpc.ucar.edu 610:  ERROR: shr_reprosum_calc ERROR: NaNs or INFs in input
dec2342.hsn.de.hpc.ucar.edu 990: Image              PC                Routine            Line        Source             
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           0000000003E84FBD  shr_abort_mod_mp_         114  shr_abort_mod.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           0000000003F59768  shr_reprosum_mod_         482  shr_reprosum_mod.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           00000000013B0C1B  gmean_mod_mp_gmea         279  gmean_mod.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           0000000001242811  check_energy_mp_c         552  check_energy.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           000000000078E45F  physpkg_mp_phys_r        1096  physpkg.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           00000000005944B6  cam_comp_mp_cam_r         260  cam_comp.F90
dec2342.hsn.de.hpc.ucar.edu 990: cesm.exe           0000000000572344  atm_comp_nuopc_mp        1157  atm_comp_nuopc.F90

[bstephens82]

Thanks @cecilehannay. I have seen the SHR_REPROSUM_CALC error before but I personally don't understand it although I have taken it as a sign of instability. I should have a new parameter set to try by the end of the week. I have at least 2 that have gone for 16+ years without any issues.