Skip to content

Commit

Permalink
Quartet of bug fixes for: c3 scheme, quilting restart with 32-bit phy…
Browse files Browse the repository at this point in the history
…sics, and string length mismatch in dycore (plus PR #1913, #1917, and #1926) (#1893)

* GFDL_atmos_cubed_sphere: consistent string lengths in array

* stop FV3_HRRR_c3 from crashing with gnu debug

* 1hr forecast limit for conus13km_debug_qr

* fv3atm: bug fix from Dusan to recover_fields crash

* disable conus13km_debug_qr_gnu due to 25% failure rate on Hera

* FV3 dycore: initialize srf_wnd_var2 and tracers_var3 arrays

* enable conus13km_debug_qr_gnu

* Fix race condition in GFS_phys_time_vary.fv3.F90 error detection

* More bug fixes to GFS_phys_time_vary.fv3.F90:
1. detect empty errmsg from subroutines
2. Initialize err variables in set_soilveg.f, which is called from GFS_phys_time_vary.fv3.F90

* ccpp-physics: initialize errmsg & errflg in noahmp_tables.f90

* ccpp-physics: only read h2odata, ozdata and noahmp table when they are needed

* "point to the dimension fix of smc for dust emission"

* FV3: more dycore bug fixes from GFDL_atmos_cubed_sphere PR 285

* merge #1926

* merge GFDL_atmos_cubed_sphere #276

* bugfix: 12hr hrrr tests

* add GAEA rocoto support

* fv3: merge GF radar fixes

* check that baseline directory exists and is non-empty

---------

Co-authored-by: Haiqin.Li <Haiqin.Li@noaa.gov>
  • Loading branch information
SamuelTrahanNOAA and haiqinli committed Oct 3, 2023
1 parent 83e477f commit 41811b8
Show file tree
Hide file tree
Showing 25 changed files with 11,497 additions and 11,691 deletions.
2 changes: 1 addition & 1 deletion tests/bl_date.conf
Original file line number Diff line number Diff line change
@@ -1 +1 @@
export BL_DATE=20230922
export BL_DATE=20231002
1 change: 1 addition & 0 deletions tests/default_vars.sh
Original file line number Diff line number Diff line change
Expand Up @@ -1327,6 +1327,7 @@ export OUTPUT_GRID='gaussian_grid'
export NSTF_NAME='2,0,0,0,0'
export WRITE_DOPOST=.true.
export IAER=5111
export FHMAX=12

export FRAC_GRID=.false.
export FRAC_ICE=.true.
Expand Down
54 changes: 27 additions & 27 deletions tests/logs/OpnReqTests_control_p8_hera.log
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
Thu Sep 28 06:19:01 UTC 2023
Thu Sep 28 21:48:32 UTC 2023
Start Operation Requirement Test

Compile bit elapsed time 200 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -DMPI=ON -DCMAKE_BUILD_TYPE=Release
Compile dbg elapsed time 104 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DDEBUG=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Debug
Compile std elapsed time 199 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release
Compile bit elapsed time 204 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -DMPI=ON -DCMAKE_BUILD_TYPE=Release
Compile dbg elapsed time 107 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DDEBUG=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Debug
Compile std elapsed time 205 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release

baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_bit_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_bit_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_bit_base
Checking test bit_base control_p8_gnu results ....
Moving baseline bit_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -54,14 +54,14 @@ Moving baseline bit_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 296.142721
0: The maximum resident set size (KB) = 1288112
0: The total amount of wall time = 296.374253
0: The maximum resident set size (KB) = 1292332

Test bit_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_dbg_base
Checking test dbg_base control_p8_gnu results ....
Moving baseline dbg_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -109,14 +109,14 @@ Moving baseline dbg_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 981.239219
0: The maximum resident set size (KB) = 1281056
0: The total amount of wall time = 993.563417
0: The maximum resident set size (KB) = 1282092

Test dbg_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_dcp
Checking test dcp control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -163,14 +163,14 @@ Checking test dcp control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 263.993791
0: The maximum resident set size (KB) = 1271608
0: The total amount of wall time = 262.808211
0: The maximum resident set size (KB) = 1270520

Test dcp control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_mpi
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_mpi
Checking test mpi control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -217,14 +217,14 @@ Checking test mpi control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 261.821420
0: The maximum resident set size (KB) = 1272220
0: The total amount of wall time = 268.679158
0: The maximum resident set size (KB) = 1272352

Test mpi control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_rst
Checking test rst control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -271,14 +271,14 @@ Checking test rst control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 265.397416
0: The maximum resident set size (KB) = 1272888
0: The total amount of wall time = 260.067262
0: The maximum resident set size (KB) = 1274240

Test rst control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_std_base
Checking test std_base control_p8_gnu results ....
Moving baseline std_base control_p8_gnu files ....
Moving sfcf000.nc .........OK
Expand Down Expand Up @@ -326,14 +326,14 @@ Moving baseline std_base control_p8_gnu files ....
Moving RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Moving RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 264.007621
0: The maximum resident set size (KB) = 1272592
0: The total amount of wall time = 263.023352
0: The maximum resident set size (KB) = 1271516

Test std_base control_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/control_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_14060/control_p8_gnu_thr
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_27762/control_p8_gnu_thr
Checking test thr control_p8_gnu results ....
Comparing sfcf000.nc .........OK
Comparing sfcf021.nc .........OK
Expand Down Expand Up @@ -380,11 +380,11 @@ Checking test thr control_p8_gnu results ....
Comparing RESTART/20210323.060000.sfc_data.tile5.nc .........OK
Comparing RESTART/20210323.060000.sfc_data.tile6.nc .........OK

0: The total amount of wall time = 264.748647
0: The maximum resident set size (KB) = 1271232
0: The total amount of wall time = 269.517142
0: The maximum resident set size (KB) = 1271620

Test thr control_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Thu Sep 28 06:43:18 UTC 2023
Elapsed time: 00h:24m:18s. Have a nice day!
Thu Sep 28 22:11:54 UTC 2023
Elapsed time: 00h:23m:23s. Have a nice day!
26 changes: 13 additions & 13 deletions tests/logs/OpnReqTests_cpld_control_nowave_noaero_p8_hera.log
Original file line number Diff line number Diff line change
@@ -1,11 +1,11 @@
Thu Sep 28 06:24:59 UTC 2023
Thu Sep 28 21:50:19 UTC 2023
Start Operation Requirement Test

Compile dbg elapsed time 118 seconds. -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DDEBUG=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Debug -DMOM6SOLO=ON
Compile std elapsed time 234 seconds. -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DMPI=ON -DCMAKE_BUILD_TYPE=Release -DMOM6SOLO=ON
Compile std elapsed time 237 seconds. -DAPP=S2SW -DCCPP_SUITES=FV3_GFS_v17_coupled_p8 -DMPI=ON -DCMAKE_BUILD_TYPE=Release -DMOM6SOLO=ON

baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_dbg_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_25055/cpld_control_nowave_noaero_p8_gnu_dbg_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_29412/cpld_control_nowave_noaero_p8_gnu_dbg_base
Checking test dbg_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -68,14 +68,14 @@ Moving baseline dbg_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 1338.997302
0: The maximum resident set size (KB) = 1411472
0: The total amount of wall time = 1344.688199
0: The maximum resident set size (KB) = 1416032

Test dbg_base cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_25055/cpld_control_nowave_noaero_p8_gnu_rst
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_29412/cpld_control_nowave_noaero_p8_gnu_rst
Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing sfcf021.tile1.nc .........OK
Comparing sfcf021.tile2.nc .........OK
Expand Down Expand Up @@ -137,14 +137,14 @@ Checking test rst cpld_control_nowave_noaero_p8_gnu results ....
Comparing RESTART/iced.2021-03-23-21600.nc .........OK
Comparing RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 406.961254
0: The maximum resident set size (KB) = 1404968
0: The total amount of wall time = 414.121734
0: The maximum resident set size (KB) = 1398040

Test rst cpld_control_nowave_noaero_p8_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/cpld_control_c96_noaero_p8_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_25055/cpld_control_nowave_noaero_p8_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_29412/cpld_control_nowave_noaero_p8_gnu_std_base
Checking test std_base cpld_control_nowave_noaero_p8_gnu results ....
Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving sfcf021.tile1.nc .........OK
Expand Down Expand Up @@ -207,11 +207,11 @@ Moving baseline std_base cpld_control_nowave_noaero_p8_gnu files ....
Moving RESTART/iced.2021-03-23-21600.nc .........OK
Moving RESTART/ufs.cpld.cpl.r.2021-03-23-21600.nc .........OK

0: The total amount of wall time = 410.847589
0: The maximum resident set size (KB) = 1403172
0: The total amount of wall time = 412.414378
0: The maximum resident set size (KB) = 1384124

Test std_base cpld_control_nowave_noaero_p8_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Thu Sep 28 06:54:27 UTC 2023
Elapsed time: 00h:29m:29s. Have a nice day!
Thu Sep 28 22:19:36 UTC 2023
Elapsed time: 00h:29m:18s. Have a nice day!
26 changes: 13 additions & 13 deletions tests/logs/OpnReqTests_regional_control_hera.log
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
Thu Sep 28 06:30:35 UTC 2023
Thu Sep 28 21:52:18 UTC 2023
Start Operation Requirement Test

Compile std elapsed time 201 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release
Compile std elapsed time 204 seconds. -DAPP=ATM -DCCPP_SUITES=FV3_GFS_v16,FV3_GFS_v16_flake,FV3_GFS_v17_p8,FV3_GFS_v17_p8_rrtmgp,FV3_GFS_v15_thompson_mynn_lam3km,FV3_WoFS_v0,FV3_GFS_v17_p8_mynn -D32BIT=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release

baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_4490/regional_control_gnu_dcp
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_31058/regional_control_gnu_dcp
Checking test dcp regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -15,14 +15,14 @@ Checking test dcp regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 513.747809
0: The maximum resident set size (KB) = 804952
0: The total amount of wall time = 513.311871
0: The maximum resident set size (KB) = 812328

Test dcp regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_4490/regional_control_gnu_std_base
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_31058/regional_control_gnu_std_base
Checking test std_base regional_control_gnu results ....
Moving baseline std_base regional_control_gnu files ....
Moving dynf000.nc .........OK
Expand All @@ -34,14 +34,14 @@ Moving baseline std_base regional_control_gnu files ....
Moving NATLEV.GrbF00 .........OK
Moving NATLEV.GrbF06 .........OK

0: The total amount of wall time = 505.577903
0: The maximum resident set size (KB) = 805676
0: The total amount of wall time = 515.195176
0: The maximum resident set size (KB) = 807004

Test std_base regional_control_gnu PASS


baseline dir = /scratch1/NCEPDEV/stmp4/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/OPNREQ_TEST/regional_control_std_base_gnu
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_4490/regional_control_gnu_thr
working dir = /scratch1/NCEPDEV/stmp2/Fernando.Andrade-maldonado/FV3_OPNREQ_TEST/opnReqTest_31058/regional_control_gnu_thr
Checking test thr regional_control_gnu results ....
Comparing dynf000.nc .........OK
Comparing dynf006.nc .........OK
Expand All @@ -52,11 +52,11 @@ Checking test thr regional_control_gnu results ....
Comparing NATLEV.GrbF00 .........OK
Comparing NATLEV.GrbF06 .........OK

0: The total amount of wall time = 516.790745
0: The maximum resident set size (KB) = 807676
0: The total amount of wall time = 508.059579
0: The maximum resident set size (KB) = 807656

Test thr regional_control_gnu PASS

OPERATION REQUIREMENT TEST WAS SUCCESSFUL
Thu Sep 28 06:59:22 UTC 2023
Elapsed time: 00h:28m:48s. Have a nice day!
Thu Sep 28 22:19:53 UTC 2023
Elapsed time: 00h:27m:36s. Have a nice day!
Loading

0 comments on commit 41811b8

Please sign in to comment.