Update GOME processing in read_ozone.f90

[RussTreadon-NOAA]

Starting 20240131 18z the GOME bufr mnemonic DOYR replaced MNTH DAYS. This requires a corresponding change in src/gsi/read_ozone.f90. Specifically, lines 198 - 199 in read_ozone.f90 need to change from

  data ozgstr &
       / 'SAID CLAT CLON YEAR DOYR HOUR MINU SECO SOZA SOLAZI' /

to

  data ozgstr &
       / 'SAID CLAT CLON YEAR MNTH DAYS HOUR MINU SECO SOZA SOLAZI' /

This change, in turn, necessitates several additional changes in read_ozone.f90

• increase size of hdrozg from 10 to 11
• modify how idate5 is loaded
• change select hdrozg indices

This issue is opened to document implementation of the above changes to read_ozone.f90.

[RussTreadon-NOAA]

FYI @CatherineThomas-NOAA
We missed updating GSI processing of GOME ozone data when NESDIS changed mnemonics in late January 2024. We do not assimilate GOME ozone data. We only monitor it so the failure to update the operational and develop GSI does not impact analysis results. Nonetheless, we should make the necessary correction in GSI develop. Of course, doing so means the updated GSI will be unable to process GOME data prior to 20240131 18Z.

[RussTreadon-NOAA]

Tagging @ilianagenkova and @HaixiaLiu-NOAA for awareness.

[RussTreadon-NOAA]

After discussion with @CatherineThomas-NOAA it is clear that we need to fix read_ozone.f90 whether or not GOME ozone data is assimilated. NCO will run GFS v17 gsi.x in debug mode as part of their IT testing. If gsi.x is run for a case after 20240131 18Z with the GOME bufr dump file present, gsi.x will abort. This is not acceptable.

RussTreadon-NOAA/GSI:feature/gome has been created to update read_ozone.f90 to work with GOME dump files before and after the 20240131 18Z change.

[RussTreadon-NOAA]

@ilianagenkova suggested adding logic to read_ozone.f90 which keys off mnemonic ORBN. The mnemonic is in GOME bufr dump files prior to 20240131 18Z. This mnemonic is not present after this date.

First attempt at implementing this idea committed to feature/gome at 81831fa

[RussTreadon-NOAA]

WCOSS2 test

Compile feature/gome in debug mode on Cactus. 81831fa includes a debug print to see if the ORBN check works.

Run global_4denvar for 20240223 00Z case. This case has a GOME bufr dump file with the new format. As shown from the debug print, gsi.x did not find ORBN in the input GOME dump file and correctly set the number of mnemonics (11) to extract. The debug gsi.x successfully ran to completion for global_4denvar_loproc. Stop test here.

Repeat the above test for 20221109 00Z case. This case has a GOME bufr dump file with the old format. The debug print shows that ORBN was found and correctly set the number of mnemonics (10) to extract. The debug gsi.x successfully ran to completion for global_4denvar_loproc. Stop test here.

The global_4denvar ctest does not check diagnostic files. We need to ensure that the ozone diagnostic files contain the correct information when processing the old and new format GOME bufr dump file.

[RussTreadon-NOAA]

Orion ctests
Install feature/gome and run ctest with the following results

Test project /work2/noaa/da/rtreadon/git/gsi/gome/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  483.39 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  606.23 sec
3/7 Test #7: global_enkf ......................***Failed  849.33 sec
4/7 Test #2: rtma .............................   Passed  1148.03 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1397.08 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1576.68 sec
7/7 Test #1: global_4denvar ...................   Passed  1984.40 sec

86% tests passed, 1 tests failed out of 7

Total Test time (real) = 1984.41 sec

The following tests FAILED:
          7 - global_enkf (Failed)

The global_enkf failure is due to

The runtime for global_enkf_hiproc_updat is 147.688745 seconds.  This has exceeded maximum allowable threshold time of 142.550184 seconds, resulting in Failure timethresh2 of the regression test.

A check of the enkf.x wall times shows

global_enkf_hiproc_contrl/stdout:The total amount of wall time                        = 129.591077
global_enkf_hiproc_updat/stdout:The total amount of wall time                        = 147.688745
global_enkf_loproc_contrl/stdout:The total amount of wall time                        = 152.694215
global_enkf_loproc_updat/stdout:The total amount of wall time                        = 167.089731

The loproc and hiproc updat_ wall times are greater than the contrl. This PR, however, only changes gsi.x. The EnKF source code is identical for feature/gome and develop. The enkf.x wall time differences may reflect filesystem or compute issues. This is not a fatal fail.

[RussTreadon-NOAA]

Hera ctests
Run ctests on Hera with the following results

Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/gome/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #7: global_enkf ......................***Failed  1911.23 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  2301.84 sec
3/7 Test #4: netcdf_fv3_regional ..............   Passed  2358.16 sec
4/7 Test #1: global_4denvar ...................   Passed  3075.46 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  3155.43 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  3755.48 sec
7/7 Test #2: rtma .............................   Passed  5966.85 sec

86% tests passed, 1 tests failed out of 7

Total Test time (real) = 5966.97 sec

The following tests FAILED:
          7 - global_enkf (Failed)

The global_enkf failure is due to

The runtime for global_enkf_loproc_updat is 186.299056 seconds.  This has exceeded maximum allowable threshold time of 173.885933 seconds, resulting in Failure timethresh of the regression test.

A check of the enkf.x wall times shows

global_enkf_hiproc_contrl/stdout:The total amount of wall time                        = 138.506692
global_enkf_hiproc_updat/stdout:The total amount of wall time                        = 128.723597
global_enkf_loproc_contrl/stdout:The total amount of wall time                        = 158.078121
global_enkf_loproc_updat/stdout:The total amount of wall time                        = 186.299056

The hiproc_updat is faster than the hiproc_contrl. The opposite is true for the lproc jobs. This variability is likely due to fileset or compute node usage. This is not a fatal fail.

[RussTreadon-NOAA]

Hercules ctest
Run ctests on Hercules with the following results

Test project /work2/noaa/da/rtreadon/git/gsi/gome/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  483.52 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  544.43 sec
3/7 Test #7: global_enkf ......................   Passed  726.76 sec
4/7 Test #2: rtma .............................   Passed  965.13 sec
5/7 Test #6: hafs_3denvar_hybens ..............***Failed  1096.16 sec
6/7 Test #5: hafs_4denvar_glbens ..............***Failed  1332.51 sec
7/7 Test #1: global_4denvar ...................   Passed  1681.64 sec

71% tests passed, 2 tests failed out of 7

Total Test time (real) = 1681.68 sec

The following tests FAILED:
          5 - hafs_4denvar_glbens (Failed)
          6 - hafs_3denvar_hybens (Failed)

The hafs_3denvar_hybens test failed with

The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (hafs_3denvar_hybens_loproc_updat and hafs_3denvar_hybens_hiproc_updat) are not reproducible
Thus, the case has Failed siganl of the regression tests.

The hafs_4denvar_glbens test failed with

The fv3_sfcdata are reproducible
The fv3_tracer are reproducible
The results between the two runs (hafs_4denvar_glbens_loproc_updat and hafs_4denvar_glbens_hiproc_updat) are not reproducible
Thus, the case has Failed siganl of the regression tests.

Both of these failures are known issues and are being addressed by PR #698

[RussTreadon-NOAA]

WCOSS2 ctest
Run ctests on Cactus with the following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/gome/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_glbens
    Start 4: netcdf_fv3_regional
    Start 5: hafs_4denvar_glbens
    Start 6: hafs_3denvar_hybens
    Start 7: global_enkf
1/7 Test #4: netcdf_fv3_regional ..............   Passed  483.61 sec
2/7 Test #3: rrfs_3denvar_glbens ..............   Passed  546.25 sec
3/7 Test #7: global_enkf ......................   Passed  850.57 sec
4/7 Test #2: rtma .............................   Passed  969.02 sec
5/7 Test #6: hafs_3denvar_hybens ..............   Passed  1153.62 sec
6/7 Test #5: hafs_4denvar_glbens ..............   Passed  1212.59 sec
7/7 Test #1: global_4denvar ...................   Passed  1684.61 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) = 1684.62 sec