GSI scripts change to assimilate Metop-c AVHRR radiance

[XuLi-NOAA]

This issue is to replace Metop-a AVHRR , which discontinued in November 2021, with Metop-c AVHRR radiance.
The necessary GSI change is limited to scripts, once the Metop-c AVHRR radiance is added to avcsam bufr data file.

[dtkleist]

@XuLi-NOAA -- Metop-c has replaced Metop-a (not -b) as the operational satellite in that orbit.

[XuLi-NOAA]

@dtkleist , thanks! It is Metop-a, not Metop-b. Corrected.

[XuLi-NOAA]

A quick update:

I planned to run two cycling runs, starting 2022012818, at C384, GFS cold start, 12 GDAS cycles (2022012900 to 2022013118).

v16nst7: updated global-work flow, no metop-c avhrr, and additional in situ data (saildrone, Argo & Glider)

v16nst8: The same as v16nst8 but with metop-c avhrr, and additional in situ data

v16nst7 (control run) has been done, the results analysis is underway.
v16nst8 has run two cycles (will be 12 GDAS cycles as well).

From v16nst8, there seems issues for some AVHRR radiance data, e.g., for metop-c, the cloud flag (CLAVR) is always missing. That means all the data will not be used since the observation with non-zero cloud flag (non perfect clear based on CLAVR) ) will be tossed in GSI. I will see how about the later cycles. But ObsProc people can check why cloud flag is missing for avcsam*2022012900.

[XuLi-NOAA]

The issue in metop-c AVHRR bufr data file (in avcsam) has been fixed since 2022021118 cycle, nothing is changed at the NCEP side, something is changed at the source, NESDIS side.
Two cycling runs have been done, with C384 cold start, initial conditions are generated from operational C768. The two runs, v16nst11: w/o metop-c AVHRR

and

v16nst12: w/ metop-c AVHRR

have been run for 4 days (16 cycles), starting from 2022021118.

It shows the metop-c AVHRR has been assimilated successfully, the evaluation shows the NSST analysis between the two runs are small, but one concern is that it is getting slightly colder with metop-c AVHRR in. The final check is underway. A PR will be submitted after that.

[XuLi-NOAA]

In order to make sure the slightly colder NSST analysis due to metop-c AVHRR radaince is fine, another cycling run with the current operational GFS, pr16mc, starting from 2022021118 operational initial conditions, at C768, warm start, has been done for 12 cycles. The results have shown now colder NSST analysis in pr16mc is not a concern anymore (gone), compared operational GFS/NSST analysis. Therefore, the metop-c AVHRR can be turned on without any issue. And indeed, the NSST analysis become a little bit closer to OSTIA analysis, which is a positive impact by metop-c avhrr (see figure).
One experience: It is better not to use the cold start in the impact evaluation test, since there is some adjustment of the system behavior in the initial time period in the cold start (of FV3 GFS). We can see, The cold start runs, N11 (w/o metop-c avhrr) and N12 (witho metop-c AVHRR), are much further from OSTIA than the warm start run (pr16mc).
The regression test will be done, a branch, feature/gsi_avhmc will be generated and a PR will be submitted.

[XuLi-NOAA]

Regression test results (Oh Hera):
Test project /scratch1/NCEPDEV/da/Xu.Li/git/GSI_avhmc/build
Start 1: global_T62
Start 2: global_T62_ozonly
Start 3: global_4dvar_T62
Start 4: global_4denvar_T126
Start 5: global_fv3_4denvar_T126
Start 6: global_fv3_4denvar_C192
Start 7: global_lanczos_T62
Start 8: arw_netcdf
1/19 Test #8: arw_netcdf .......................***Failed 960.88 sec
Start 9: arw_binary
2/19 Test #2: global_T62_ozonly ................ Passed 1024.88 sec
Start 10: nmm_binary
3/19 Test #6: global_fv3_4denvar_C192 ..........***Failed 1381.58 sec
Start 11: nmm_netcdf
4/19 Test #9: arw_binary .......................***Failed 840.92 sec
Start 12: nmmb_nems_4denvar
5/19 Test #11: nmm_netcdf .......................***Failed 480.32 sec
Start 13: hwrf_nmm_d2
6/19 Test #13: hwrf_nmm_d2 ......................***Failed 120.27 sec
Start 14: hwrf_nmm_d3
7/19 Test #10: nmm_binary .......................***Failed 2220.74 sec
Start 15: rtma
8/19 Test #14: hwrf_nmm_d3 ......................***Failed 2641.57 sec
Start 16: global_enkf_T62
9/19 Test #3: global_4dvar_T62 ................. Passed 4808.50 sec
Start 17: netcdf_fv3_regional
10/19 Test #17: netcdf_fv3_regional ..............***Failed 360.29 sec
Start 18: global_C96_fv3aero
11/19 Test #4: global_4denvar_T126 .............. Passed 6548.87 sec
Start 19: global_C96_fv3aerorad
12/19 Test #16: global_enkf_T62 .................. Passed 2287.97 sec
13/19 Test #12: nmmb_nems_4denvar ................***Failed 6546.63 sec
14/19 Test #18: global_C96_fv3aero ...............***Failed 8114.36 sec
15/19 Test #7: global_lanczos_T62 ............... Passed 13397.41 sec
16/19 Test #15: rtma ............................. Passed 10642.12 sec
17/19 Test #1: global_T62 ....................... Passed 14180.65 sec
18/19 Test #5: global_fv3_4denvar_T126 .......... Passed 14422.96 sec
19/19 Test #19: global_C96_fv3aerorad ............ Passed 8775.19 sec

47% tests passed, 10 tests failed out of 19

Total Test time (real) = 15324.10 sec

The following tests FAILED:
6 - global_fv3_4denvar_C192 (Failed)
8 - arw_netcdf (Failed) : Only Failed the scalability test
9 - arw_binary (Failed):
10 - nmm_binary (Failed)
11 - nmm_netcdf (Failed)
12 - nmmb_nems_4denvar (Failed)
13 - hwrf_nmm_d2 (Failed)
14 - hwrf_nmm_d3 (Failed)
17 - netcdf_fv3_regional (Failed)
18 - global_C96_fv3aero (Failed): This has exceeded maximum allowable threshold time
Errors while running CTest

I saw the failure reasons for a couple of tests, but didn't see every.
Mike, can you check to see what happened and if it is OK. I will continue looking into the failure reasons, but it has to go through your check anyway.

[XuLi-NOAA]

@MichaelLueken-NOAA : Hera is down today and I checked out the branch feature/gsi_avhmc on dell and then ran the regression tests, it show only two tests are failed:
The following tests FAILED:
6 - global_fv3_4denvar_C192 (Failed)
19 - global_C96_fv3aerorad (Failed)
Errors while running CTest

I checked /gpfs/dell2/emc/modeling/noscrub/Xu.Li/regression/global_fv3_4denvar_C192_regression_results.txt, it shows the problem is the results are not reproducible. I checked
/gpfs/dell2/ptmp/Xu.Li/_gpfs_dell2_emc_modeling_noscrub_Xu.Li_git_GSI_avhmc_build/tmpreg_global_fv3_4denvar_C192/global_fv3_4denvar_C192_loproc_contrl

and

/gpfs/dell2/ptmp/Xu.Li/_gpfs_dell2_emc_modeling_noscrub_Xu.Li_git_GSI_avhmc_build/tmpreg_global_fv3_4denvar_C192/global_fv3_4denvar_C192_loproc_updat,

I can see the number of observations read from nsstbufr is different,

read_nsstbufr : 22 0 20752 10401 (contrl)

read_nsstbufr : 22 0 20752 10350 (updat)

I don't understand how does this difference come from. And I cannot the metop-c AVHRR is used in the regression test, it looks like the GSI analysis scripts used in the regression test are not from my branch (feature/gsi_avhmc)? Can you check?

As to global_C96_fv3aerorad, I cannot see why it failed from /gpfs/dell2/emc/modeling/noscrub/Xu.Li/regression/global_C96_fv3aero_regression_results.txt,

everything is OK from it.

[MichaelLueken]

@XuLi-NOAA Looking at your feature/gsi_avhmc branch, this doesn't have your update that was merged to the authoritative repository on Monday, 02/28/2022 (Adding assimilation of Saildrone, Argo, and Glider data to NSST). This will affect the number of observations read from nsstbufr for global_fv3_4denvar_C192. For global_C96_fv3aerorad, in global_C96_fv3aerorad_regression_results.txt (not global_C96_fv3aero_regression_results.txt), I see:

The runtime for global_C96_fv3aerorad_loproc_updat is 1276.097176 seconds.  This has exceeded maximum allowable operational time of 1200 seconds,
resulting in Failure of max-time in the regression test.

This is fine and expected behavior for this test on non-Hera machines. As you said, all other regression tests passed without issue.

[XuLi-NOAA]

@MichaelLueken-NOAA : Yes, the merge with "Adding assimilation of Saildrone, Argo, and Glider data to NSST" is not included here.
Ok, I will go ahead to submit a PR.

[XuLi-NOAA]

@MichaelLueken-NOAA : Looks like I need to update my branch and even my fork master since the update of the authoritative repository on Monday, 02/28/2022, and rerun the regression tests, or is there a better way to go ahead?

[MichaelLueken]

@XuLi-NOAA: Yes, it would be best to update both your feature/gsi_avhmc branch and master branch in your fork will need to be updated with the latest authoritative master.

It would probably be best to rerun just the global_fv3_4denvar_C192 regression test to ensure that it works as expected, then a new PR with this work can be created. To run a single test:

In your build directory, open CTestTestfile.cmake and comment out the add_test and set_tests_properties for all configurations that aren't global_fv3_4denvar_C192.

This will allow you to run a single regression test.

[XuLi-NOAA]

Summary:

1. Scripts changes
   exglobal_atmos_analysis.sh
   exgdas_enkf_update.sh
   exgdas_efsoi.sh
   exgdas_efsoi_update.sh
   exglobal_diag.sh (One more change included in this issue is actually for the use of viirs radiance.
   It is in exglobal_diag.sh, add viirs-m_npp & viirs-m_j1).
   add iasi_metop-c to exgdas_efsoi.sh as well (not my task, but it should be added).
2. Fix file change
   /gpfs/dell2/emc/modeling/noscrub/Xu.Li/git/GSI_avhmc/fix/global_satinfo.txt

[XuLi-NOAA]

The regression test, global_fv3_4denvar_C192, passed.

ctest
Test project /gpfs/dell2/emc/modeling/noscrub/Xu.Li/git/GSI_avhmc/build
Start 1: global_fv3_4denvar_C192
1/1 Test #1: global_fv3_4denvar_C192 .......... Passed 5061.56 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) = 5062.26 sec