Failures in GSI analysis due to inconsistent correlated error

[CatherineThomas-NOAA]

What is wrong?

Several users have experienced failures in the atmospheric analysis step:

63: [063]correlated_obsmod*upd_varch_(): >>> ERROR <<<  improperly set GSI_BundleErrorCov

This appears to be due to the inconsistency between the anavinfo file correlated error block and the satinfo. In the failed cases, the anavinfo file lists CrIS N21 while the historical satinfo files (set in config.anal) that were used did not. Many of the failed cases occurred for SDATEs in 2021.

See /scratch1/NCEPDEV/climate/Jessica.Meixner/cycling/turnonmarine/c48ci01/COMROOT/c48t01/logs/2021032418/failed.gdasanal.log.0

What should have happened?

The atmospheric analysis should have completed without failure.

What machines are impacted?

All or N/A

Steps to reproduce

1. Set up an experiment with atmospheric DA.
2. Choose a date that uses a historical satinfo file (2021 in these cases).
3. Run the GSI analysis step (gfsanal/gdasanal).

Additional information

N/A

Do you have a proposed solution?

@ADCollard Do we need historical anavinfo files?

[CatherineThomas-NOAA]

FYI @JessicaMeixner-NOAA @guillaumevernieres @CoryMartin-NOAA

[RussTreadon-NOAA]

Removing the cris-fsr_n21 entry from the correlated_observations section of global_anavinfo.l127.txt works.

Another option is to leave the anavinfo file alone and move code around in src/gsi/correlated_obsmod.F90

diff --git a/src/gsi/correlated_obsmod.F90 b/src/gsi/correlated_obsmod.F90
index 7a14cd32..17cd94ef 100644
--- a/src/gsi/correlated_obsmod.F90
+++ b/src/gsi/correlated_obsmod.F90
@@ -961,14 +961,18 @@ subroutine upd_varch_
             enddo
             nchanl1=jc

-            if(nchanl1==0) call die(myname_,' improperly set GSI_BundleErrorCov')
             if(.not.amiset_(GSI_BundleErrorCov(itbl))) then
-               if (iamroot_) write(6,*) 'WARNING: Error Covariance not set for ',trim(idnames(itbl))
+               if (iamroot_) write(6,*) trim(myname_), ' WARNING: Error Covariance not set for ',trim(idnames(itbl))
                cycle read_tab
             endif

             nch_active=GSI_BundleErrorCov(itbl)%nch_active
-            if(nch_active<0) return
+            if(nch_active<0) then
+               if (iamroot_) write(6,*) trim(myname_), ' WARNING: No active channels for ',trim(idnames(itbl))
+               return
+            endif
+
+            if(nchanl1==0) call die(myname_,' improperly set GSI_BundleErrorCov')

             if(GMAO_ObsErrorCov)then
                do jj=1,nch_active

The nchanl1==0 line is the one causing gsi.x to abort.

For cris-fsr_n21, amiset_(GSI_BundleErrorCov(itbl)) = .false. and execution cycles to the next read_tab.

gsi.x built with the above change successfully ran the 2021032418 case. The above change is in /scratch1/NCEPDEV/da/Russ.Treadon/git/gsi/develop_test

GSI ctests were run using the modified gsi.x. Tests pass.

[RussTreadon-NOAA]

To build g-w with the revised GSI source code, open a g-w PR to update the gsi_enkf.fd hash to 457510c

[RussTreadon-NOAA]

The gsi_enkf.fd hash will be updated to 457510c in RussTreadon-NOAA/feature/gsihash.