Add MODIS Terra/Aqua satwinds assimilation to end-to-end GDASApp vali…

…dation (#1030)

Adding satwinds from the Moderate Resolution Imaging Spectroradiometer
(MODIS) from Terra/Aqua to GDASApp end-to-end testing

new files include:
parm/atm/obs/config/satwind_modis_terra.yaml.j2: QC filter YAML for
MODIS Terra satwinds (jinja2 standard)
parm/atm/obs/config/satwind_modis_aqua.yaml.j2: QC filter YAML for MODIS
Aqua satwinds (jinja2 standard)
parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_modis.json: JSON containing
data format, sensor, and satellite information for MODIS Terra/Aqua
satwinds
ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_modis.py: bufr2ioda code for
extracting MODIS Terra/Aqua satwinds from BUFR

See #1029 for end-to-end testing results. Acceptance and ob-errors agree
well with some expected deviation. Thinning is currently turned off in
both GSI and JEDI for these tests.

Note: We are still using `qualityInformationWithoutForecast` as the
variable-name for QI in the IODA converter. This variable-name is
currently not registered in the IODA ObsSpace.yaml, but there is an
ongoing discussion with JCSDA to have it added, issue is here:
JCSDA-internal/ioda#1233

---------

Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>

parm/atm/obs/config/satwind_modis_aqua.yaml.j2

@@ -0,0 +1,356 @@
+- obs space:
+    name: satwind_modis_aqua
+    obsdatain:
+      engine:
+        type: H5File
+        obsfile: '{{ DATA }}/obs/{{ OPREFIX }}satwnd.modis_aqua.tm00.nc'
+    obsdataout:
+      engine:
+        type: H5File
+        obsfile: '{{ DATA }}/diags/diag_satwind_modis_aqua_{{ current_cycle | to_YMDH }}.nc'
+    io pool:
+      max pool size: 1
+    simulated variables: [windEastward, windNorthward]
+
+  obs operator:
+    name: VertInterp
+    hofx scaling field: SurfaceWindScalingPressure
+    hofx scaling field group: DerivedVariables
+
+  linear obs operator:
+    name: VertInterp
+
+  # NOTE: Tests using the Gaussian Thinning filter (below) to duplicate GSI's thinning of AHI/Himawari-8 satwinds
+  #       results in more JEDI satwinds in the diag file than in GSI, but far fewer JEDI satwinds assimilated than
+  #       GSI. JEDI under-counts assimilated winds by roughly 25-40%, relative to GSI, and this under-count is not
+  #       even including the temporal thinning which is applied in GSI but not JEDI (by this filter below). See
+  #       GDASApp Issue #741 for details: https://github.com/NOAA-EMC/GDASApp/issues/741
+  #obs pre filters:
+  #- filter: Gaussian Thinning
+  #  horizontal_mesh: 200
+  #  vertical_mesh: 10000
+  #  use_reduced_horizontal_grid: true
+  #  round_horizontal_bin_count_to_nearest: true
+  #  partition_longitude_bins_using_mesh: true
+
+  obs prior filters:
+  # Apply variable changes needed for wind scaling
+  # For wind observations with pressure provided
+  - filter: Variable Transforms
+    Transform: SurfaceWindScalingPressure
+    SkipWhenNoObs: False
+
+  # Calculate error inflation factor for duplicate observations 
+  #- filter: Variable Assignment
+  #  assignments:
+  #  - name: ObsErrorFactorDuplicateCheck/windEastward
+  #    type: float
+  #    function:
+  #      name: ObsFunction/ObsErrorFactorDuplicateCheck
+  #      options:
+  #        use_air_pressure: true
+  #        variable: windEastward
+
+  #- filter: Variable Assignment
+  #  assignments:
+  #  - name: ObsErrorFactorDuplicateCheck/windNorthward
+  #    type: float
+  #    function:
+  #      name: ObsFunction/ObsErrorFactorDuplicateCheck
+  #      options:
+  #        use_air_pressure: true
+  #        variable: windNorthward
+
+  obs post filters:
+  # Assign the initial observation error, based on height/pressure
+  # Hard-wiring to prepobs_errtable.global by Type
+  # ObsError is currently not updating in diag file, but passes directly to EffectiveError when no inflation is specified in YAML
+
+  # Type 257 (MODIS LWIR)
+  - filter: Perform Action
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 257
+    action:
+      name: assign error
+      error function:
+        name: ObsFunction/ObsErrorModelStepwiseLinear
+        options:
+          xvar:
+            name: MetaData/pressure
+          xvals: [110000.,105000.,100000.,95000.,90000.,85000.,80000.,75000.,70000.,65000.,60000.,55000.,50000.,45000.,40000.,35000.,30000.,25000.,20000.,15000.,10000.,7500.,5000.,4000.,3000.,2000.,1000.,500.,400.,300.,200.,100.,0.]   #Pressure (Pa)
+          errors: [3.8,3.8,3.8,3.8,3.8,3.8,3.8,3.8,3.9,3.9,4.,4.,4.1,5.,6.,6.3,6.6,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.]
+
+  # Type 258 (MODIS WVCT): Some levels assigned dummy values
+  - filter: Perform Action
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 258
+    action:
+      name: assign error
+      error function:
+        name: ObsFunction/ObsErrorModelStepwiseLinear
+        options:
+          xvar:
+            name: MetaData/pressure
+          xvals: [110000.,60000.,55000.,50000.,45000.,40000.,35000.,30000.,25000.,20000.,15000.,10000.,7500.,5000.,4000.,3000.,2000.,1000.,500.,400.,300.,200.,100.,0.]   #Pressure (Pa)
+          errors: [1000000000.,1000000000.,4.,4.1,5.,6.,6.3,6.6,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.]
+
+  # Type 259 (MODIS WVDL): Some levels assigned dummy values
+  - filter: Perform Action
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 259
+    action:
+      name: assign error
+      error function:
+        name: ObsFunction/ObsErrorModelStepwiseLinear
+        options:
+          xvar:
+            name: MetaData/pressure
+          xvals: [110000.,60000.,55000.,50000.,45000.,40000.,35000.,30000.,25000.,20000.,15000.,10000.,7500.,5000.,4000.,3000.,2000.,1000.,500.,400.,300.,200.,100.,0.]   #Pressure (Pa)
+          errors: [1000000000.,1000000000.,4.,4.1,5.,6.,6.3,6.6,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.,7.]
+
+  # sanity-check criteria
+  # Observation Range Sanity Check
+  # NOT EXPLICITLY CLEARED: No obs in this range in file, so 0 Bounds Check rejects (which is correct) but essentially untested
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    minvalue: -130.
+    maxvalue: 130.
+    action:
+      name: reject
+
+  # Velocity Sanity Check
+  # NOT EXPLICITLY CLEARED: No obs in this range in file, so 0 Bounds Check rejects (which is correct) but essentially untested
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    test variables:
+    - name: ObsFunction/Velocity
+    maxvalue: 130.
+    action:
+      name: reject
+
+  # MODIS-Aqua/Terra (257) and (259), reject when pressure less than 249 mb.
+  # CLEARED: minvalue is rejecting <, not <= as per a Perform Action, so threshold is unchanged
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 257,259
+    test variables:
+    - name: MetaData/pressure
+    minvalue: 24900.
+    action:
+      name: reject
+
+  # MODIS-Aqua/Terra (258) and (259), reject when pressure greater than 600 mb.
+  # NOT EXPLICITLY CLEARED: No obs in this range in file, so 0 Bounds Check rejects (which is correct) but essentially untested
+  # maxvalue is rejecting >, not >= as per a Perform Action, so threshold is unchanged
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 258, 259
+    test variables:
+    - name: MetaData/pressure
+    maxvalue: 60000.
+    action:
+      name: reject
+
+  # Reject any observation with a /=0 surface type (non-water surface) within
+  # 200 hPa of the surface pressure (as part of the LNVD check).
+  # CLEARED: maxvalue is rejecting >, not >= as per a Perform Action, so threshold is unchanged
+  - filter: Difference Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable:
+        name: GeoVaLs/water_area_fraction
+      maxvalue: 0.99
+    reference: GeoVaLs/surface_pressure
+    value: MetaData/pressure
+    maxvalue: -20000.                   # within 200 hPa above surface pressure, negative p-diff
+    action:
+      name: reject
+
+  # LNVD check
+  # CLEARED: maxvalue is rejecting >, not >= as per a Perform Action, so threshold is unchanged
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    test variables:
+    - name: ObsFunction/SatWindsLNVDCheck
+    maxvalue: 3.
+    action:
+      name: reject
+
+  # GSI setupw routine QC
+  # Reject any ob Type [240–260] when pressure greater than 950 mb.
+  # CLEARED: minvalue/maxvalue are >=/<=, not >/<, so editing range by 1 Pa    
+  - filter: Bounds Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    where:
+    - variable: ObsType/windEastward
+      is_in: 240-260
+    test variables:
+    - name: MetaData/pressure
+    maxvalue: 95001.
+    action:
+      name: reject
+
+  # Multiple satellite platforms, reject when pressure is more than 50 mb above tropopause.
+  # CLEARED: minvalue is rejecting <, not <= as per a Perform Action, so threshold is unchanged
+  # Notes (eliu): This tropopause check reject too many obs; probably due to tropopause pressure estimation 
+  #               Turn this check off for now. 
+  #               Need to check if troposphere pressure was implemented correctly in fv3-jed
+  - filter: Difference Check
+    filter variables:
+    - name: windEastward
+    - name: windNorthward
+    reference: GeoVaLs/tropopause_pressure
+    value: MetaData/pressure
+    minvalue: -5000.                   # 50 hPa above tropopause level, negative p-diff
+    action:
+      name: reject
+
+  # All satwinds must adjust errors based on ObsErrorFactorPressureCheck
+  # prior to the SPDB check (i.e. the gross-error check). The gross-error
+  # check uses the adjusted errors for error-bound tightening and rejection,
+  # so this check has to come first. This check will inflate errors for obs
+  # that are too close to either the model top or bottom.
+  # Notes (eliu): GMAO added a required parameter: adjusted_error_name.
+  - filter: Perform Action
+    filter variables:
+    - name: windEastward
+    where:
+      - variable:
+          name: ObsType/windEastward
+        is_in: 240-260
+    action:
+      name: inflate error
+      inflation variable:
+        name: ObsFunction/ObsErrorFactorPressureCheck
+        options:
+          surface_obs: false 
+          variable: windEastward
+          inflation factor: 4.0
+
+  - filter: Perform Action
+    filter variables:
+    - name: windNorthward
+    where:
+      - variable:
+          name: ObsType/windNorthward
+        is_in: 240-260
+    action:
+      name: inflate error
+      inflation variable:
+        name: ObsFunction/ObsErrorFactorPressureCheck
+        options:
+          variable: windNorthward
+          inflation factor: 4.0
+
+  # All satwinds subject to a gross-error check that contains significant
+  # modifiers for satwinds with a negative speed-bias. ALL wind gross-error
+  # checks are currently being done by the SatWindsSPDBCheck.
+  # CLEARED
+  - filter: Background Check
+    filter variables:
+    - name: windEastward
+    function absolute threshold:
+    - name: ObsFunction/WindsSPDBCheck
+      options:
+        wndtype: [  240,  241,  242,  243,  244,  245,  246,  247,  248,  249, 250,  251,  252,  253,   254,  255,  256,  257,  258,  259,  260]
+        cgross:  [  2.5,  2.5,  2.5,  1.5,  2.5,  1.3,  1.3,  2.5,  2.5,  2.5, 2.5,  1.3,  2.5,  1.5,   1.5,  2.5,  2.5,  2.5,  2.5,  2.5,  2.5]
+        error_min: [1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4, 1.4,  1.4,  1.4,  1.4,   1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4]
+        error_max: [6.1,  6.1, 15.0, 15.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.1, 20.1, 20.1, 20.1, 20.1, 20.1]
+        variable: windEastward
+    action:
+      name: reject
+
+  - filter: Background Check
+    filter variables:
+    - name: windNorthward
+    function absolute threshold:
+    - name: ObsFunction/WindsSPDBCheck
+      options:
+        wndtype: [  240,  241,  242,  243,  244,  245,  246,  247,  248,  249, 250,  251,  252,  253,   254,  255,  256,  257,  258,  259,  260]
+        cgross:  [  2.5,  2.5,  2.5,  1.5,  2.5,  1.3,  1.3,  2.5,  2.5,  2.5, 2.5,  1.3,  2.5,  1.5,   1.5,  2.5,  2.5,  2.5,  2.5,  2.5,  2.5]
+        error_min: [1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4, 1.4,  1.4,  1.4,  1.4,   1.4,  1.4,  1.4,  1.4,  1.4,  1.4,  1.4]
+        error_max: [6.1,  6.1, 15.0, 15.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.0, 20.1, 20.1, 20.1, 20.1, 20.1, 20.1]
+        variable: windNorthward
+    action:
+      name: reject
+
+  # The last error inflation check is for duplicate observations. This one needs
+  # to come last, because we don't want to inflate errors for duplication if one
+  # of the duplicates should be rejected.
+  # Notes (eliu): ObsErrorFactorDuplicateCheck obsfunction requires PreUseFlag (usage parameter from read_satwnd.f90).
+  #             : Turn off duplicate check for now.
+  #- filter: Perform Action
+  #  filter variables:
+  #  - name: windEastward
+  #  action:
+  #    name: inflate error
+  #    inflation variable:
+  #      name: ObsErrorFactorDuplicateCheck/windEastward
+
+  #- filter: Perform Action
+  #  filter variables:
+  #  - name: windNorthward
+  #  action:
+  #    name: inflate error
+  #    inflation variable:
+  #      name: ObsErrorFactorDuplicateCheck/windNorthward
+
+  # We are extending this to an additional filter that inflates final ob-errors across-the-board by
+  # 1/0.8 = 1.25. This is caused by the GSI value of nvqc being set to .true. in the global operational
+  # configuration, see: https://github.com/NOAA-EMC/global-workflow/blob/d5ae3328fa4041b177357b1133f6b92e81c859d7/scripts/exglobal_atmos_analysis.sh#L750
+  # This setting activates Line 1229 of setupw.f90 to scale ratio_errors by 0.8, which is applied in
+  # the denominator of the final ob-error, so 1/0.8 = 1.25 factor of ob-error inflation.
+  # 
+  # If this nvqc functionality were to be switched off (i.e. if variational qc were to be turned off),
+  # you would want to remove this last inflation filter.
+  #- filter: Perform Action
+  #  filter variables:
+  #  - name: windEastward
+  #  where:
+  #  - variable: ObsType/windEastward
+  #    is_in: 240-260
+  #  action:
+  #    name: inflate error
+  #    inflation factor: 1.25
+
+  #- filter: Perform Action
+  #  filter variables:
+  #  - name: windNorthward
+  #  where:
+  #  - variable: ObsType/windNorthward
+  #    is_in: 240-260
+  #  action:
+  #    name: inflate error
+  #    inflation factor: 1.25
+
+  # End of Filters