[End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing #766
Conversation
…te obsfile for satwnd.abi_goes-16
Two updates for GOES AMV 1. BUFR Converter: - remove wind direction and speed - change sensorZenithAngle to satelliteZenithAngle 2. UFO Filters: - change `sensorZenithAngle` to `satelliteZenithAngle` - add `linear obs operator` section The declaration of `linear obs operator" is necessary for GOES AMV since there is no linearized component for applying the 10-meter factor. The increments look reasonable: ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-1.642335e-01 Max:+1.660654e-01 RMS:+2.746685e-04 0: northward_wind | Min:-2.120068e-01 Max:+1.663539e-01 RMS:+2.631295e-04 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` The UFO filter results vs. GSI -- looks good  
Rather than each new ob type being manually added to the list, we can use `glob.glob` to get the list of available observation types.
Two updates for MetOp SCATWIND; one minor updates for GOES AMV
1. BUFR Converter:
- remove wind direction and speed
- add `stationElevation` as zero
2. UFO Filters:
- add linear obs operator section
The declaration of `linear obs operator" is necessary for GOES AMV since
there is no
linearized component for applying the 10-meter factor.
- add `Gaussian Thinning` filter to thin data with 75 km box (equal box
size, prefer data close to central of the box)
3. For GOES AMV YAML files, change output data extension from `nc4` to
`nc`
UFO vs. GSI O-F and QC comparison: O-F and data count passed look close
3Dvar increments are reasonable with MetOp-B SCATWIND alone.
```
0: ----------------------------------------------------------------------------------------------------
0: Increment print | number of fields = 8 | cube sphere face size: C768
0: eastward_wind | Min:-1.741092e+00 Max:+1.712287e+00 RMS:+2.282584e-03
0: northward_wind | Min:-1.825957e+00 Max:+1.930385e+00 RMS:+2.496075e-03
0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: cloud_liquid_ice | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: cloud_liquid_water | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00
0: ----------------------------------------------------------------------------------------------------
```
3DVar increments are also reasonable with MetOp-B + MetOp-A assimilated
…oda.x (#730) This PR allows for all obtypes to be processed in two loops. We still need to figure out a way to process these in parallel (somewhat).
Using python multiprocessing to generate obs in parallel. The current list of obs goes from 10+ minutes to completing in ~5.5 minutes.
@RussTreadon-NOAA noted that `obs linear operator` should be `linear obs operator` in the GOES-17 satwind YAML. This fixes that.
This reduces runtime further to run everything in one parallel pool. Now, things complete in under 4 minutes (as of now).
Adding satwinds from the Advanced Himawari Imager (AHI) from Himawari-8 to GDASApp end-to-end testing new files include: parm/atm/obs/config/satwind_ahi_h8.yaml: QC filter YAML for AHI Himawari-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.json: JSON containing data format, sensor, and satellite information for AHI Himawari-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_ahi.py: bufr2ioda code for extracting AHI Himawari-8 satwinds from BUFR modified files include: ush/ioda/bufr2ioda/run_bufr2ioda.py: added `"satwind_amv_ahi"` to list `BUFR_py` See #741 for details on testing. JEDI/GSI comparisons look good with GSI thinning turned off, but there are large ob-count disparities both before and after QC when comparing GSI+thinning to JEDI+thinning. The tested `Gaussian Thinning` filter is included in the YAML but commented out with a note. --------- Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Instead of writing the filled out `.json` and `.yaml` files to `$COM_OBS`, we write them to `$DATA` so that they get purged unless `KEEPDATA="YES"`
Updating GNSSRO converter and adding the yaml file to GDASApp end-to-end testing based on [#712](#712). New file: parm/atm/obs/config/gnssro.yaml: Added QC filters in the YAML Updated files: parm/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.json: Updated the satellite information ush/ioda/bufr2ioda/bufr2ioda_gnssro_bufr.py: Updated pccf, qfro, and satelliteAscending flag Refer to [issue #750](#750) for testing details. JEDI/GSI comparisons revealed a cutoff near obs=0.03 Rad due to super refraction 2. The number of observations after QC in JEDI is also inconsistent with GSI.
List of items to be added in this PR: - BUFR Converter: Add YAML for converting ATMS BUFR to IODA - The ObsValue is antenna temperature - Input BUFR is normal feel - Testing YAML: re-evaluation results due to update CRTM from v2.3.0 to v2.4.1-jedi.1 - atms_npp_noqc.yaml --- 100% replication - atms_npp.yaml --- 100 % replication - atms_n20_noqc.yaml --- 100 % replication - atms_n20.yaml --- ~ 1100% replication: two channel 7 observations difference - Config YAML: * add two read routine QC: data thinning and removal of data from scan edge - atms_npp.yaml - atms_n20.yaml - Paring PR in JCSDA UFO repository - This PR also has a [paring PR in JCSDA UFO repository](JCSDA-internal/ufo#3094). - It is OK to merge this one into GDASApp before the paring UFO PR - The related test data (obs and geoval files from GSI) for UFO Evaluation have been updated (due to CRTM-2.4.1 update) on ORION and HERA. Notes: - End-to-end testing completed without problem. - However, the filter results show one of the filters, 88-165 GHz scattering index check, tossed more data compared to GSI. - The excess screening from the 88-165 GHz scattering index check is under investigation and will be reported separately. - Please check comments below for validation results
…o-end testing (#764) Adding satwinds from the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 to GDASApp end-to-end testing new files include: parm/atm/obs/config/satwind_seviri_m8.yaml: QC filter YAML for SEVIRI METEOSAT-8 satwinds parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json: JSON containing data format, sensor, and satellite information for SEVIRI METEOSAT-8 satwinds ush/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.py: bufr2ioda code for extracting SEVIRI METEOSAT-8 satwinds from BUFR End-to-end testing results are provided in #758 - Note: Thinning filter is provided in YAML but turned off due to irreconcilable differences with GSI in testing. --------- Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
…765) End-to-end testing for Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11 new files include: parm/ioda/bufr2ioda//bufr2ioda_sevcsr.json : JSON containing data format, sensor/bufr2ioda_sevcsr.json, and satellite information ush/ioda/bufr2ioda/bufr2ioda_sevcsr.py: bufr2ioda code for extracting SEVIRI METEOSAT-8 and METEOSAT-11 from BUFR
azadeh-gh
requested review from
CoryMartin-NOAA,
emilyhcliu,
ADCollard,
nicholasesposito,
BrettHoover-NOAA and
XuanliLi-NOAA
|
Orion test
in a working copy of Copy Ensure seviri satbias, satbias_cov, and tlapse files are in appropriate Execute prepatmiodaobs, atmanlinit, and atmanlrun. The first two jobs run to completion. SEVIRI dump files are written to the appropriate COM_OBS directory atmanlrun begins but Notice that atms ioda format dump files contain MetaData |
|
I changed |
Small code-change to include SEVIRI/METEOSAT-11 satwinds in end-to-end validation. This uses the hook-ins already provided by the SEVIRI/METEOSAT-8 satwinds code that was merged in #764. New file: parm/atm/obs/config/satwind_seviri_m11.yaml - this is nearly a direct copy of parm/atm/obs/config/satwind_seviri_m8.yaml, since they utilize the same filters. Only the header information has changed. Modified file: parm/ioda/bufr2ioda/bufr2ioda_satwind_amv_seviri.json - METEOSAT-11 was added to the `satellite_info` group End-to-end testing looks very similar to what was seen for SEVIRI/METEOSAT-8 satwinds, detailed in #758. A brief run-down: (LW)IR winds are in QC agreement down to a 0.02% difference (cloud-top)WV winds are in QC agreement down to a 0.20% difference VIS winds differ by 4.5%, but there are only 22 accepted winds in GSI and 23 in JEDI, the difference is a single satwind All ob, HofX, and ob-minus-HofX differences look good Increment summary, from gdasatmanlrun.log: ``` 0: ---------------------------------------------------------------------------------------------------- 0: Increment print | number of fields = 8 | cube sphere face size: C768 0: eastward_wind | Min:-3.479765e-01 Max:+3.430433e-01 RMS:+4.990993e-04 0: northward_wind | Min:-4.226316e-01 Max:+3.376561e-01 RMS:+4.889429e-04 0: air_temperature | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: surface_pressure | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: specific_humidity | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: cloud_liquid_ice | Min:+0.000000e+00 Max:+1.618770e-20 RMS:+1.293217e-23 0: cloud_liquid_water | Min:+0.000000e+00 Max:+1.474788e-19 RMS:+2.167418e-22 0: ozone_mass_mixing_ratio | Min:+0.000000e+00 Max:+0.000000e+00 RMS:+0.000000e+00 0: ---------------------------------------------------------------------------------------------------- ``` Co-authored-by: Brett Hoover <bhoover@Orion-login-1.HPC.MsState.Edu>
I changed the base from feature/gdas-validation to develop. |
…etaData/cloudAmount"
…from bufr.Modified the filters in the YAMLs
|
Please update |
|
@azadeh-gh , File These yamls are not listed in Do you want assistance with this? |
|
@RussTreadon-NOAA Do I need to update the seviri YAMLs to convert to jinja2? |
|
Yes, the seviri yamls need to be updated to jinja2 templates
Beware: If you change the file suffix via |
Please enter the commit message for your changes. Lines starting
|
@RussTreadon-NOAA Thank you. I updated the YAMLS to jinja2. |
|
Reviewers. Please look at the T2O Issue #99 related to this PR for test results. @azadeh-gh I have two comments:
|
|
@azadeh-gh , the conflict in |
…lis to resolve the conflict in gdas_prototype_3d.yaml.j2
|
Summary of validation resultsNo Thinning without QC (UFO vs. GSI) - Channel 5 Time Level 3
No Thinning with QC: UFO vs. GSI - Channel 5 Time Level 1
Here are the channel statistics plots (#obs, bias/std for OMF and final obs error) for JEDI and GDAS runs (Thinned) O-F Bias/Standard Deviation
Final Obs Error
Number Obs Passed QC
O-F Bias/Standard Deviation
Final Obs Error
Number Obs Passed QC
|
Thank you Russ. Updated the |
Thank you for your suggestions.
|
|
@azadeh-gh , I went to test Do we know why all 2024 |
|
@RussTreadon-NOAA We cannot do a real time test of SEVIRI as the CSR stream has been discontinued. The same information is in the ASR stream, so the next step after this is to create an ASR converter. |
|
Thank you @ADCollard for explaining what I am seeing. |
* origin/develop: Use <filesystem> on a non c++17 supported machine (WCOSS ACORN) (#1026) Change generate_com to declare_from_tmpl (#1025) Commenting out more of the marine bufr 2 ioda stuff (#1018) make driver consistent with workflow driver (#1016) Update hashes now that GSI-B is working for EnVar (#1015) Add GitHub CLI to path for CI (#1014) Use _anl rather than _ges dimensions for increments in FV3 increment converter YAML (#1013) Fix inconsistent VIIRS preprocessing test (#1012) remove gdas_ prefix from executable filename in test_gdasapp_fv3jedi_fv3inc (#1010) Bugfix on Broken GHRSST Ioda Converter (#1004) Moved the marine converters to a "safe" place (#1007) restore ATM local ensemble ctest functionality (#1003) Add BUFR2IODA python API converter to prepoceanobs task (#914) Remove sst's from obs proc (#1001) JEDI increment write to cubed sphere history (#983) [End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing (#766)
(#765)
End-to-end testing for Spinning Enhanced Visible and InfraRed Imager (SEVIRI) from METEOSAT-8 and to METEOSAT-11
New files include:
parm/ioda/bufr2ioda//bufr2ioda_sevcsr.json: JSON containing data format, sensor/bufr2ioda_sevcsr.json, and satellite informationush/ioda/bufr2ioda/bufr2ioda_sevcsr.py: bufr2ioda python code for converting SEVIRI METEOSAT-8 and METEOSAT-11 from BUFR to IODAparm/atm/obs/config/seviri_m08.yaml.j2parm/atm/obs/config/seviri_m11.yaml.j2parm/atm/obs/lists/gdas_prototype_3d.yaml.j2