Changes needed to run OU-MAP GSI-EnKF assimilation of reflectivity #2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
observations, in a second step after EnKF assimilation of conventional observations.
hu5970
approved these changes
May 26, 2022
hu5970
pushed a commit
that referenced
this pull request
Oct 6, 2023
… (see issue NOAA-EMC#601) (NOAA-EMC#614) Adding code to analyze the siginificant wave heigh in GSI 3D Analysis, esp. for FV3-LAM model based DA, eg. RRFS-DA, RRFS-3DRTMA. (Also see the issue in EMC GSI github repository: NOAA-EMC#601 Adding I/O for Analysis of Significant Wave Height for 3DRTMA) <!-- PLEASE READ --> <!-- Before opening a PR, please note these guidelines: - Each PR should only address ONE topic and have an associated issue - No hardcoded or paths to personal directories should be present - No temporary or backup files should be committed - Any code that was disabled by being commented out should be removed --> **Description** <!-- Please include relevant motivation and context. --> Significant Wave Height (hereafter as SWH) is one of the standard products provided by the operational (2D)RTMA. To continuously provide the same products in 3DRTMA, the next-generation RTMA, some efforts in GSI code need to be made in order to analyze the SWH in 3D analysis of GSI. <!-- Please include a summary of the change and which issue is fixed. --> The kernel subroutines to assimilate SWH in GSI (such as stphowv.f90, setuphowv.f90, inthowv.f90, gsi_howvOper.f90 and m_howvNode.f90) already had been added for (2D)RTMA years ago by Manuel Pondeca, so for this issue, the code work mainly focus on adding the I/O of SWH in background and analysis fields for 3DRTMA (esp. RRFS-based 3DRTMA), and some necessary modifications in background error, options, variables related to analysis of SWH, etc. Modified code in GSI: 1. rapidrefresh_cldsurf_mod.f90: adding a few variables related to the analysis of howv in 3D analysis 2. gsimod.F90: adding namelist options used for analysis of howv in 3D analysis 3. m_berror_stats_reg.f90: added some code for the special treatment to the static background error (BE) of howv 4. read_prepbufr.f90: adding code to decode the observation of howv in prepbufr file when howv is available in firstguess 5. setuphowv.f90: adding code to use obs of howv when howv is available in firstguess 6. gsi_rfv3io_mod.f90: adding I/O code to read in howv from firstguess and write out howv into analysis. <!-- List any dependencies that are required for this change. --> No dependencies are required for this change. <!-- Please provide reference to the issue this pull request is addressing. --> This PR is addressing the issue [NOAA-EMC#601](NOAA-EMC#601): Adding code to analyze the siginificant wave heigh in GSI 3D Analysis". <!-- For e.g. Fixes #IssueNumber --> Fixes NOAA-EMC#601 **Type of change** Please delete options that are not relevant. - [*] New feature (non-breaking change which adds functionality) **How Has This Been Tested?** <!-- Please describe the tests that you ran to verify your changes and on the platforms these tests were conducted. --> - Brief results from ctest (regression test) with the modified code (on WCOSS2 - Cactus): [gang.zhao@clogin07:build] (feature/3drtma_howv)$ ctest -N Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Test #1: global_3dvar Test #2: global_4dvar Test #3: global_4denvar Test #4: hwrf_nmm_d2 Test #5: hwrf_nmm_d3 Test #6: rtma Test #7: rrfs_3denvar_glbens Test #8: netcdf_fv3_regional Test #9: global_enkf Total Tests: 9 Test #1: global_3dvar [gang.zhao@clogin04:build] (feature/3drtma_howv)$ ctest -R global_3dvar Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 1: global_3dvar 1/1 Test #1: global_3dvar ..................... Passed 1631.12 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 1631.14 sec Test #2: global_4dvar [gang.zhao@clogin09:build] (feature/3drtma_howv)$ ctest -R global_4dvar Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 2: global_4dvar 1/1 Test #2: global_4dvar ..................... Passed 2462.19 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 2462.23 sec Test #3: global_4denvar [gang.zhao@clogin04:build] (feature/3drtma_howv)$ ctest -R global_4denvar Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 3: global_4denvar 1/1 Test #3: global_4denvar ................... Passed 1922.43 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 1922.46 sec Test #4: hwrf_nmm_d2 [gang.zhao@clogin09:build] (feature/3drtma_howv)$ ctest -R hwrf_nmm_d2 Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 4: hwrf_nmm_d2 1/1 Test #4: hwrf_nmm_d2 ...................... Passed 1214.10 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 1214.20 sec Test #5: hwrf_nmm_d3 [gang.zhao@clogin09:build] (feature/3drtma_howv)$ ctest -R hwrf_nmm_d3 Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 5: hwrf_nmm_d3 1/1 Test #5: hwrf_nmm_d3 ...................... Passed 736.38 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 736.50 sec Test #6: rtma [gang.zhao@clogin05:build] (feature/3drtma_howv)$ ctest -R rtma Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 6: rtma 1/1 Test #6: rtma ............................. Passed 1027.01 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 1027.01 sec Test #7: rrfs_3denvar_glbens [gang.zhao@clogin06:build] (feature/3drtma_howv)$ ctest -R rrfs_3denvar_glbens Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 7: rrfs_3denvar_glbens 1/1 Test #7: rrfs_3denvar_glbens .............. Passed 484.69 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 484.70 sec Test #8: netcdf_fv3_regional [gang.zhao@clogin03:build] (feature/3drtma_howv)$ ctest -R netcdf_fv3_regional Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 8: netcdf_fv3_regional 1/1 Test #8: netcdf_fv3_regional .............. Passed 483.08 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 483.11 sec Test #9: global_enkf [gang.zhao@clogin03:build] (feature/3drtma_howv)$ ctest -R global_enkf Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/WaveHgt/develop/build Start 9: global_enkf 1/1 Test #9: global_enkf ...................... Passed 488.50 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 488.57 sec - The modified GSI code passed the regression tests (all 9 tasks) on Hera and WCOSS2 (Cactus). - adding the analysis of howv only has very trial influences on the analyses of other variables. Here is the statistics of the differences of other variables (u/v/t/ps/q/t2m/q2m) from the runs of GSI without howv vs. with howv (from a testing case 2023-07-12_14:00:00 UTC in 3km North-American domain): comparing two netcdf files: fcst_fv3lam_hyb_betas/INPUT/fv_core.res.tile1.nc fcst_fv3lam_nodata_noinfo/INPUT/fv_core.res.tile1.nc ... Variable Group Count Sum AbsSum Min Max Range Mean StdDev u / 602135550 3926.84 25760.8 -0.1026 0.485788 0.588388 6.52152e-06 0.00115817 v / 620166777 -4891.34 32582.5 -0.835774 0.268402 1.10418 -7.88714e-06 0.00197793 T / 155987083 178.048 6497.51 -0.0246582 0.0384064 0.0630646 1.14143e-06 0.000218737 delp / 19559676 -281.532 3008.29 -0.00292969 0.00219727 0.00512695 -1.43935e-05 0.000183727 comparing two netcdf files: fcst_fv3lam_hyb_betas/INPUT/fv_tracer.res.tile1.nc fcst_fv3lam_nodata_noinfo/INPUT/fv_tracer.res.tile1.nc ... Variable Group Count Sum AbsSum Min Max Range Mean StdDev sphum / 430707614 0.594287 2.77816 -2.6139e-05 3.1759e-05 5.7898e-05 1.37979e-09 8.03072e-08 comparing two netcdf files: fcst_fv3lam_hyb_betas/INPUT/sfc_data.nc fcst_fv3lam_nodata_noinfo/INPUT/sfc_data.nc ... Variable Group Count Sum AbsSum Min Max Range Mean StdDev t2m / 10665000 43.3899 135.095 -0.00152825 0.00686629 0.00839454 4.06844e-06 5.02866e-05 q2m / 10665000 0.0192553 0.124707 -3.1476e-06 1.77554e-05 2.0903e-05 1.80547e-09 5.89657e-08 It could be seen that the differences are trivial and ignorable. <!-- Provide instructions so we can reproduce. --> The regression tests were done by following the instructions of "[GSI Ctests (regression tests)](https://github.com/NOAA-EMC/GSI/wiki/GSI-Ctests-(regression-tests))" in [GSI Wiki](https://github.com/NOAA-EMC/GSI/wiki) <!-- Please also list any relevant details for your test configuration --> The modified code had also been tested with a testing case 2023-07-12_14:00:00 UTC for 3km North-American domain Here is a brief summary of the test results: 1. Here is the analysis increment of Significant Wave Height (aka howv hereafter): pure 3dvar, static background error of howv is 0.42 meters, and the de-correlation length scale is 170km.  2. The following figure shows the distribution of howv in the analysis (used obs is in green, rejected in red). Obviously the location of used obs of howv match the area of non-zero analysis increments of howv.  3. The following figure is the analysis increment of howv with hybrid envar analysis (using gdas ensemble 80 members and the ensemble weight is 84%), and the static BE of howv is tuned/inflated. The analysis increments are very similar to the results from pure 3dvar run (see the first figure)  4. The last figure shows the analysis increments of howv with hybrid envar analysis (using gdas ensemble 80 members and the ensemble weight is 84%), but the static BE of howv is NOT tuned. It can be observed that the analysis increments is less than the results from the hybrid run with tuning the static BE of howv. That is because the weight of static BE (16%) reduced the background error of howv (ensemble of howv is not available yet), so the impact of obs is decreased.  **Checklist** - [*] My code follows the style guidelines of this project - [*] I have performed a self-review of my own code - [*] I have commented my code, particularly in hard-to-understand areas - [*] New and existing tests pass with my changes - [*] Any dependent changes have been merged and published **DUE DATE for this PR is 10/5/2023.** If this PR is not merged into `develop` by this date, the PR will be closed and returned to the developer.
hu5970
pushed a commit
that referenced
this pull request
May 7, 2024
**DUE DATE for merger of this PR into `develop` is 2/19/2024 (six weeks after PR creation).** **DUE DATE for this PR is extended to 3/19/2024 because @XuLu-NOAA is on leave.** **Description** Xu Lu (xu.lu@noaa.gov) and Biju Thomas (biju.thomas@noaa.gov) fixed bugs regarding HAFS GSI debug build and run issues. This is in corresponding to issue NOAA-EMC#661 Fixes NOAA-EMC#661 1. In read_radar.f90, uninitialized toff is making all the ground-based radar observations be placed at -3h instead of 0h, which creates wrong increments for FGAT and 4DEnVar. 2. In read_radar.f90, uninitialized zsges will crash the debug mode. 3. In read_radar.f90, t4dvo should be used instead of t4dv in the read_radar_l2rw_novadqc subroutine. 4. In radinfo.90, maxscan should be increased to at least 252 to allow more scans, or it will crash the debug mode. 5. In read_fl_hdob.f90, dlnpsob is replaced with 1000. since the SFMR does not sample surface pressure, and the uninitialized dlnpsob creates issues later in setupspd.f90 in the debug mode. 6. In mod_fv3_lola.f90, (i,j+1) should be used instead of (i+1,j) in searching for V edges. 7. In stpcalc.f90, when tried to find the best stepsize from outpen around L838-864, the minimum outstp(i) is stored in stp(ii), but the istp_use is asigned with i instead of ii. Create inconsistency when assigning stp(istp_use) to stpinout at L872. Should use istp_use=ii instead. **Type of change** - [Yes] Bug fix (non-breaking change which fixes an issue) **How Has This Been Tested?** Regression test on Orion: ``` Test project /work/noaa/hwrf/save/xulu/mergeversions/GSI/build CMake Warning (dev) at CTestTestfile.cmake:9 (subdirs): Syntax Warning in cmake code at /work/noaa/hwrf/save/xulu/mergeversions/GSI/build/regression/CTestTestfile.cmake:7:10 1/7 Test #4: [=[netcdf_fv3_regional]=] ........ Passed 365.11 sec 2/7 Test #7: [=[global_enkf]=] ................ Passed 430.29 sec 3/7 Test #3: [=[rrfs_3denvar_glbens]=] ........ Passed 605.35 sec 4/7 Test #2: [=[rtma]=] ....................... Passed 969.78 sec 5/7 Test #6: [=[hafs_3denvar_hybens]=] ........***Failed 1455.47 sec 6/7 Test #1: [=[global_4denvar]=] ............. Passed 1682.40 sec 7/7 Test #5: [=[hafs_4denvar_glbens]=] ........***Failed 1758.90 sec ``` The failed hafs_3denvar and 4denvar are within expectation due to the fix for toff. As demonstrated in the single observation tests in the following figure, the uninitialized toff can result in increment degradations due to wrongly assigned observation times: 
hu5970
pushed a commit
that referenced
this pull request
May 29, 2024
…ed 3DRTMA (NOAA-EMC#730) <!-- PLEASE READ --> <!-- Before opening a PR, please note these guidelines: - Each PR should only address ONE topic and have an associated issue - No hardcoded or paths to personal directories should be present - No temporary or backup files should be committed - Any code that was disabled by being commented out should be removed --> **Description** <!-- Please include relevant motivation and context. --> <!-- Please include a summary of the change and which issue is fixed. --> <!-- List any dependencies that are required for this change. --> To improve the analysis of the near-surface wind gust in 3DRTMA, the observations of near-surface wind gust would be analyzed directly in GSI (3DVar and Hybrid 3DEnVar), instead of being a derived product from the near-surface wind analysis. Since the core subroutines for direct variational assimilation of wind gust (e.g., setupgust.f90, intgust.f90, stpgust.f90, etc.) had already been implemented in GSI for 2DRTMA, so in the work the development in GSI mainly focus on adding I/O of 2-D wind gust firstguess and analysis fields for RRFS-based 3DRTMA, and some minor modifications in observation and background error for wind gust, options to control the analysis of wind gust, etc. <!-- Please provide reference to the issue this pull request is addressing. --> <!-- For e.g. Fixes #IssueNumber --> This PR is to address the issue NOAA-EMC#726 : Adding I/O for direct analysis of near-surface wind gust for RRFS-based 3DRTMA Fixes NOAA-EMC#726 **Type of change** Please delete options that are not relevant. - [x] New feature (non-breaking change which adds functionality) **How Has This Been Tested?** <!-- Please describe the tests that you ran to verify your changes and on the platforms these tests were conducted. --> <!-- Provide instructions so we can reproduce. --> <!-- Please also list any relevant details for your test configuration --> **Checklist** - [x] My code follows the style guidelines of this project - [x] I have performed a self-review of my own code - [x] I have commented my code, particularly in hard-to-understand areas - [x] New and existing tests pass with my changes tested with a real case - 2024-02-20_12:00Z, 1. dry-run (using my updated GSI code with wind gust analysis, but actually no wind gust obs is analyzed, so-called dryrun) is compared to control-run (original GSI code running without wind gust obs): the results are identical. This indicates that if without analyzing wind-gust obs, then the updated code generates the analysis identical to the analysis of original/control code. Or say, the added code does not have influence on the other part of code. 2. real case run with updated GSI code to analyze the obs of wind gust: The following figure shows the used observations of near-surface wind gust:  the following figure shows the analysis increments:  - [x] Any dependent changes have been merged and published - [x] Regression tests on WCOSS2 (Cactus) and Hera (Rocky-8) : my updated GSI commit [#f91f247d](GangZhao-NOAA@f91f247)) vs control/original GSI code (commit [#6d9ebbb7](NOAA-EMC@6d9ebbb)) Here is the reports of the regression tests on WCOSS2 (Cactus): ~~~ [gang.zhao@clogin02:build] (feature/windgust_in_3dvar_for_3drtma)$ ctest -j 7 Test project /lfs/h2/emc/da/save/gang.zhao/WorkDir/ProdGSI_Dev/gsi_dev/build Start 1: global_4denvar Start 2: rtma Start 3: rrfs_3denvar_glbens Start 4: netcdf_fv3_regional Start 5: hafs_4denvar_glbens Start 6: hafs_3denvar_hybens Start 7: global_enkf 1/7 Test #4: netcdf_fv3_regional .............. Passed 483.15 sec 2/7 Test #3: rrfs_3denvar_glbens .............. Passed 486.74 sec 3/7 Test #7: global_enkf ...................... Passed 850.98 sec 4/7 Test #2: rtma ............................. Passed 970.28 sec 5/7 Test #6: hafs_3denvar_hybens .............. Passed 1152.82 sec 6/7 Test #5: hafs_4denvar_glbens .............. Passed 1213.93 sec 7/7 Test #1: global_4denvar ................... Passed 1683.16 sec 100% tests passed, 0 tests failed out of 7 Total Test time (real) = 1683.19 sec ~~~ Here is the reports of the regression tests on Hera (Rocky8): ~~~ (base) [Gang.Zhao@hfe11:build] (feature/windgust_in_3dvar_for_3drtma)$ ctest -j 7 Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_dev/build Start 1: global_4denvar Start 2: rtma Start 3: rrfs_3denvar_glbens Start 4: netcdf_fv3_regional Start 5: hafs_4denvar_glbens Start 6: hafs_3denvar_hybens Start 7: global_enkf 1/7 Test #4: netcdf_fv3_regional .............. Passed 491.53 sec 2/7 Test #3: rrfs_3denvar_glbens ..............***Failed 495.27 sec 3/7 Test #2: rtma ............................. Passed 982.45 sec 4/7 Test #6: hafs_3denvar_hybens .............. Passed 1168.99 sec 5/7 Test #7: global_enkf ...................... Passed 1239.77 sec 6/7 Test #5: hafs_4denvar_glbens ..............***Failed 1347.87 sec 7/7 Test #1: global_4denvar ................... Passed 1974.45 sec 71% tests passed, 2 tests failed out of 7 Total Test time (real) = 1974.91 sec The following tests FAILED: 3 - rrfs_3denvar_glbens (Failed) 5 - hafs_4denvar_glbens (Failed) Errors while running CTest Output from these tests are in: /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_dev/build/Testing/Temporary/LastTest.log Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely. (base) [Gang.Zhao@hfe11:build] (feature/windgust_in_3dvar_for_3drtma)$ ctest -R rrfs_3denvar_glbens Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_dev/build Start 3: rrfs_3denvar_glbens 1/1 Test #3: rrfs_3denvar_glbens .............. Passed 430.52 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 430.55 sec (base) [Gang.Zhao@hfe11:build] (feature/windgust_in_3dvar_for_3drtma)$ ctest -R hafs_4denvar_glbens Test project /scratch1/NCEPDEV/da/Gang.Zhao/ProdGSI_dev/gsi_dev/build Start 5: hafs_4denvar_glbens 1/1 Test #5: hafs_4denvar_glbens .............. Passed 1225.37 sec 100% tests passed, 0 tests failed out of 1 Total Test time (real) = 1225.39 sec ~~~ **Note**: _When I was running the regression tests, GSI code was just updated to the latest commit [#b53740a7](GangZhao-NOAA@f91f247). Considering the frequent update in EMC GSI code recently and saving the time, after this PR has been reviewed and approved by peer-reviewers, I will update the code to latest EMC GSI commit, then re-run the regression tests for final approval.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
observations, in a second step after EnKF assimilation of
conventional observations.