New "atmanlfv3inc" Rocoto job

[DavidNew-NOAA]

This PR, a companion to GDASApp PR #983, creates a new Rocoto job called "atmanlfv3inc" that computes the FV3 atmosphere increment from the JEDI variational increment using a JEDI OOPS app in GDASApp, called fv3jedi_fv3inc.x, that replaces the GDASApp Python script, jediinc2fv3.py, for the variational analysis. The "atmanlrun" job is renamed "atmanlvar" to better reflect the role it plays of running now one of two JEDI executables for the atmospheric analysis jobs.

Description

Previously, the JEDI variational executable would interpolate and write its increment, during the atmanlrun job, to the Gaussian grid, and then the python script, jediinc2fv3.py, would read it and then write the FV3 increment on the Gaussian grid during the atmanlfinal job. Following the new changes, the JEDI increment will be written directly to the cubed sphere. Then during the atmanlfv3inc job, the OOPS app will read it and compute the FV3 increment directly on the cubed sphere and write it out onto the Gaussian grid.

The reason for writing first to the cubed sphere grid is that otherwise the OOPS app would have to interpolate twice, once from Gaussian to cubed sphere before computing the increment and then back to the Gaussian, since all the underlying computations in JEDI are done on the native grid.

The motivation for this new app and job is that eventually we wish to transition all intermediate data to the native cubed sphere grid, and the OOPS framework allows us the flexibility to read and write to/from any grid format we wish by just changing the YAML configuration file rather than hardcoding. When we do switch to the cubed sphere, it will be an easy transition. Moreover, it the computations the OOPS app will be done with a compiled executable rather than an interpreted Python script, providing some performance increase.

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES
• Does this change require a documentation update? NO

How has this been tested?

It has been tested with a cycling experiment with JEDI in both Hera and Orion to show that it runs without issues, and I have compared the FV3 increments computed by the original and news codes. The delp and hydrostatic delz increments, the key increments produced during this step, differ by a relative error of 10^-7 and 10^-2 respectively. This difference is most likely due to the original python script doing its internal computation on the interpolated Gaussian grid, while the new OOPS app does its computations on the native cubed sphere before interpolating the the Gaussian grid.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[DavidNew-NOAA]

Closing until changes made

[RussTreadon-NOAA]

Thank you both for your quick replies. Hopefully we can move this PR forward and get it into develop soon.

[aerorahul]

@DavidNew-NOAA Can you please update the pointer to gdasapp from this AM's commit after fc62ef5 and we can kick off the test again.

[DavidNew-NOAA]

@aerorahul hash updated

[RussTreadon-NOAA]

Orion tests
Complete the following on Orion:

1. Install DavidNew-NOAA:feature/jediinc2fv3 at 94d28d56 in HOMEgfs=/work2/noaa/da/rtreadon/git/global-workflow/jediinc2fv3
2. cd $HOMEgfs/sorc
3. ./build_all.sh -g -u -v
4. link_workflow.sh

Enable and run g-w JEDI ATM CI as follows:

1. cd $HOMEgfs/ci/cases/pr. Remove - orion entry from C96C48_ufs_hybatmDA.yaml
2. cd /work2/noaa/stmp/rtreadon
3. edit setup_ci.sh to point at above $HOMEgfs. Execute ./setup_ci.sh. Create EXPDIR and COMROOT for prtest in which to run C96C48 JEDI ATM cycle.
4. activate crontab to rocotorun prtest

The 2024022318 half cycle is complete. The 2024022400 gdas and enkfgdas cycle is complete. JEDI ATM jobs ran successfully in the gdas, enkfgdas, and gfs cycles

202402240000      gdasprepatmiodaobs                    17549303           SUCCEEDED                   0         1         362.0
202402240000          gdasatmanlinit                    17549333           SUCCEEDED                   0         1          66.0
202402240000           gdasatmanlvar                    17549356           SUCCEEDED                   0         1         614.0
202402240000        gdasatmanlfv3inc                    17549385           SUCCEEDED                   0         1          68.0
202402240000         gdasatmanlfinal                    17549387           SUCCEEDED                   0         1         141.0
...
202402240000    enkfgdasatmensanlinit                    17549334           SUCCEEDED                   0         1          66.0
202402240000    enkfgdasatmensanlrun                    17549357           SUCCEEDED                   0         1         478.0
202402240000    enkfgdasatmensanlfinal                    17549379           SUCCEEDED                   0         1         130.0
...
202402240000       gfsprepatmiodaobs                    17549304           SUCCEEDED                   0         1         367.0
202402240000           gfsatmanlinit                    17549335           SUCCEEDED                   0         1          70.0
202402240000            gfsatmanlvar                    17549358           SUCCEEDED                   0         1         516.0
202402240000         gfsatmanlfv3inc                    17549380           SUCCEEDED                   0         1          75.0
202402240000          gfsatmanlfinal                    17549386           SUCCEEDED                   0         1         111.0

Enable and run GDASApp test_gdasapp as follows:

1. cd $HOMEgfs/sorc/gdas.cd
2. module use $HOMEgfs/sorc/gdas.cd/modulefiles
3. module load GDAS/orion.intel
4. set environment variables for test_gdasapp
5. cd $HOMEgfs/sorc/gdas.cd/build
6. ctest -R test_gdasapp

All 45 tests pass.

Test project /work/noaa/da/rtreadon/git/global-workflow/jediinc2fv3/sorc/gdas.cd/build
      Start 1393: test_gdasapp_util_coding_norms
 1/54 Test #1393: test_gdasapp_util_coding_norms ........................   Passed    1.92 sec
      Start 1394: test_gdasapp_util_ioda_example
 2/54 Test #1394: test_gdasapp_util_ioda_example ........................   Passed    3.24 sec
      Start 1395: test_gdasapp_util_prepdata
 3/54 Test #1395: test_gdasapp_util_prepdata ............................   Passed    1.00 sec
      Start 1396: test_gdasapp_util_rads2ioda
 4/54 Test #1396: test_gdasapp_util_rads2ioda ...........................   Passed    0.09 sec
      Start 1397: test_gdasapp_util_ghrsst2ioda
 5/54 Test #1397: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.06 sec
      Start 1398: test_gdasapp_util_smap2ioda
 6/54 Test #1398: test_gdasapp_util_smap2ioda ...........................   Passed    0.06 sec
      Start 1399: test_gdasapp_util_smos2ioda
 7/54 Test #1399: test_gdasapp_util_smos2ioda ...........................   Passed    0.06 sec
      Start 1400: test_gdasapp_util_viirsaod2ioda
 8/54 Test #1400: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.06 sec
      Start 1401: test_gdasapp_util_icecamsr2ioda
 9/54 Test #1401: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.06 sec
      Start 1739: test_gdasapp_check_python_norms
10/54 Test #1739: test_gdasapp_check_python_norms .......................   Passed    2.68 sec
      Start 1740: test_gdasapp_check_yaml_keys
11/54 Test #1740: test_gdasapp_check_yaml_keys ..........................   Passed    0.06 sec
      Start 1741: test_gdasapp_jedi_increment_to_fv3
12/54 Test #1741: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    0.32 sec
      Start 1742: test_gdasapp_setup_cycled_exp
13/54 Test #1742: test_gdasapp_setup_cycled_exp .........................   Passed    4.98 sec
      Start 1743: test_gdasapp_fv3jedi_fv3inc
14/54 Test #1743: test_gdasapp_fv3jedi_fv3inc ...........................   Passed   22.19 sec
      Start 1744: test_gdasapp_convert_bufr_temp_dbuoy
15/54 Test #1744: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    0.16 sec
      Start 1745: test_gdasapp_convert_bufr_salt_dbuoy
16/54 Test #1745: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.13 sec
      Start 1746: test_gdasapp_convert_bufr_temp_mbuoyb
17/54 Test #1746: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.15 sec
      Start 1747: test_gdasapp_convert_bufr_salt_mbuoyb
18/54 Test #1747: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.13 sec
      Start 1748: test_gdasapp_convert_bufr_tesacprof
19/54 Test #1748: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.14 sec
      Start 1749: test_gdasapp_convert_bufr_trkobprof
20/54 Test #1749: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.13 sec
      Start 1750: test_gdasapp_convert_bufr_sfcships
21/54 Test #1750: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.13 sec
      Start 1751: test_gdasapp_convert_bufr_sfcshipsu
22/54 Test #1751: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.13 sec
      Start 1752: test_gdasapp_soca_nsst_increment_to_mom6
23/54 Test #1752: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   62.23 sec
      Start 1753: test_gdasapp_soca_prep
24/54 Test #1753: test_gdasapp_soca_prep ................................   Passed    1.60 sec
      Start 1754: test_gdasapp_soca_run_clean
25/54 Test #1754: test_gdasapp_soca_run_clean ...........................   Passed    0.01 sec
      Start 1755: test_gdasapp_soca_setup_obsprep
26/54 Test #1755: test_gdasapp_soca_setup_obsprep .......................   Passed   29.45 sec
      Start 1756: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/54 Test #1756: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   52.85 sec
      Start 1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/54 Test #1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   74.14 sec
      Start 1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/54 Test #1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   42.14 sec
      Start 1759: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/54 Test #1759: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.14 sec
      Start 1760: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN
31/54 Test #1760: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN ....   Passed   42.16 sec
      Start 1761: test_gdasapp_soca_copy_scratch
32/54 Test #1761: test_gdasapp_soca_copy_scratch ........................   Passed    0.43 sec
      Start 1762: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
33/54 Test #1762: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.14 sec
      Start 1763: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
34/54 Test #1763: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   10.14 sec
      Start 1764: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
35/54 Test #1764: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....   Passed  170.14 sec
      Start 1765: test_gdasapp_soca_socahybridweights
36/54 Test #1765: test_gdasapp_soca_socahybridweights ...................   Passed   42.12 sec
      Start 1766: test_gdasapp_soca_incr_handler
37/54 Test #1766: test_gdasapp_soca_incr_handler ........................   Passed    2.10 sec
      Start 1767: test_gdasapp_soca_ens_handler
38/54 Test #1767: test_gdasapp_soca_ens_handler .........................   Passed   10.10 sec
      Start 1768: test_gdasapp_snow_create_ens
39/54 Test #1768: test_gdasapp_snow_create_ens ..........................   Passed    0.92 sec
      Start 1769: test_gdasapp_snow_imsproc
40/54 Test #1769: test_gdasapp_snow_imsproc .............................   Passed    2.01 sec
      Start 1770: test_gdasapp_snow_apply_jediincr
41/54 Test #1770: test_gdasapp_snow_apply_jediincr ......................   Passed    2.60 sec
      Start 1771: test_gdasapp_snow_letkfoi_snowda
42/54 Test #1771: test_gdasapp_snow_letkfoi_snowda ......................   Passed    8.37 sec
      Start 1772: test_gdasapp_convert_bufr_adpsfc_snow
43/54 Test #1772: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    2.28 sec
      Start 1773: test_gdasapp_convert_bufr_adpsfc
44/54 Test #1773: test_gdasapp_convert_bufr_adpsfc ......................   Passed    2.99 sec
      Start 1774: test_gdasapp_convert_gsi_satbias
45/54 Test #1774: test_gdasapp_convert_gsi_satbias ......................   Passed    1.35 sec
      Start 1775: test_gdasapp_setup_atm_cycled_exp
46/54 Test #1775: test_gdasapp_setup_atm_cycled_exp .....................   Passed    0.85 sec
      Start 1776: test_gdasapp_atm_jjob_var_init
47/54 Test #1776: test_gdasapp_atm_jjob_var_init ........................   Passed   44.11 sec
      Start 1777: test_gdasapp_atm_jjob_var_run
48/54 Test #1777: test_gdasapp_atm_jjob_var_run .........................   Passed  106.16 sec
      Start 1778: test_gdasapp_atm_jjob_var_inc
49/54 Test #1778: test_gdasapp_atm_jjob_var_inc .........................   Passed   42.15 sec
      Start 1779: test_gdasapp_atm_jjob_var_final
50/54 Test #1779: test_gdasapp_atm_jjob_var_final .......................   Passed   42.16 sec
      Start 1780: test_gdasapp_atm_jjob_ens_init
51/54 Test #1780: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.90 sec
      Start 1781: test_gdasapp_atm_jjob_ens_run
52/54 Test #1781: test_gdasapp_atm_jjob_ens_run .........................   Passed  266.18 sec
      Start 1782: test_gdasapp_atm_jjob_ens_final
53/54 Test #1782: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.17 sec
      Start 1783: test_gdasapp_aero_gen_3dvar_yaml
54/54 Test #1783: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.72 sec

100% tests passed, 0 tests failed out of 54

Label Time Summary:
gdas-utils    =   6.55 sec*proc (9 tests)
script        =   6.55 sec*proc (9 tests)

Total Test time (real) = 1271.07 sec

[RussTreadon-NOAA]

@WalterKolczynski-NOAA , what else remains to be done before this PR can be merged into develop?

[TerrenceMcGuinness-NOAA]

@RussTreadon-NOAA The Continuous Integration tests to check the PR agains the develop build and run base line almost passed, but we ran against discrepancy in our validation of success that produced a false negative in this last case. We are working to fix that now.

[RussTreadon-NOAA]

Thank you @TerrenceMcGuinness-NOAA for sharing what's going on. Does this problem prevent merger of this PR into develop?

[TerrenceMcGuinness-NOAA]

Yeah, @RussTreadon-NOAA that's the role of the Branch Manger of course, but not sure why it would be a problem.