Change $(foo) to {{ foo }} in YAMLs #928

CoryMartin-NOAA · 2024-02-16T17:56:05Z

To be consistent throughout GFS/GDAS, we should use {{ foo }} syntax whenever possible in our templates.

Succumbing to peer pressure, ChatGPT helped me write a quick script to do this :-)

guillaumevernieres · 2024-02-16T18:03:57Z

The Cory Pre-trained Transformer ingested by GPT!

…SApp into feature/yaml_syntax

CoryMartin-NOAA · 2024-02-16T18:50:53Z

@RussTreadon-NOAA @guillaumevernieres I'm removing a couple of tests that seem more trouble than they're worth. Please let me know if you think I should try to make them work. I'm of the opinion that the EWOK to GDAS YAML tool needs to be something a bit more comprehensive to be useful, and the Aero YAML generator should be tested through the workflow tests.

andytangborn

Looks good, ChatGPT.

RussTreadon-NOAA

The tile for this PR is simple. The amount of work involved is far from simple. Looks good to me. Approve pending Passed from CI tests.

CoryMartin-NOAA · 2024-02-16T19:02:10Z

I think the CI kicked off before I removed these annoying tests, so we need to verify that if there are failures that they are just the 2 I have since removed. If other failures, I'll investigate next week.

emcbot · 2024-02-16T19:09:35Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Feb 16 18:07:00 UTC 2024 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 16 18:51:53 UTC 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Feb 16 19:09:35 UTC 2024
Tests: 67% tests passed, 17 tests failed out of 52
	1698 - test_gdasapp_convert_ewok_yaml (Failed)
	1713 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1714 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1715 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1717 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1718 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1719 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
	1720 - test_gdasapp_soca_socahybridweights (Failed)
	1721 - test_gdasapp_soca_incr_handler (Failed)
	1722 - test_gdasapp_soca_ens_handler (Failed)
	1731 - test_gdasapp_atm_jjob_var_init (Failed)
	1732 - test_gdasapp_atm_jjob_var_run (Failed)
	1733 - test_gdasapp_atm_jjob_var_final (Failed)
	1734 - test_gdasapp_atm_jjob_ens_init (Failed)
	1735 - test_gdasapp_atm_jjob_ens_run (Failed)
	1736 - test_gdasapp_atm_jjob_ens_final (Failed)
	1737 - test_gdasapp_aero_gen_3dvar_yaml (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/928/global-workflow/sorc/gdas.cd/build/log.ctest

RussTreadon-NOAA · 2024-03-20T16:36:58Z

Hera is now 2/3 Rocky 8. Logins automatically take users to Rocky 8 nodes. The following g-w components failed to build

build_ufs_utils.sh failed with status 2!
build_ww3prepost.sh failed with status 2!
build_ufs.sh failed with status 2!
build_upp.sh failed with status 2!

These components were successfully built

build_gfs_utils.sh completed successfully!
build_gsi_monitor.sh completed successfully!
build_gsi_utils.sh completed successfully!
build_gdas.sh completed successfully!

g-w issue #2329 documents the transition of g-w components to Hera Rocky 8.

RussTreadon-NOAA · 2024-03-20T18:36:03Z

Manually copy changes from PR #981 into working copy of feature/yaml_syntax on Hera. Rerun test_gdasapp. Following failures remain

        1395 - test_gdasapp_util_viirsaod2ioda (Subprocess aborted)
        1750 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1751 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1752 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1753 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN (Failed)
        1755 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1756 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
        1758 - test_gdasapp_soca_socahybridweights (Failed)
        1759 - test_gdasapp_soca_incr_handler (Failed)
        1760 - test_gdasapp_soca_ens_handler (Failed)

Rerun test_gdasapp_util_viirsaod2ioda with -VV. Output shows

1395: OOPS_STATS util::Timers::Total                                 :        13.95       13.95       13.95      100.00        0.00
1395: OOPS_STATS ---------------------------------- Parallel Timing Statistics (   1 MPI tasks) -----------------------------------
1395:
1395: OOPS_STATS Run end                                  - Runtime:      0.02 sec,  Memory: total:    60.02 Mb, per task: min =    60.02 Mb, max =    60.02 Mb
1395: Run: Finishing gdasapp::ObsProvider2IodaApp with status = 0
1395: terminate called after throwing an instance of 'oops::TestReferenceFloatMismatchError'
1395:   what():  Test reference Float mismatch @ Line:4
1395: Test Val : 2.2290000000000001e-03
1395: Ref  Val : 2.1952299999999998e-03
1395: Delta    : 3.3770000000000241e-05
1395: Relative tolerance: 2.2121149999999997e-09
1395: Absolute tolerance: 0.0000000000000000e+00
1395: Test Line: '    Min: 0.002229'
1395: Ref Line : '    Min: 0.00219523'
1/1 Test #1395: test_gdasapp_util_viirsaod2ioda ...Subprocess aborted***Exception:   0.62 sec

Do we need to update the test reference to account for the move to Hera Rocky 8?

Rerun sequence of test_gdasapp_soca jobs leading up to test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP failure. Run this job with -VV. Job log file JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP.out contains

+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[39]: export PYTHONPATH
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[44]: EXSCRIPT=/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/scripts/exgdas_global_marine\
_analysis_prep.py
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[45]: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/scripts/exgdas_global_marine_analysis\
_prep.py
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py", line 9, in <module>
    import f90nml
ModuleNotFoundError: No module named 'f90nml'
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP 1710959269 1

What library or module do we need to load so that f90nml is in the job environment?

CoryMartin-NOAA · 2024-03-20T18:56:57Z

I think module load py-f90nml/1.4.3 can solve one of these problems. For the other, I'm a bit concerned by how much different the results are on Rocky8 from CentOS7... presumably this test passes between Hera Centos7 and Orion, but not Hera Rocky8. I wonder, does it pass on Hercules, which is already Rocky 8?

RussTreadon-NOAA · 2024-03-20T19:38:53Z

Loading py-f90nml/1.4.3 from the command line or adding it to modulefiles/GDAS/hera.intel.lua isn't helping. JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP still fails. Based on -VV output the PYTHONPATH does not include py-f90nml Not sure why.

CoryMartin-NOAA · 2024-03-20T19:48:22Z

@RussTreadon-NOAA the ctest pythonpaths are set at compile time. So if they are just set and then the test is re-ran, that won't help. Sorry if that wasn't clear!

RussTreadon-NOAA · 2024-03-20T19:50:05Z

Is a simple make -j 8 in the build directory sufficient or do I need to rm build and recompile everything?

CoryMartin-NOAA · 2024-03-20T19:53:15Z

I think it has to be cleaned, or at least the cmake cache needs cleaned (not 100% sure how to do this) because the cmake sets the PYTHONPATH.

RussTreadon-NOAA · 2024-03-20T19:55:50Z

make -j 8 didn't do the trick. I'll remove build and recompile from the top with load("py-f90nml/1.4.3") added to modulefiles/GDAS/hera.intel.lua

RussTreadon-NOAA · 2024-03-20T20:56:36Z

Rebuilt from the top. test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP still Failed but now the error message is

 'state variables': ['cicen', 'hicen', 'hsnon', 'socn', 'tocn', 'uocn', 'vocn', 'ssh']}, 'background error': {'covariance model': 'SABER', 'saber cen\
tral block': {'saber block name': 'EXPLICIT_DIFFUSION', 'geometry': {'mom6_input_nml': 'mom_input.nml', 'fields metadata': './fields_metadata.yaml'},\
 'calibration': {'normalization': {'method': 'randomization', 'iterations': 1000}, 'groups': [{'name': 'vt_ocean', 'vertical': {'as gaussian': True, \
'fixed value': 5.0}, 'write': {'filename': 'vt_ocean.nc'}}]}}}}
Traceback (most recent call last):
  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_prep.py", line 274, in <module>
    ufsda.yamltools.save_check(varconfig.as_dict(), target=var_yaml, app='var')
TypeError: 'dict' object is not callable
+ JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP[1]: postamble JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP 1710968012 1
+ preamble.sh[70]: set +x
End JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP at 20:53:51 with error code 1 (time elapsed: 00:00:19)
+ slurm_script[1]: postamble slurm_script 1710968011 1
+ preamble.sh[70]: set +x
End slurm_script at 20:53:51 with error code 1 (time elapsed: 00:00:20)

The above is taken from /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test/sorc/gdas.cd/build/gdas/test/soca/gw/testrun/JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP.out

I'll hit pause on this for the time being.

CoryMartin-NOAA · 2024-03-21T13:04:50Z

@RussTreadon-NOAA this is odd, this PR now no longer modifies soca files (I thought)

RussTreadon-NOAA · 2024-03-21T13:11:14Z

Agreed. The observed behavior is odd.

@guillaumevernieres or @AndrewEichmann-NOAA: have either of you recently run GDASApp g-w ctests using develop or one of your branches? Did all the g-w soca tests pass?

guillaumevernieres · 2024-03-21T15:25:17Z

Agreed. The observed behavior is odd.

@guillaumevernieres or @AndrewEichmann-NOAA: have either of you recently run GDASApp g-w ctests using develop or one of your branches? Did all the g-w soca tests pass?

It's fine in containers, we'll test on hpc.

RussTreadon-NOAA · 2024-03-21T15:54:29Z

Installed GDASApp develop at e029bd7 inside g-w develop at 03ba78a on Hera. The same test_gdasapp failures occur

        1395 - test_gdasapp_util_viirsaod2ioda (Subprocess aborted)
        1751 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1752 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1753 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1754 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN (Failed)
        1756 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1758 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
        1759 - test_gdasapp_soca_socahybridweights (Failed)
        1760 - test_gdasapp_soca_incr_handler (Failed)
        1761 - test_gdasapp_soca_ens_handler (Failed)
        1770 - test_gdasapp_atm_jjob_var_init (Failed)
        1771 - test_gdasapp_atm_jjob_var_run (Failed)
        1772 - test_gdasapp_atm_jjob_var_final (Failed)
        1773 - test_gdasapp_atm_jjob_ens_init (Failed)
        1774 - test_gdasapp_atm_jjob_ens_run (Failed)
        1775 - test_gdasapp_atm_jjob_ens_final (Failed)

Repeat the above test on Orion but install GDASApp feature/yaml_syntax inside g-w develop at 03ba78a. The following test_gdasapp tests fail

        1751 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1752 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1753 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1756 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1757 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1758 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
        1759 - test_gdasapp_soca_socahybridweights (Failed)
        1760 - test_gdasapp_soca_incr_handler (Failed)
        1761 - test_gdasapp_soca_ens_handler (Failed)
        1770 - test_gdasapp_atm_jjob_var_init (Failed)
        1771 - test_gdasapp_atm_jjob_var_run (Failed)
        1772 - test_gdasapp_atm_jjob_var_final (Failed)
        1773 - test_gdasapp_atm_jjob_ens_init (Failed)
        1774 - test_gdasapp_atm_jjob_ens_run (Failed)
        1775 - test_gdasapp_atm_jjob_ens_final (Failed)

Note that test_gdasapp_util_viirsaod2ioda does not fail on Orion as it does on Hera Rocky 8`. Not sure how to resolve this. We should have the same test reference on each machine, right?

It's interesting that test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN did not fail on Orion. Not sure why the job fails on Hera but passes on Orion.

GDASApp PR #981 fixes the test_gdasapp_atm_jjob failures.

Based on the above the test_gdasapp_soca_JGDAS_GLOBAL failures are not due to feature/yaml_syntax. These failures occur in develop.

RussTreadon-NOAA · 2024-03-22T20:28:28Z

Build feature/yaml_syntax on Orion in g-w develop. Run test_gdasapp. 54 out of 54 tests pass.

Test project /work2/noaa/da/rtreadon/git/global-workflow/pr928/sorc/gdas.cd/build
      Start 1389: test_gdasapp_util_coding_norms
 1/53 Test #1389: test_gdasapp_util_coding_norms ........................   Passed    2.86 sec
      Start 1390: test_gdasapp_util_ioda_example
 2/53 Test #1390: test_gdasapp_util_ioda_example ........................   Passed    7.46 sec
      Start 1391: test_gdasapp_util_prepdata
 3/53 Test #1391: test_gdasapp_util_prepdata ............................   Passed    4.46 sec
      Start 1392: test_gdasapp_util_rads2ioda
 4/53 Test #1392: test_gdasapp_util_rads2ioda ...........................   Passed    0.71 sec
      Start 1393: test_gdasapp_util_ghrsst2ioda
 5/53 Test #1393: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.19 sec
      Start 1394: test_gdasapp_util_smap2ioda
 6/53 Test #1394: test_gdasapp_util_smap2ioda ...........................   Passed    0.19 sec
      Start 1395: test_gdasapp_util_smos2ioda
 7/53 Test #1395: test_gdasapp_util_smos2ioda ...........................   Passed    0.21 sec
      Start 1396: test_gdasapp_util_viirsaod2ioda
 8/53 Test #1396: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.21 sec
      Start 1397: test_gdasapp_util_icecamsr2ioda
 9/53 Test #1397: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.18 sec
      Start 1733: test_gdasapp_check_python_norms
10/53 Test #1733: test_gdasapp_check_python_norms .......................   Passed    2.05 sec
      Start 1734: test_gdasapp_check_yaml_keys
11/53 Test #1734: test_gdasapp_check_yaml_keys ..........................   Passed    0.71 sec
      Start 1735: test_gdasapp_jedi_increment_to_fv3
12/53 Test #1735: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    5.19 sec
      Start 1736: test_gdasapp_setup_cycled_exp
13/53 Test #1736: test_gdasapp_setup_cycled_exp .........................   Passed    2.58 sec
      Start 1737: test_gdasapp_fv3jedi_jediinc2fv3
14/53 Test #1737: test_gdasapp_fv3jedi_jediinc2fv3 ......................   Passed   29.94 sec
      Start 1738: test_gdasapp_convert_bufr_temp_dbuoy
15/53 Test #1738: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    4.98 sec
      Start 1739: test_gdasapp_convert_bufr_salt_dbuoy
16/53 Test #1739: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    0.95 sec
      Start 1740: test_gdasapp_convert_bufr_temp_mbuoyb
17/53 Test #1740: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.66 sec
      Start 1741: test_gdasapp_convert_bufr_salt_mbuoyb
18/53 Test #1741: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.56 sec
      Start 1742: test_gdasapp_convert_bufr_tesacprof
19/53 Test #1742: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.30 sec
      Start 1743: test_gdasapp_convert_bufr_trkobprof
20/53 Test #1743: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.34 sec
      Start 1744: test_gdasapp_convert_bufr_sfcships
21/53 Test #1744: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.28 sec
      Start 1745: test_gdasapp_convert_bufr_sfcshipsu
22/53 Test #1745: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.30 sec
      Start 1746: test_gdasapp_soca_nsst_increment_to_mom6
23/53 Test #1746: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   56.51 sec
      Start 1747: test_gdasapp_soca_prep
24/53 Test #1747: test_gdasapp_soca_prep ................................   Passed    5.74 sec
      Start 1748: test_gdasapp_soca_run_clean
25/53 Test #1748: test_gdasapp_soca_run_clean ...........................   Passed    0.10 sec
      Start 1749: test_gdasapp_soca_setup_obsprep
26/53 Test #1749: test_gdasapp_soca_setup_obsprep .......................   Passed    6.04 sec
      Start 1750: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
27/53 Test #1750: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   43.01 sec
      Start 1751: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
28/53 Test #1751: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   74.31 sec
      Start 1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
29/53 Test #1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   74.79 sec
      Start 1753: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
30/53 Test #1753: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.69 sec
      Start 1754: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN
31/53 Test #1754: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN ....   Passed   42.78 sec
      Start 1755: test_gdasapp_soca_copy_scratch
32/53 Test #1755: test_gdasapp_soca_copy_scratch ........................   Passed    3.10 sec
      Start 1756: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
33/53 Test #1756: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.74 sec
      Start 1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
34/53 Test #1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   42.36 sec
      Start 1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
35/53 Test #1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....   Passed  298.84 sec
      Start 1759: test_gdasapp_soca_socahybridweights
36/53 Test #1759: test_gdasapp_soca_socahybridweights ...................   Passed   44.17 sec
      Start 1760: test_gdasapp_soca_incr_handler
37/53 Test #1760: test_gdasapp_soca_incr_handler ........................   Passed   10.57 sec
      Start 1761: test_gdasapp_soca_ens_handler
38/53 Test #1761: test_gdasapp_soca_ens_handler .........................   Passed   10.19 sec
      Start 1762: test_gdasapp_snow_create_ens
39/53 Test #1762: test_gdasapp_snow_create_ens ..........................   Passed   11.24 sec
      Start 1763: test_gdasapp_snow_imsproc
40/53 Test #1763: test_gdasapp_snow_imsproc .............................   Passed   12.93 sec
      Start 1764: test_gdasapp_snow_apply_jediincr
41/53 Test #1764: test_gdasapp_snow_apply_jediincr ......................   Passed    5.16 sec
      Start 1765: test_gdasapp_snow_letkfoi_snowda
42/53 Test #1765: test_gdasapp_snow_letkfoi_snowda ......................   Passed   27.98 sec
      Start 1766: test_gdasapp_convert_bufr_adpsfc_snow
43/53 Test #1766: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed    6.97 sec
      Start 1767: test_gdasapp_convert_bufr_adpsfc
44/53 Test #1767: test_gdasapp_convert_bufr_adpsfc ......................   Passed    5.18 sec
      Start 1768: test_gdasapp_convert_gsi_satbias
45/53 Test #1768: test_gdasapp_convert_gsi_satbias ......................   Passed    2.07 sec
      Start 1769: test_gdasapp_setup_atm_cycled_exp
46/53 Test #1769: test_gdasapp_setup_atm_cycled_exp .....................   Passed    2.04 sec
      Start 1770: test_gdasapp_atm_jjob_var_init
47/53 Test #1770: test_gdasapp_atm_jjob_var_init ........................   Passed   47.61 sec
      Start 1771: test_gdasapp_atm_jjob_var_run
48/53 Test #1771: test_gdasapp_atm_jjob_var_run .........................   Passed  106.19 sec
      Start 1772: test_gdasapp_atm_jjob_var_final
49/53 Test #1772: test_gdasapp_atm_jjob_var_final .......................   Passed   42.19 sec
      Start 1773: test_gdasapp_atm_jjob_ens_init
50/53 Test #1773: test_gdasapp_atm_jjob_ens_init ........................   Passed   45.00 sec
      Start 1774: test_gdasapp_atm_jjob_ens_run
51/53 Test #1774: test_gdasapp_atm_jjob_ens_run .........................   Passed  362.27 sec
      Start 1775: test_gdasapp_atm_jjob_ens_final
52/53 Test #1775: test_gdasapp_atm_jjob_ens_final .......................   Passed  138.64 sec
      Start 1776: test_gdasapp_aero_gen_3dvar_yaml
53/53 Test #1776: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    4.56 sec

100% tests passed, 0 tests failed out of 53

Label Time Summary:
gdas-utils    =  16.47 sec*proc (9 tests)
script        =  16.47 sec*proc (9 tests)

Total Test time (real) = 1688.94 sec

Will trigger orion-GW-RT. hera-GW-RT still fails while building some g-w components.

One lingering question is whether or not to include the new file ush/modify_yaml_syntax.py in this PR. Adding it doesn't hurt but we will likely remove it at a later date.

emcbot · 2024-03-22T22:00:29Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Mar 22 15:36:30 CDT 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Mar 22 16:29:43 CDT 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Mar 22 17:00:29 CDT 2024
Tests: 100% tests passed, 0 tests failed out of 53

RussTreadon-NOAA

g-w ctests pass on Orion. Whether or not we merge ush/modify_yaml_syntax.py into develop is an open question. We can merge it in now if we think others will benefit from it as they update yamls. It can be removed at a later date. Alternatively, we remove it now.

Approve.

RussTreadon-NOAA · 2024-03-25T11:03:40Z

@guillaumevernieres , any thoughts about including or removing the utility script, ush/modify_yaml_syntax.py, included in this PR?

This script replaces the old template format with the new template format. Several parm/soca yamls still use $(foo). Would you or other marine DA developers like to use ush/modify_yaml_syntax.py to update soca yamls?

guillaumevernieres · 2024-03-25T12:27:19Z

@guillaumevernieres , any thoughts about including or removing the utility script, ush/modify_yaml_syntax.py, included in this PR?

This script replaces the old template format with the new template format. Several parm/soca yamls still use $(foo). Would you or other marine DA developers like to use ush/modify_yaml_syntax.py to update soca yamls?

@RussTreadon-NOAA , we can keep it until we're done modifying our templates. Thanks for the heads up again.

RussTreadon-NOAA · 2024-03-25T12:30:54Z

OK. I'll merge this PR into develop. It's been approved and passes GW-RT on Orion.

Change $(foo) to {{ foo }}

dc990c9

CoryMartin-NOAA requested review from RussTreadon-NOAA, aerorahul, guillaumevernieres, jiaruidong2017, AndrewEichmann-NOAA and andytangborn February 16, 2024 17:56

CoryMartin-NOAA self-assigned this Feb 16, 2024

CoryMartin-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Feb 16, 2024

Norm

0e3c98c

CoryMartin-NOAA added 6 commits February 16, 2024 13:19

Remove ewok to gdas yaml test, modify aero 3dvar yaml test

42d2ed3

Merge branch 'develop' into feature/yaml_syntax

0795289

Try this

4b0cb8b

Merge branch 'feature/yaml_syntax' of https://github.com/NOAA-EMC/GDA…

33a2cfa

…SApp into feature/yaml_syntax

Update genyaml_3dvar.sh

71ecb0f

Remove useless test that is a waste of time

1406718

andytangborn previously approved these changes Feb 16, 2024

View reviewed changes

RussTreadon-NOAA previously approved these changes Feb 16, 2024

View reviewed changes

Flat out remove the ewok conversion test

ac5c801

CoryMartin-NOAA dismissed stale reviews from RussTreadon-NOAA and andytangborn via ac5c801 February 16, 2024 19:04

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 20, 2024

Merge branch 'develop' into feature/yaml_syntax

dcc51bb

RussTreadon-NOAA added the orion-GW-RT Queue for automated testing with global-workflow on Orion label Mar 22, 2024

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 22, 2024

emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 22, 2024

RussTreadon-NOAA self-requested a review March 23, 2024 00:00

RussTreadon-NOAA approved these changes Mar 23, 2024

View reviewed changes

RussTreadon-NOAA merged commit c14ff22 into develop Mar 25, 2024
11 checks passed

CoryMartin-NOAA deleted the feature/yaml_syntax branch March 28, 2024 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Change $(foo) to {{ foo }} in YAMLs #928

Change $(foo) to {{ foo }} in YAMLs #928

CoryMartin-NOAA commented Feb 16, 2024

guillaumevernieres commented Feb 16, 2024

CoryMartin-NOAA commented Feb 16, 2024

andytangborn left a comment

RussTreadon-NOAA left a comment

CoryMartin-NOAA commented Feb 16, 2024

emcbot commented Feb 16, 2024

RussTreadon-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 21, 2024

RussTreadon-NOAA commented Mar 21, 2024

guillaumevernieres commented Mar 21, 2024

RussTreadon-NOAA commented Mar 21, 2024

RussTreadon-NOAA commented Mar 22, 2024

emcbot commented Mar 22, 2024

RussTreadon-NOAA left a comment

RussTreadon-NOAA commented Mar 25, 2024

guillaumevernieres commented Mar 25, 2024

RussTreadon-NOAA commented Mar 25, 2024

Change $(foo) to {{ foo }} in YAMLs #928

Change $(foo) to {{ foo }} in YAMLs #928

Conversation

CoryMartin-NOAA commented Feb 16, 2024

guillaumevernieres commented Feb 16, 2024

CoryMartin-NOAA commented Feb 16, 2024

andytangborn left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

CoryMartin-NOAA commented Feb 16, 2024

emcbot commented Feb 16, 2024

RussTreadon-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

RussTreadon-NOAA commented Mar 20, 2024

CoryMartin-NOAA commented Mar 21, 2024

RussTreadon-NOAA commented Mar 21, 2024

guillaumevernieres commented Mar 21, 2024

RussTreadon-NOAA commented Mar 21, 2024

RussTreadon-NOAA commented Mar 22, 2024

emcbot commented Mar 22, 2024

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

RussTreadon-NOAA commented Mar 25, 2024

guillaumevernieres commented Mar 25, 2024

RussTreadon-NOAA commented Mar 25, 2024