Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix to marine DA prep #989

Merged
merged 1 commit into from
Mar 22, 2024
Merged

Fix to marine DA prep #989

merged 1 commit into from
Mar 22, 2024

Conversation

guillaumevernieres
Copy link
Contributor

@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 22, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 22, 2024
RussTreadon-NOAA
RussTreadon-NOAA approved these changes Mar 22, 2024
View reviewed changes
Copy link
Contributor

@RussTreadon-NOAA RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested the change to scripts/exgdas_global_marine_analysis_prep.py in a working copy of RussTreadon-NOAA:feature/test_gdasapp_atm_jjob_var installed on Orion.

Rerun test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP

(gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr928/sorc/gdas.cd/build$ ctest -R test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr928/sorc/gdas.cd/build
    Start 1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
1/1 Test #1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ...   Passed   74.46 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =  75.12 sec

Given the above success, rerun all test_gdasapp. 54 out of 54 tests pass on Orion.

(gdasapp) Orion-login-3:/work2/noaa/da/rtreadon/git/global-workflow/pr928/sorc/gdas.cd/build$ ctest -R test_gdasapp
Test project /work2/noaa/da/rtreadon/git/global-workflow/pr928/sorc/gdas.cd/build
      Start 1389: test_gdasapp_util_coding_norms
 1/54 Test #1389: test_gdasapp_util_coding_norms ........................   Passed    2.33 sec
      Start 1390: test_gdasapp_util_ioda_example
 2/54 Test #1390: test_gdasapp_util_ioda_example ........................   Passed    8.09 sec
      Start 1391: test_gdasapp_util_prepdata
 3/54 Test #1391: test_gdasapp_util_prepdata ............................   Passed    4.09 sec
      Start 1392: test_gdasapp_util_rads2ioda
 4/54 Test #1392: test_gdasapp_util_rads2ioda ...........................   Passed    0.47 sec
      Start 1393: test_gdasapp_util_ghrsst2ioda
 5/54 Test #1393: test_gdasapp_util_ghrsst2ioda .........................   Passed    0.25 sec
      Start 1394: test_gdasapp_util_smap2ioda
 6/54 Test #1394: test_gdasapp_util_smap2ioda ...........................   Passed    0.20 sec
      Start 1395: test_gdasapp_util_smos2ioda
 7/54 Test #1395: test_gdasapp_util_smos2ioda ...........................   Passed    0.20 sec
      Start 1396: test_gdasapp_util_viirsaod2ioda
 8/54 Test #1396: test_gdasapp_util_viirsaod2ioda .......................   Passed    0.18 sec
      Start 1397: test_gdasapp_util_icecamsr2ioda
 9/54 Test #1397: test_gdasapp_util_icecamsr2ioda .......................   Passed    0.19 sec
      Start 1733: test_gdasapp_check_python_norms
10/54 Test #1733: test_gdasapp_check_python_norms .......................   Passed    2.12 sec
      Start 1734: test_gdasapp_check_yaml_keys
11/54 Test #1734: test_gdasapp_check_yaml_keys ..........................   Passed    0.35 sec
      Start 1735: test_gdasapp_jedi_increment_to_fv3
12/54 Test #1735: test_gdasapp_jedi_increment_to_fv3 ....................   Passed    8.18 sec
      Start 1736: test_gdasapp_convert_ewok_yaml
13/54 Test #1736: test_gdasapp_convert_ewok_yaml ........................   Passed    0.24 sec
      Start 1737: test_gdasapp_setup_cycled_exp
14/54 Test #1737: test_gdasapp_setup_cycled_exp .........................   Passed    1.21 sec
      Start 1738: test_gdasapp_fv3jedi_jediinc2fv3
15/54 Test #1738: test_gdasapp_fv3jedi_jediinc2fv3 ......................   Passed   34.32 sec
      Start 1739: test_gdasapp_convert_bufr_temp_dbuoy
16/54 Test #1739: test_gdasapp_convert_bufr_temp_dbuoy ..................   Passed    4.69 sec
      Start 1740: test_gdasapp_convert_bufr_salt_dbuoy
17/54 Test #1740: test_gdasapp_convert_bufr_salt_dbuoy ..................   Passed    1.91 sec
      Start 1741: test_gdasapp_convert_bufr_temp_mbuoyb
18/54 Test #1741: test_gdasapp_convert_bufr_temp_mbuoyb .................   Passed    0.30 sec
      Start 1742: test_gdasapp_convert_bufr_salt_mbuoyb
19/54 Test #1742: test_gdasapp_convert_bufr_salt_mbuoyb .................   Passed    0.29 sec
      Start 1743: test_gdasapp_convert_bufr_tesacprof
20/54 Test #1743: test_gdasapp_convert_bufr_tesacprof ...................   Passed    0.29 sec
      Start 1744: test_gdasapp_convert_bufr_trkobprof
21/54 Test #1744: test_gdasapp_convert_bufr_trkobprof ...................   Passed    0.29 sec
      Start 1745: test_gdasapp_convert_bufr_sfcships
22/54 Test #1745: test_gdasapp_convert_bufr_sfcships ....................   Passed    0.29 sec
      Start 1746: test_gdasapp_convert_bufr_sfcshipsu
23/54 Test #1746: test_gdasapp_convert_bufr_sfcshipsu ...................   Passed    0.28 sec
      Start 1747: test_gdasapp_soca_nsst_increment_to_mom6
24/54 Test #1747: test_gdasapp_soca_nsst_increment_to_mom6 ..............   Passed   27.65 sec
      Start 1748: test_gdasapp_soca_prep
25/54 Test #1748: test_gdasapp_soca_prep ................................   Passed    5.55 sec
      Start 1749: test_gdasapp_soca_run_clean
26/54 Test #1749: test_gdasapp_soca_run_clean ...........................   Passed    0.33 sec
      Start 1750: test_gdasapp_soca_setup_obsprep
27/54 Test #1750: test_gdasapp_soca_setup_obsprep .......................   Passed    5.49 sec
      Start 1751: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS
28/54 Test #1751: test_gdasapp_soca_JGLOBAL_PREP_OCEAN_OBS ..............   Passed   43.70 sec
      Start 1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
29/54 Test #1752: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ....   Passed   42.56 sec
      Start 1753: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT
30/54 Test #1753: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT ....   Passed   74.34 sec
      Start 1754: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
31/54 Test #1754: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN .....   Passed   42.62 sec
      Start 1755: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN
32/54 Test #1755: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_ECEN ....   Passed   42.80 sec
      Start 1756: test_gdasapp_soca_copy_scratch
33/54 Test #1756: test_gdasapp_soca_copy_scratch ........................   Passed    3.36 sec
      Start 1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT
34/54 Test #1757: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT ...   Passed   42.35 sec
      Start 1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
35/54 Test #1758: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ....   Passed   42.86 sec
      Start 1759: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
36/54 Test #1759: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ....   Passed  269.20 sec
      Start 1760: test_gdasapp_soca_socahybridweights
37/54 Test #1760: test_gdasapp_soca_socahybridweights ...................   Passed   45.24 sec
      Start 1761: test_gdasapp_soca_incr_handler
38/54 Test #1761: test_gdasapp_soca_incr_handler ........................   Passed   10.46 sec
      Start 1762: test_gdasapp_soca_ens_handler
39/54 Test #1762: test_gdasapp_soca_ens_handler .........................   Passed   11.46 sec
      Start 1763: test_gdasapp_snow_create_ens
40/54 Test #1763: test_gdasapp_snow_create_ens ..........................   Passed   18.73 sec
      Start 1764: test_gdasapp_snow_imsproc
41/54 Test #1764: test_gdasapp_snow_imsproc .............................   Passed   12.46 sec
      Start 1765: test_gdasapp_snow_apply_jediincr
42/54 Test #1765: test_gdasapp_snow_apply_jediincr ......................   Passed    7.02 sec
      Start 1766: test_gdasapp_snow_letkfoi_snowda
43/54 Test #1766: test_gdasapp_snow_letkfoi_snowda ......................   Passed   13.55 sec
      Start 1767: test_gdasapp_convert_bufr_adpsfc_snow
44/54 Test #1767: test_gdasapp_convert_bufr_adpsfc_snow .................   Passed   11.28 sec
      Start 1768: test_gdasapp_convert_bufr_adpsfc
45/54 Test #1768: test_gdasapp_convert_bufr_adpsfc ......................   Passed    5.92 sec
      Start 1769: test_gdasapp_convert_gsi_satbias
46/54 Test #1769: test_gdasapp_convert_gsi_satbias ......................   Passed    2.43 sec
      Start 1770: test_gdasapp_setup_atm_cycled_exp
47/54 Test #1770: test_gdasapp_setup_atm_cycled_exp .....................   Passed    1.93 sec
      Start 1771: test_gdasapp_atm_jjob_var_init
48/54 Test #1771: test_gdasapp_atm_jjob_var_init ........................   Passed   45.68 sec
      Start 1772: test_gdasapp_atm_jjob_var_run
49/54 Test #1772: test_gdasapp_atm_jjob_var_run .........................   Passed  106.66 sec
      Start 1773: test_gdasapp_atm_jjob_var_final
50/54 Test #1773: test_gdasapp_atm_jjob_var_final .......................   Passed   43.02 sec
      Start 1774: test_gdasapp_atm_jjob_ens_init
51/54 Test #1774: test_gdasapp_atm_jjob_ens_init ........................   Passed   45.73 sec
      Start 1775: test_gdasapp_atm_jjob_ens_run
52/54 Test #1775: test_gdasapp_atm_jjob_ens_run .........................   Passed  335.73 sec
      Start 1776: test_gdasapp_atm_jjob_ens_final
53/54 Test #1776: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.35 sec
      Start 1777: test_gdasapp_aero_gen_3dvar_yaml
54/54 Test #1777: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    3.89 sec

100% tests passed, 0 tests failed out of 54

Label Time Summary:
gdas-utils    =  16.00 sec*proc (9 tests)
script        =  16.00 sec*proc (9 tests)

Total Test time (real) = 1485.89 sec

Approve!

PS: How did you know or figure out that removing .as_dict() was the fix?

@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Mar 22, 2024
Closed
@emcbot
Copy link

emcbot commented Mar 22, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Mar 22 14:13:47 UTC 2024 on hfe07
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Fri Mar 22 15:01:44 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/989/global-workflow/sorc/log.build

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 22, 2024
@RussTreadon-NOAA
Copy link
Contributor

The hera-GW-RT-Failed result is expected. Not all g-w components build on Hera Rocky 8.

build_gfs_utils.sh completed successfully!
build_ufs_utils.sh failed with status 2!
build_ww3prepost.sh failed with status 2!
build_gsi_monitor.sh completed successfully!
build_ufs.sh failed with status 2!
build_gsi_utils.sh completed successfully!
build_upp.sh failed with status 2!
build_gdas.sh completed successfully!
BUILD ERROR: One or more components failed to build

GDASApp builds but other components fail and so the overall build fails. Hence, the failed hera-GW-RT results.

@guillaumevernieres
Copy link
Contributor Author

guillaumevernieres commented Mar 22, 2024 •
edited
Loading

Thanks @RussTreadon-NOAA !
I think the hint for the bug was in the log.

@emcbot
Copy link

emcbot commented Mar 22, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Mar 22 09:04:59 CDT 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Mar 22 10:02:20 CDT 2024
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Mar 22 10:25:06 CDT 2024
Tests: 89% tests passed, 6 tests failed out of 54
	1771 - test_gdasapp_atm_jjob_var_init (Failed)
	1772 - test_gdasapp_atm_jjob_var_run (Failed)
	1773 - test_gdasapp_atm_jjob_var_final (Failed)
	1774 - test_gdasapp_atm_jjob_ens_init (Failed)
	1775 - test_gdasapp_atm_jjob_ens_run (Failed)
	1776 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/989/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 22, 2024
@RussTreadon-NOAA
Copy link
Contributor

The orion-GW-RT failures for test_gdasapp_atm_jjob are expected. GDASApp PR #981 restores these tests to Passed.

@RussTreadon-NOAA
Copy link
Contributor

Given the above test results, I'll merge this PR into develop before noon today (3/22).

After this, I'll update PR #981 with the updated develop. Would you, @guillaumevernieres , be able to take a quick peek at PR #981? It would be great if we can also get PR #981 into develop today. Then all our g-w tests pass again ... at least on Orion.

@RussTreadon-NOAA RussTreadon-NOAA merged commit a575b46 into develop Mar 22, 2024
17 checks passed
@RussTreadon-NOAA RussTreadon-NOAA mentioned this pull request Mar 22, 2024
Merged
@CoryMartin-NOAA CoryMartin-NOAA deleted the bugfix/prep branch March 28, 2024 15:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@RussTreadon-NOAA RussTreadon-NOAA RussTreadon-NOAA approved these changes

Assignees
No one assigned
Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Marine DA is broken
3 participants
@guillaumevernieres @emcbot @RussTreadon-NOAA