Invoke err_chk for fatal errors in the wave J-Jobs and ex-scripts #3654

DavidHuber-NOAA · 2025-05-06T17:34:25Z

Description

This invokes err_chk in all wave processing J-Jobs and ex-scripts. A number of shellcheck issues were addressed along the way.

Resolves #3512

Type of change

Bug fix (fixes something broken)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? NO

How has this been tested?

Extended CI tests on WCOSS2

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
This change is covered by an existing CI test or a new one has been added

jobs/JGLOBAL_WAVE_INIT

AminIlia-NOAA

Generally looks fine but may have some conflict with #3646. e.g. {RUN}wave should be changed to {RUN}.wave.

DavidHuber-NOAA · 2025-05-06T18:30:24Z

@AminIlia-NOAA Yes, you are correct. I'm going to leave this as a draft until #3646 is merged.

scripts/exgfs_wave_prdgen_bulls.sh

DavidHuber-NOAA · 2025-05-07T15:38:03Z

Offline testing was successful on WCOSS2. Marking this PR ready for review.

AminIlia-NOAA

Looks fine to me, thanks!

JessicaMeixner-NOAA

Thanks @DavidHuber-NOAA - I just had one quick question on one copy statement that I don't think should change. Otherwise things look good to me.

scripts/exgfs_wave_init.sh

DavidHuber-NOAA · 2025-05-07T15:55:07Z

Launching CI on C6.

.gitignore

scripts/exgfs_wave_prdgen_bulls.sh

emcbot · 2025-05-14T22:05:13Z

Experiment C96mx100_S2S FAILED on Hera in Build# 2 with error logs:

/scratch1/NCEPDEV/global/glopara/CI/3654/RUNTESTS/COMROOT/C96mx100_S2S_29f051e4/logs/1994050100/sfs_fcst_mem000_seg0.log
/scratch1/NCEPDEV/global/glopara/CI/3654/RUNTESTS/COMROOT/C96mx100_S2S_29f051e4/logs/1994050100/sfs_fcst_mem001_seg0.log
/scratch1/NCEPDEV/global/glopara/CI/3654/RUNTESTS/COMROOT/C96mx100_S2S_29f051e4/logs/1994050100/sfs_fcst_mem002_seg0.log

Follow link here to view the contents of the above file(s): (sfs_fcst_mem000_seg0.log) (sfs_fcst_mem001_seg0.log) (sfs_fcst_mem002_seg0.log)

emcbot · 2025-05-14T22:05:29Z

Experiment C96mx100_S2S FAILED on Hera in Build# 2 in
/scratch1/NCEPDEV/global/glopara/CI/3654/RUNTESTS/EXPDIR/C96mx100_S2S_29f051e4

DavidHuber-NOAA · 2025-05-15T11:17:08Z

The SFS forecasts failed with the MPI_Abort messages we have been seeing on other machines periodically. I'm going to clone my branch locally on Hera and see if I can repeat this error. If it does not repeat, then I will reopen #3609 to investigate.

DavidHuber-NOAA · 2025-05-15T12:16:04Z

Rerunning the SFS forecasts after building sfs_model.x on the head node was successful. I have reopened #3609 to investigate.

DavidHuber-NOAA · 2025-05-15T13:25:37Z

forecast_postdet.sh has a bug in it introduced by #3606. Line 765

global-workflow/ush/forecast_postdet.sh

Line 765 in e8fdc0f

    
           gfs|gefs|gcafs) # Copy CICE restarts at the end of the forecast segment to COM for RUN=gfs|gefs|sfs|gcafs

Should read

    gfs|gefs|sfs|gcafs) # Copy CICE restarts at the end of the forecast segment to COM for RUN=gfs|gefs|sfs|gcafs

…workflow into feature/err_wave

emcbot · 2025-05-15T18:08:59Z

CI Tests set up to run in /lfs/h2/emc/ptmp/emc.global/PR/PR_3654/RUNTESTS on WCOSS

KateFriedman-NOAA · 2025-05-16T13:33:30Z

CI on WCOSS2 completed successfully minus some gempak jobs (known issue #3630). Setting label to passed.

Fri May 16 13:28:06 UTC 2025
******** C48_ATM_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 15 2025 18:09:02    May 15 2025 19:15:38

******** C48mx500_3DVarAOWCDA_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 15 2025 18:09:04    May 15 2025 18:25:38
202103250000        Done    May 15 2025 18:09:04    May 15 2025 19:45:50

******** C48mx500_hybAOWCDA_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    May 15 2025 18:09:07    May 15 2025 18:20:39
202103250000        Done    May 15 2025 18:09:07    May 15 2025 19:30:30

******** C48_S2SW_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 15 2025 18:09:10    May 15 2025 19:20:36

******** C48_S2SWA_gefs_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    May 15 2025 18:09:13    May 15 2025 19:16:00

******** C96_atm3DVar_extended_3654 ********
202112210000    gfs_gempak_f123-f144                   196085937                DEAD                 -29         2        1824.0
202112210000    gfs_gempak_f147-f168                   196086936                DEAD                 -29         2        1851.0
202112210600    gfs_gempak_f123-f144                   196090894                DEAD                 -29         2        1852.0
202112210600    gfs_gempak_f147-f168                   196092134                DEAD                 -29         2        1851.0
202112211200    gfs_gempak_f123-f144                   196094972                DEAD                 -29         2        1852.0
202112211200    gfs_gempak_f147-f168                   196095636                DEAD                 -29         2        1855.0

******** C96C48_hybatmaerosnowDA_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 15 2025 18:09:19    May 15 2025 18:31:23
202112201800        Done    May 15 2025 18:09:19    May 15 2025 20:36:05
202112210000        Done    May 15 2025 18:09:19    May 15 2025 20:26:07

******** C96C48_hybatmDA_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    May 15 2025 18:09:22    May 15 2025 18:26:37
202112210000        Done    May 15 2025 18:09:22    May 15 2025 20:31:02
202112210600        Done    May 15 2025 18:09:22    May 15 2025 20:16:02

******** C96C48mx500_S2SW_cyc_gfs_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    May 15 2025 18:09:25    May 15 2025 18:26:44
202112201800        Done    May 15 2025 18:09:25    May 15 2025 20:36:13
202112210000        Done    May 15 2025 18:09:25    May 15 2025 21:35:58
202112211800        Done    May 15 2025 18:31:30    May 15 2025 21:35:59

******** C96C48_ufs_hybatmDA_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    May 15 2025 18:09:28    May 15 2025 18:26:53
202402240000        Done    May 15 2025 18:09:28    May 15 2025 21:02:00
202402240600        Done    May 15 2025 18:09:28    May 15 2025 20:51:41

******** C96mx100_S2S_3654 ********
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
199405010000        Done    May 15 2025 18:09:33    May 15 2025 19:41:30

KateFriedman-NOAA

Looks good, thanks @DavidHuber-NOAA !

DavidHuber-NOAA added 2 commits May 6, 2025 09:44

Add err_chk to wave jobs

f25827c

Merge remote-tracking branch 'origin/develop' into feature/err_wave

343c3e5

github-advanced-security bot found potential problems May 6, 2025

View reviewed changes

jobs/JGLOBAL_WAVE_INIT Fixed Show fixed Hide fixed

JessicaMeixner-NOAA requested a review from AminIlia-NOAA May 6, 2025 17:36

Shellcheck double-quotes

6723a29

AminIlia-NOAA suggested changes May 6, 2025

View reviewed changes

DavidHuber-NOAA added 2 commits May 6, 2025 18:54

Merge 3646

355dab8

Merge develop, resolve conflicts

6b412e6

github-advanced-security bot found potential problems May 7, 2025

View reviewed changes

scripts/exgfs_wave_prdgen_bulls.sh Fixed Show fixed Hide fixed

scripts/exgfs_wave_prdgen_bulls.sh Fixed Show fixed Hide fixed

scripts/exgfs_wave_prdgen_bulls.sh Fixed Show fixed Hide fixed

DavidHuber-NOAA added 2 commits May 7, 2025 11:31

Add linked control file to ignore list

f70dea2

Resolve last of the conflicts

da11461

DavidHuber-NOAA marked this pull request as ready for review May 7, 2025 15:37

DavidHuber-NOAA requested review from JessicaMeixner-NOAA and sbanihash as code owners May 7, 2025 15:37

DavidHuber-NOAA requested a review from AminIlia-NOAA May 7, 2025 15:38

AminIlia-NOAA previously approved these changes May 7, 2025

View reviewed changes

Change cpreq to cpfs when copying to COM

97af715

JessicaMeixner-NOAA reviewed May 7, 2025

View reviewed changes

scripts/exgfs_wave_init.sh Show resolved Hide resolved

DavidHuber-NOAA dismissed AminIlia-NOAA’s stale review via 97af715 May 7, 2025 15:52

DavidHuber-NOAA requested review from KateFriedman-NOAA and aerorahul May 7, 2025 15:54

DavidHuber-NOAA added the CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 label May 7, 2025

emcbot added CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 and removed CI-Gaeac6-Ready **CM use only** PR is ready for CI testing on Gaea C6 labels May 7, 2025

aerorahul reviewed May 7, 2025

View reviewed changes

.gitignore Outdated Show resolved Hide resolved

scripts/exgfs_wave_prdgen_bulls.sh Show resolved Hide resolved

emcbot added CI-Gaeac6-Running **Bot use only** CI testing on Gaea C6 for this PR is in-progress and removed CI-Gaeac6-Building **Bot use only** CI testing is cloning/building on Gaea C6 labels May 7, 2025

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels May 14, 2025

DavidHuber-NOAA added 2 commits May 15, 2025 13:27

Add sfs back in when copying restarts

e314992

Merge branch 'feature/err_wave' of github.com:davidhuber-noaa/global-…

98ccab5

…workflow into feature/err_wave

DavidHuber-NOAA dismissed stale reviews from aerorahul and JessicaMeixner-NOAA via 98ccab5 May 15, 2025 13:30

DavidHuber-NOAA requested a review from WalterKolczynski-NOAA as a code owner May 15, 2025 13:30

Merge branch 'develop' into feature/err_wave

9afb9b2

emcbot added CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Ready PR is ready for CI testing on WCOSS2. CI-Wcoss2-Building CI testing is cloning/building on WCOSS2 labels May 15, 2025

KateFriedman-NOAA added CI-Wcoss2-Passed CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running CI testing on WCOSS for this PR is in-progress labels May 16, 2025

Merge branch 'develop' into feature/err_wave

0b32194

KateFriedman-NOAA approved these changes May 16, 2025

View reviewed changes

DavidHuber-NOAA merged commit 5401603 into NOAA-EMC:develop May 16, 2025
5 checks passed

DavidHuber-NOAA deleted the feature/err_wave branch May 16, 2025 14:15

DavidHuber-NOAA mentioned this pull request May 20, 2025

S2SW ensemble forecasts are aborting on Hercules and Hera #3673

Closed

Invoke err_chk for fatal errors in the wave J-Jobs and ex-scripts #3654

Invoke err_chk for fatal errors in the wave J-Jobs and ex-scripts #3654

Uh oh!

Conversation

DavidHuber-NOAA commented May 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

Uh oh!

AminIlia-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

DavidHuber-NOAA commented May 6, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

DavidHuber-NOAA commented May 7, 2025

Uh oh!

AminIlia-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

DavidHuber-NOAA commented May 7, 2025

Uh oh!

Uh oh!

Uh oh!

emcbot commented May 14, 2025

Uh oh!

emcbot commented May 14, 2025

Uh oh!

DavidHuber-NOAA commented May 15, 2025

Uh oh!

DavidHuber-NOAA commented May 15, 2025

Uh oh!

DavidHuber-NOAA commented May 15, 2025

Uh oh!

emcbot commented May 15, 2025

Uh oh!

KateFriedman-NOAA commented May 16, 2025

Uh oh!

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DavidHuber-NOAA commented May 6, 2025 •

edited

Loading