Update gdas.cd and gsi_utils hashes #2641

RussTreadon-NOAA · 2024-05-30T13:24:51Z

Description

This PR updates the sorc/gdas.cd and sorc/gsi_utils hashes. The updated hashes bring in bug fixes, new UFS DA functionality, and a Gaea build for gsi_utils.

Resolves #2640

Type of change

Maintenance: update UFS DA hashes

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Clone, build, and cycle on WCOSS2

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
New and existing tests pass with my changes

RussTreadon-NOAA · 2024-05-30T13:35:28Z

NOTE

This PR will remain in draft mode until the following PRs are merged into their respective develop

OOPS PR #2645
jcb-gdas PR #8
GDASApp PR #1123

These PRs must be acted upon in sequence

OOPS and jcb-gdas PRs merged into their respective develop
update OOPS and jcb-gdas hashes in GDASApp PR #1123
GDASApp PR #1123 merged into develop
update sorc/gdas.cd hash in RussTreadon-NOAA:feature/update_gdasapp to point at updated GDASApp develop

The PR will be marked Ready for review once the above tasks are completed.

RussTreadon-NOAA · 2024-05-30T14:53:38Z

Orion test
Install RussTreadon-NOAA:feature/update_gdasapp at 215cd32. Enable and run C96C48_ufs_hybatmDA. All jobs successfully run to completion

Orion-login-4:/work2/noaa/stmp/rtreadon/EXPDIR/prsub$ rocotostat -d prsub.db -w prsub.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    May 30 2024 12:13:17    May 30 2024 12:30:28
202402240000        Done    May 30 2024 12:13:17    May 30 2024 14:30:27

RussTreadon-NOAA · 2024-05-31T15:47:34Z

Hera test
Install RussTreadon-NOAA:feature/update_gdasapp at 215cd32. Enable and run C96C48_ufs_hybatmDA. All jobs except one successfully run to completion. The single failure was

202402240000          enkfgdasearc00                    61083915                DEAD                   1         2          21.0

enkfgdasearc00.log contained the error message

  File "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/update_gdasapp/ush/python/pygfs/task/archive.py", line 199, in _create_fileset
    raise FileNotFoundError(f"FATAL ERROR: Required file, directory, or glob {item} not found!")
FileNotFoundError: FATAL ERROR: Required file, directory, or glob logs/2024022400/enkfgdasatmensanlrun.log not found!

The atmensanlrun log file is no longer generated. With the merger of PR #2592 into g-w develop, job atmensanlrun was replaced by atmensanlletkf. A new job, atmensanlfv3inc was added in the same PR. See workflow/rocoto/gfs_tasks.py from PR #2592.

We need to update parm/archive/enkf.yaml.j2

-        {% set steps = ["atmensanlinit", "atmensanlrun", "atmensanlfinal"] %}
+        {% set steps = ["atmensanlinit", "atmensanlletkf", "atmensanlfv3inc", "atmensanlfinal"] %}

After this change was made in the Hera working copy of feature/update_gdasapp, job enkfgdasearc00 was rerun and successfully completed. All jobs executed by C96C48_ufs_hybatmDA successfully ran to completion.

/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/prtest$ rocotostat -d prtest.db -w prtest.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    May 31 2024 13:19:32    May 31 2024 13:40:15
202402240000        Done    May 31 2024 13:19:32    May 31 2024 15:43:10

…dasapp

RussTreadon-NOAA · 2024-06-06T11:26:59Z

Orion and Hera tests

Install RussTreadon-NOAA:feature/update_gdasapp at 1dc1a11 on Orion and Hera. Two sets of tests run on each machine

Run test_gdasapp ctests. All 47 ctests pass on both machines.

Orion

100% tests passed, 0 tests failed out of 47

Label Time Summary:
gdas-utils    =  12.48 sec*proc (9 tests)
script        =  12.48 sec*proc (9 tests)

Total Test time (real) = 3679.77 sec

Hera

100% tests passed, 0 tests failed out of 47

Label Time Summary:
gdas-utils    =   9.32 sec*proc (9 tests)
script        =   9.32 sec*proc (9 tests)

Total Test time (real) = 1745.66 sec

Enable and run g-w CI for C96C48_ufs_hybatmDA. All jobs successfully run to completion on both machines

Orion

Orion-login-4:/work2/noaa/stmp/rtreadon/EXPDIR/pr2641$ rocotostat -d pr2641.db -w pr2641.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Jun 06 2024 01:33:40    Jun 06 2024 01:50:23
202402240000        Done    Jun 06 2024 01:33:40    Jun 06 2024 04:25:19

Hera

Hera(hfe07):/scratch1/NCEPDEV/stmp2/Russ.Treadon/EXPDIR/pr2641$ rocotostat -d pr2641.db -w pr2641.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Jun 06 2024 01:39:38    Jun 06 2024 04:25:12
202402240000        Done    Jun 06 2024 01:39:38    Jun 06 2024 07:30:14

RussTreadon-NOAA · 2024-06-06T18:50:56Z

Hercules and Cactus tests

Install RussTreadon-NOAA:feature/update_gdasapp at 1ca534c on Hercules and Cactus. Enable and run g-w CI for C96C48_ufs_hybatmDA. All jobs successfully run to completion on both machines

Hercules

(gdasapp) hercules-login-3:/work2/noaa/stmp/rtreadon/EXPDIR/pr2641_hercules$ rocotostat -d pr2641_hercules.db -w pr2641_hercules.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED
202402231800        Done    Jun 06 2024 16:15:10    Jun 06 2024 16:30:04
202402240000        Done    Jun 06 2024 16:15:10    Jun 06 2024 18:20:03

Cactus

russ.treadon@clogin07:/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/pr2641> rocotostat -d pr2641.db -w pr2641.xml -c all -s
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Jun 06 2024 16:12:12    Jun 06 2024 16:25:15
202402240000        Done    Jun 06 2024 16:12:12    Jun 06 2024 18:20:12

emcbot · 2024-06-06T22:29:26Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2641/RUNTESTS/COMROOT/C48mx500_3DVarAOWCDA_1ca534c7/logs/2021032418/gdasocnanalprep.log

Follow link here to view the contents of the above file(s): (link)

emcbot · 2024-06-06T22:29:32Z

Experiment C48mx500_3DVarAOWCDA FAILED on Hera in
/scratch1/NCEPDEV/global/CI/2641/RUNTESTS/C48mx500_3DVarAOWCDA_1ca534c7

RussTreadon-NOAA · 2024-06-07T01:45:02Z

The log file found at (link) has an "unable to copy" error on line 1016 of the log file.

ls /scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/72x35x25/soca/rossrad.nc on Hera confirms that the file does not exist. Below is ls -l of the directory in question

Hera(hfe10):~$ ls -l /scratch2/NCEPDEV/ocean/Guillaume.Vernieres/data/static/72x35x25/soca/
total 92
drwxr-sr-x 2 Guillaume.Vernieres ocean  4096 Jul 23  2022 INPUT
-rw-r--r-x 1 Guillaume.Vernieres ocean 34295 Nov  2  2022 MOM_input
-rw-r--r-x 1 Guillaume.Vernieres ocean 39619 Jul 23  2022 MOM_input_bkp
lrwxrwxrwx 1 Guillaume.Vernieres ocean    50 Apr  4 20:21 RECCAP2_region_masks_all_v20221025.nc -> ../../common/RECCAP2_region_masks_all_v20221025.nc
drwxr-sr-x 5 Guillaume.Vernieres ocean  4096 Jun  5  2023 bkgerr
-rw-r--r-x 1 Guillaume.Vernieres ocean    20 Jul 21  2022 diag_table
-rw-r--r-x 1 Guillaume.Vernieres ocean   199 Jul 21  2022 field_table
lrwxrwxrwx 1 Guillaume.Vernieres ocean    33 Aug  8  2023 fields_metadata.yaml -> ../../common/fields_metadata.yaml
lrwxrwxrwx 1 Guillaume.Vernieres ocean    31 Aug  8  2023 godas_sst_bgerr.nc -> ../../common/godas_sst_bgerr.nc
lrwxrwxrwx 1 Guillaume.Vernieres ocean    43 Mar 21  2023 obsop_name_map.yaml -> ../../1440x1080x75/soca/obsop_name_map.yaml
lrwxrwxrwx 1 Guillaume.Vernieres ocean    24 Aug  8  2023 rossrad.dat -> ../../common/rossrad.dat

File rossrad.dat is present. File rossrad.nc is absent.

emcbot · 2024-06-13T01:28:44Z

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2641

emcbot · 2024-06-13T01:40:18Z

CI Update on Wcoss2 at 06/13/24 01:40:18 AM
============================================
Cloning and Building global-workflow PR: 2641
with PID: 96375 on host: clogin05

emcbot · 2024-06-13T02:33:31Z

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Thu Jun 13 01:51:19 UTC 2024 on clogin05
---------------------------------------------------
Build: Completed at 06/13/24 02:33:18 AM
Case setup: Completed for experiment C48_ATM_872f2f5f
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_872f2f5f
Case setup: Skipped for experiment C48_S2SWA_gefs_872f2f5f
Case setup: Completed for experiment C48_S2SW_872f2f5f
Case setup: Completed for experiment C96_atm3DVar_extended_872f2f5f
Case setup: Skipped for experiment C96_atm3DVar_872f2f5f
Case setup: Skipped for experiment C96_atmaerosnowDA_872f2f5f
Case setup: Completed for experiment C96C48_hybatmDA_872f2f5f
Case setup: Completed for experiment C96C48_ufs_hybatmDA_872f2f5f

emcbot · 2024-06-13T03:42:43Z

Experiment C48_ATM_872f2f5f SUCCESS on Wcoss2 at 06/13/24 03:42:42 AM

emcbot · 2024-06-13T04:00:19Z

Experiment C48_S2SW_872f2f5f SUCCESS on Wcoss2 at 06/13/24 04:00:18 AM

emcbot · 2024-06-13T04:48:20Z

Experiment C96C48_hybatmDA_872f2f5f SUCCESS on Wcoss2 at 06/13/24 04:48:20 AM

emcbot · 2024-06-13T04:57:16Z

Experiment C96C48_ufs_hybatmDA_872f2f5f SUCCESS on Wcoss2 at 06/13/24 04:57:15 AM

emcbot · 2024-06-13T10:36:29Z

Experiment C96_atm3DVar_extended_872f2f5f SUCCESS on Wcoss2 at 06/13/24 10:36:28 AM

emcbot · 2024-06-13T10:39:09Z

All CI Test Cases Passed on Wcoss2:

Experiment C48_S2SW_872f2f5f *** SUCCESS *** at 06/13/24 04:00:18 AM
Experiment C96C48_hybatmDA_872f2f5f *** SUCCESS *** at 06/13/24 04:48:20 AM
Experiment C96C48_ufs_hybatmDA_872f2f5f *** SUCCESS *** at 06/13/24 04:57:15 AM
Experiment C96_atm3DVar_extended_872f2f5f *** SUCCESS *** at 06/13/24 10:36:28 AM

RussTreadon-NOAA · 2024-06-13T13:26:16Z

@TerrenceMcGuinness-NOAA and @WalterKolczynski-NOAA , while I can log into Hercules, I can not cd /work2/noaa. The cd command hangs. I see the CI-Hercules-Ready label on this PR. Is g-w CI currently running on Hercules?

TerrenceMcGuinness-NOAA · 2024-06-13T15:07:55Z

@RussTreadon-NOAA Yes I noticed this too on Hercules this morning when looking into the Running CI jobs. It seems to be ok now and I killed all the running jobs. Your labeled ready job never started because we had too many in the queue. I will start this one now.

JessicaMeixner-NOAA · 2024-06-13T15:18:18Z

@RussTreadon-NOAA I had the same issue and submitted a ticket, they rebooted the nodes and things seem to be better now.

* upstream/develop: Add Hercules-EMC to the Jenkins configurable parameter list (NOAA-EMC#2685) Update gdas.cd and gsi_utils hashes (NOAA-EMC#2641) Add ability to use GEFS replay ICs (NOAA-EMC#2559) Replace `sleep` with `wait_for_file` (NOAA-EMC#2586)

* origin/develop: Add observation preparation job for aerosols DA to workflow (NOAA-EMC#2624) Remove ocean daily files (NOAA-EMC#2689) Update Jenkinsfile Add Hercules-EMC to the Jenkins configurable parameter list (NOAA-EMC#2685) Update gdas.cd and gsi_utils hashes (NOAA-EMC#2641) Add ability to use GEFS replay ICs (NOAA-EMC#2559) Replace `sleep` with `wait_for_file` (NOAA-EMC#2586) Add COM template for JEDI obs (NOAA-EMC#2678) Link both global-nest fix files and non-nest ones at the same time (NOAA-EMC#2632) Update ufs-weather-model (NOAA-EMC#2663) Add ability to process ocean/ice products specific to GEFS (NOAA-EMC#2561) Update cleanup job to use COMIN/COMOUT (NOAA-EMC#2649) Add overwrite to creat experiment in BASH CI (NOAA-EMC#2676) Add handling to select CRTM cloud optical table based on cloud scheme and update calcanal_gfs.py (NOAA-EMC#2645) Refs NOAA-EMC#2475

@RussTreadon-NOAA

This PR moves JCB into GDASapp. The PR also bumps up the hash of GDASapp to what is in `feature/move_jcb`, which at time of writing is develop plus the absorption of JCB into GDASapp. Note that I also took the changes from #2641 to follow the testing @RussTreadon-NOAA has done.

This PR updates the `sorc/gdas.cd` and `sorc/gsi_utils` hashes. The updated hashes bring in bug fixes, new UFS DA functionality, and a Gaea build for gsi_utils. Resolves #2640

@RussTreadon-NOAA

This PR moves JCB into GDASapp. The PR also bumps up the hash of GDASapp to what is in `feature/move_jcb`, which at time of writing is develop plus the absorption of JCB into GDASapp. Note that I also took the changes from #2641 to follow the testing @RussTreadon-NOAA has done.

update gdas.cd and jcb hashes (NOAA-EMC#2640)

215cd32

RussTreadon-NOAA self-assigned this May 30, 2024

RussTreadon-NOAA and others added 4 commits May 31, 2024 15:48

update enkf job log filenames for enkf archive (NOAA-EMC#2640)

67da370

Merge branch 'develop' into feature/update_gdasapp

5c80e8d

update gdas.ch hash (NOAA-EMC#2640)

c25733b

Merge branch 'NOAA-EMC:develop' into feature/update_gdasapp

0488e70

RussTreadon-NOAA mentioned this pull request Jun 4, 2024

Update JEDI hashes in sorc NOAA-EMC/GDASApp#1123

Merged

RussTreadon-NOAA and others added 4 commits June 4, 2024 12:28

Merge branch 'develop' into feature/update_gdasapp

7ff01bf

Merge branch 'develop' into feature/update_gdasapp

af278a1

Merge remote-tracking branch 'upstream/develop' into feature/update_g…

cccf4ec

…dasapp

update gdas.cd hash to bring in jedi updates (NOAA-EMC#2640)

1dc1a11

RussTreadon-NOAA marked this pull request as ready for review June 5, 2024 23:07

RussTreadon-NOAA added 2 commits June 6, 2024 07:27

Merge branch 'develop' into feature/update_gdasapp

8bc2c2b

Merge branch 'develop' into feature/update_gdasapp

1ca534c

danholdaway mentioned this pull request Jun 6, 2024

Update GDASapp hash to move JCB into GDASapp #2665

Merged

7 tasks

WalterKolczynski-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label Jun 6, 2024

emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jun 6, 2024

emcbot added the CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully label Jun 13, 2024

WalterKolczynski-NOAA added CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jun 13, 2024

emcbot added CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS and removed CI-Wcoss2-Ready **CM use only** PR is ready for CI testing on WCOSS labels Jun 13, 2024

emcbot added CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress and removed CI-Wcoss2-Building **Bot use only** CI testing is cloning/building on WCOSS labels Jun 13, 2024

emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels Jun 13, 2024

Merge branch 'develop' into feature/update_gdasapp

578c01d

WalterKolczynski-NOAA removed the CI-Hercules-Ready **CM use only** PR is ready for CI testing on Hercules label Jun 13, 2024

WalterKolczynski-NOAA merged commit ebacebf into NOAA-EMC:develop Jun 13, 2024
5 of 10 checks passed

RussTreadon-NOAA deleted the feature/update_gdasapp branch June 13, 2024 17:41

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update gdas.cd and gsi_utils hashes #2641

Update gdas.cd and gsi_utils hashes #2641

RussTreadon-NOAA commented May 30, 2024 •

edited

Loading

RussTreadon-NOAA commented May 30, 2024 •

edited

Loading

RussTreadon-NOAA commented May 30, 2024

RussTreadon-NOAA commented May 31, 2024

RussTreadon-NOAA commented Jun 6, 2024

RussTreadon-NOAA commented Jun 6, 2024

emcbot commented Jun 6, 2024

emcbot commented Jun 6, 2024

RussTreadon-NOAA commented Jun 7, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

RussTreadon-NOAA commented Jun 13, 2024

TerrenceMcGuinness-NOAA commented Jun 13, 2024

JessicaMeixner-NOAA commented Jun 13, 2024

Update gdas.cd and gsi_utils hashes #2641

Update gdas.cd and gsi_utils hashes #2641

Conversation

RussTreadon-NOAA commented May 30, 2024 • edited Loading

Description

Type of change

Change characteristics

How has this been tested?

Checklist

RussTreadon-NOAA commented May 30, 2024 • edited Loading

RussTreadon-NOAA commented May 30, 2024

RussTreadon-NOAA commented May 31, 2024

RussTreadon-NOAA commented Jun 6, 2024

RussTreadon-NOAA commented Jun 6, 2024

emcbot commented Jun 6, 2024

emcbot commented Jun 6, 2024

RussTreadon-NOAA commented Jun 7, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

emcbot commented Jun 13, 2024

RussTreadon-NOAA commented Jun 13, 2024

TerrenceMcGuinness-NOAA commented Jun 13, 2024

JessicaMeixner-NOAA commented Jun 13, 2024

RussTreadon-NOAA commented May 30, 2024 •

edited

Loading

RussTreadon-NOAA commented May 30, 2024 •

edited

Loading