Upgrade spack-stack from version 1.5.1 to 1.6.0 #2093

RatkoVasic-NOAA · 2024-01-16T18:17:16Z

Commit Queue Requirements:

Fill out all sections of this template.
All sub component pull requests have been reviewed by their code managers.
Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
Add list of all failed regression tests in "Regression Tests" section.

PR Information

Change modulefiles to upgrade spack-stack libraries from version 1.5.1 to 1.6.0

Description

Change modulefiles to upgrade spack-stack libraries from version 1.5.1 to 1.6.0

Commit Message

Solves issue #2091

UFSWM:
- Updates modulefiles to spack-stack version 1.6.0
- Updates Hera GNU version to 13.3

Priority

Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
Normal

Blocking Dependencies

Git Issues Fixed By This PR

#2091

Changes

Subcomponent (with links)

Input data

No changes are expected to input data.
Changes are expected to input data:
- New input data.
- Updated input data.

Regression Tests:

No changes are expected to any regression test.
Changes are expected to the following tests:

Libraries

Not Needed
Needed
- Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
- Add issue link from JCSDA/spack-stack following this item

Testing Log:

RDHPCS
- Hera
- Orion
- Hercules
- Jet
- Gaea
- Derecho
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

RatkoVasic-NOAA · 2024-01-16T18:18:25Z

For now

Hera(Intel & GNU)
Orion (Intel)
Hercules (Intel & GNU)
Gaea-C5 (Intel)
Jet (Intel)

are done.

jkbk2004 · 2024-01-17T20:32:25Z

Should we bring in the change of NOAA-GFDL/GFDL_atmos_cubed_sphere#299 ?

zach1221 · 2024-01-22T18:06:16Z

Hi, @RatkoVasic-NOAA . I'm doing some preliminary testing against this PR, across the HPCs. Can you sync up your branch for me, please?

RatkoVasic-NOAA · 2024-01-22T18:27:11Z

Hi, @RatkoVasic-NOAA . I'm doing some preliminary testing against this PR, across the HPCs. Can you sync up your branch for me, please?

Done

junwang-noaa · 2024-01-24T14:54:24Z

@Hang-Lei-NOAA would you please let us know the module files to use on acorn for spack-stack 1.6.0 and also the hpc-stack version for wcoss2? We can test the hpc-stack version on acorn. Thanks

Hang-Lei-NOAA · 2024-01-24T15:28:29Z

Hi Jun, the spack-stack 1.6.0 has been installed on acorn under /lfs/h1/emc/nceplibs/noscrub/spack-stack/ This should work fine with ufs default modulefiles for spack-stack. For the running with hpc-stack installations, please use following modulefile. A exactly same library settings to the UFS repo was made on acorn for further testing: /lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_acorn.intel.lua /lfs/h1/emc/eib/noscrub/Hang.Lei/ufs-weather-model/modulefiles/ufs_common.lua We are still missing some latest versions of libraries on cactus and Dogwoods to match spack-stack. I will test a wcoss2 version based on current libraries on Cactus. And push missing versions there soon. Thanks Hang

…

On Wed, Jan 24, 2024 at 9:54 AM Jun Wang ***@***.***> wrote: @Hang-Lei-NOAA <https://github.com/Hang-Lei-NOAA> would you please let us know the module files to use on acorn for spack-stack 1.6.0 and also the hpc-stack version for wcoss2? We can test the hpc-stack version on acorn. Thanks — Reply to this email directly, view it on GitHub <#2093 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AKWSMFBXFUHWM6NOHEMZKZLYQEOCXAVCNFSM6AAAAABB5IDDBSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBYGI4TGMRQHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

DavidHuber-NOAA · 2024-01-25T15:05:37Z

@RatkoVasic-NOAA I have a draft PR to upgrade the UPP to spack-stack 1.6.0 (NOAA-EMC/UPP#856). There are a few more updates to go in, but if it is successful, could we update the UPP hash?

RatkoVasic-NOAA · 2024-01-25T18:59:40Z

@DavidHuber-NOAA great! You can make PR to this PR with UPP changes.

JiliDong-NOAA · 2024-01-29T17:42:41Z

I have a PR to update UPP hash for fix on missing reflectivity when using inline post

#2113

If there is a plan to update UPP in this PR soon, I will close mine. There are three RTs failed from the UPP update. These failed RTs are expected with the baseline changes from:

REFZI and REFZR are switched to missing values as expected, which is similar to what Thompson microphysics scheme does
missing value are handled differently for reflectivity when switching to the correct block: previousy missing values will be reset to dbzmin and they are now kept as is.

DavidHuber-NOAA · 2024-01-29T18:10:13Z

@JiliDong-NOAA Yes, I am planning on pushing an updated hash for the UPP shortly to capture its upgrade to spack-stack v1.6.0.

JiliDong-NOAA · 2024-01-29T18:50:41Z

@JiliDong-NOAA Yes, I am planning on pushing an updated hash for the UPP shortly to capture its upgrade to spack-stack v1.6.0.

Sounds good. Thanks.

souopgui · 2024-02-02T13:36:25Z

I ran the RT on S4 with BL_DATE=20240111;
RegressionTests_s4.log

You can ignore cpld_control_gfsv17_iau_intel and cpld_control_c48_intel, they also failed in the baseline, I have not investigated yet.

The following two failed:

rap_control_dyn64_phy32_debug_intel
cpld_control_nowave_noaero_p8_intel

DeniseWorthen · 2024-02-06T02:09:26Z

Where are we on this PR? As far as I know, we can't run coupled GNU tests on Hercules until we move to 1.6, right? Since Hercules is our only other GNU platform, this means all developer testing needs to happen on Hera, which is very slow right now.

jkbk2004 · 2024-02-06T02:24:33Z

Can we keep updating the check list at #2091 ?

S4: Upgrade to use spack stack 1.6.0 on s4

zach1221 · 2024-02-12T16:54:43Z

@RatkoVasic-NOAA I tested the 1.6.0 installation on Gaea-C5, and all the tests passed, using the installation path (/ncrc/proj/epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/modulefiles/Core). Could you sync up this branch, so that is reflected in the gaea modulefile?

…er-model into ss-160

DeniseWorthen · 2024-06-04T18:39:51Z

Modifying the MPI_Init_thread calls in two routines: ./WW3/model/src/ww3_multi.F90 and ./WW3/model/src/ww3_shel.F90
to CALL MPI_INIT_THREAD( MPI_THREAD_SERIALIZED, THRLEV, IERR_MPI)
from CALL MPI_INIT_THREAD( MPI_THREAD_FUNNELED, THRLEV, IERR_MPI)

These two programs are used for the standalone ww3 and are unused for the coupled model.

natalie-perlin · 2024-06-04T19:44:15Z

Modifying the MPI_Init_thread calls in two routines: ./WW3/model/src/ww3_multi.F90 and ./WW3/model/src/ww3_shel.F90
to CALL MPI_INIT_THREAD( MPI_THREAD_SERIALIZED, THRLEV, IERR_MPI)
from CALL MPI_INIT_THREAD( MPI_THREAD_FUNNELED, THRLEV, IERR_MPI)

These two programs are used for the standalone ww3 and are unused for the coupled model.

This is great to know, thank you , Denise, that's very important to know!! (in case my suspicions are valid)

zach1221 · 2024-06-05T11:59:14Z

Looks like @natalie-perlin was right, cpld_debug_p8_gnu and cpld_control_p8_gnu are now passing on hercules but failing on hera. I'm getting the same "OSC pt2pt component does not support MPI_THREAD_MULTIPLE" error message now.
/scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_RT/rt_3687442/cpld_debug_p8_gnu/err

The rest of the new hera gnu baselines are completed and final matching should be finished soon.

DeniseWorthen · 2024-06-05T12:18:00Z

I believe this must be an issue related to MAPL, since all non-aerosol cpld and datm tests are working for hera+gnu. I took @zach1221 run directory, turned off CHM and it is now running.

zach1221 · 2024-06-05T12:26:15Z

I added the two cases to issue 2263, just in case.

DeniseWorthen · 2024-06-05T14:55:14Z

@zach1221 I'm confused about the reference to 2263. I just ran the control_c48_gnu on Hera using the SS16 branch and it does not hang. Did you find that the control_c48_gnu test on Hera was still failing? It seems to me that the only Hera/GNU tests which are still failing are those which include aerosols (MAPL and the like).

zach1221 · 2024-06-05T15:04:30Z

@zach1221 I'm confused about the reference to 2263. I just ran the control_c48_gnu on Hera using the SS16 branch and it does not hang. Did you find that the control_c48_gnu test on Hera was still failing? It seems to me that the only Hera/GNU tests which are still failing are those which include aerosols (MAPL and the like).

I've updated the title to remove reference to the c48 case, as it is passing.

zach1221 · 2024-06-05T15:12:08Z

I've turned off the cpld_debug_p8 and cpld_control_p8 gnu cases on Hera, in rt.conf. So, we may proceed with the PR, as they're running and passing on Hercules. Natalie and I can continue to look into the issue on Hera.

natalie-perlin · 2024-06-05T15:21:16Z

/scratch1/NCEPDEV/stmp2/Zachary.Shrader/FV3_RT/rt_3687442/cpld_debug_p8_gnu/err

Sounds good.

What is CHM that @DeniseWorthen mentioned?

natalie-perlin · 2024-06-05T15:24:25Z

I've turned off the cpld_debug_p8 and cpld_control_p8 gnu cases on Hera, in rt.conf. So, we may proceed with the PR, as they're running and passing on Hercules. Natalie and I can continue to look into the issue on Hera.

@zach1221 - It would be very helpful to see the logs for these tests on Hercules - please let me know if there are working directories to take a look!

zach1221 · 2024-06-05T16:17:42Z

I've turned off the cpld_debug_p8 and cpld_control_p8 gnu cases on Hera, in rt.conf. So, we may proceed with the PR, as they're running and passing on Hercules. Natalie and I can continue to look into the issue on Hera.

@zach1221 - It would be very helpful to see the logs for these tests on Hercules - please let me know if there are working directories to take a look!

@natalie-perlin sure here is another run directory.
/work2/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_2847879

RatkoVasic-NOAA and others added 3 commits January 16, 2024 17:08

Upgrade ufs_common to spack-stack 1.6.0 and Hera (Intel + GNU)

736b26e

Upgrade ufs_gaea-c5.intel to spack-stack 1.6.0 Gaea-C5 (Intel)

56fd64a

Upgrade ufs_orion.intel to spack-stack 1.6.0 Orion (Intel)

d9802fd

RatkoVasic-NOAA self-assigned this Jan 16, 2024

RatkoVasic-NOAA added 2 commits January 16, 2024 13:47

Upgrade ufs_hercules.* to spack-stack 1.6.0 Hercules (Intel and GNU)

88dc785

Upgrade ufs_jet.intel to spack-stack 1.6.0 Jet (Intel)

60afb2c

DavidHuber-NOAA mentioned this pull request Jan 19, 2024

Update to spack-stack 1.6.0 NOAA-EMC/global-workflow#2239

Merged

7 tasks

Merge branch 'ufs-community:develop' into ss-160

bce073b

Merge branch 'ufs-community:develop' into ss-160

84b98c0

Upgrade to use spack stack 1.6.0 on s4

1f40d60

RatkoVasic-NOAA added 3 commits February 6, 2024 08:15

Merge pull request #1 from souopgui-redline/ss-160

3c5bcd3

S4: Upgrade to use spack stack 1.6.0 on s4

Merge branch 'ufs-community:develop' into ss-160

343958f

Merge branch 'ufs-community:develop' into ss-160

d40cefa

RatkoVasic-NOAA added 2 commits February 12, 2024 16:57

Merge branch 'develop' into ss-160

3f59c67

Merge branch 'ss-160' of https://github.com/RatkoVasic-NOAA/ufs-weath…

ea2de14

…er-model into ss-160

FernandoAndrade-NOAA and others added 2 commits June 4, 2024 16:00

add jet RT log passed

41bab87

add gaea RT log passed

1e33b95

BrianCurtis-NOAA and others added 4 commits June 4, 2024 18:44

WCOSS2 RT Log: Passed

9306490

add orion RT logs: passed

d868d90

Acorn RT Log: Passed

5ed9e37

add derecho RT logs: passed

de19f0a

add hercules RT logs: passed

420edab

zach1221 and others added 5 commits June 5, 2024 15:00

add control_p8_gnu ORT logs: passed

d61ea75

add regional_control_gnu ORT logs: passed

75b58dd

add cpld_control_gnu ORT logs: passed

1281a4a

add Hera RT logs: passed

8f8aaa1

Update rt.conf disable cpld gnu tests on Hera

42d437b

zach1221 requested review from jkbk2004 and BrianCurtis-NOAA June 5, 2024 15:12

jkbk2004 approved these changes Jun 5, 2024

View reviewed changes

jkbk2004 requested review from zach1221 and FernandoAndrade-NOAA June 5, 2024 17:32

zach1221 approved these changes Jun 5, 2024

View reviewed changes

zach1221 merged commit 485ccdf into ufs-community:develop Jun 5, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade spack-stack from version 1.5.1 to 1.6.0 #2093

Upgrade spack-stack from version 1.5.1 to 1.6.0 #2093

RatkoVasic-NOAA commented Jan 16, 2024 •

edited by zach1221

RatkoVasic-NOAA commented Jan 16, 2024 •

edited

jkbk2004 commented Jan 17, 2024 •

edited

zach1221 commented Jan 22, 2024

RatkoVasic-NOAA commented Jan 22, 2024

junwang-noaa commented Jan 24, 2024

Hang-Lei-NOAA commented Jan 24, 2024 via email

DavidHuber-NOAA commented Jan 25, 2024

RatkoVasic-NOAA commented Jan 25, 2024

JiliDong-NOAA commented Jan 29, 2024

DavidHuber-NOAA commented Jan 29, 2024

JiliDong-NOAA commented Jan 29, 2024

souopgui commented Feb 2, 2024

DeniseWorthen commented Feb 6, 2024

jkbk2004 commented Feb 6, 2024

zach1221 commented Feb 12, 2024

DeniseWorthen commented Jun 4, 2024

natalie-perlin commented Jun 4, 2024

zach1221 commented Jun 5, 2024

DeniseWorthen commented Jun 5, 2024 •

edited

zach1221 commented Jun 5, 2024

DeniseWorthen commented Jun 5, 2024

zach1221 commented Jun 5, 2024 •

edited

zach1221 commented Jun 5, 2024

natalie-perlin commented Jun 5, 2024

natalie-perlin commented Jun 5, 2024

zach1221 commented Jun 5, 2024

Upgrade spack-stack from version 1.5.1 to 1.6.0 #2093

Upgrade spack-stack from version 1.5.1 to 1.6.0 #2093

Conversation

RatkoVasic-NOAA commented Jan 16, 2024 • edited by zach1221

Commit Queue Requirements:

PR Information

Description

Commit Message

Priority

Blocking Dependencies

Git Issues Fixed By This PR

Changes

Subcomponent (with links)

Input data

Regression Tests:

Libraries

Testing Log:

RatkoVasic-NOAA commented Jan 16, 2024 • edited

jkbk2004 commented Jan 17, 2024 • edited

zach1221 commented Jan 22, 2024

RatkoVasic-NOAA commented Jan 22, 2024

junwang-noaa commented Jan 24, 2024

Hang-Lei-NOAA commented Jan 24, 2024 via email

DavidHuber-NOAA commented Jan 25, 2024

RatkoVasic-NOAA commented Jan 25, 2024

JiliDong-NOAA commented Jan 29, 2024

DavidHuber-NOAA commented Jan 29, 2024

JiliDong-NOAA commented Jan 29, 2024

souopgui commented Feb 2, 2024

DeniseWorthen commented Feb 6, 2024

jkbk2004 commented Feb 6, 2024

zach1221 commented Feb 12, 2024

DeniseWorthen commented Jun 4, 2024

natalie-perlin commented Jun 4, 2024

zach1221 commented Jun 5, 2024

DeniseWorthen commented Jun 5, 2024 • edited

zach1221 commented Jun 5, 2024

DeniseWorthen commented Jun 5, 2024

zach1221 commented Jun 5, 2024 • edited

zach1221 commented Jun 5, 2024

natalie-perlin commented Jun 5, 2024

natalie-perlin commented Jun 5, 2024

zach1221 commented Jun 5, 2024

RatkoVasic-NOAA commented Jan 16, 2024 •

edited by zach1221

RatkoVasic-NOAA commented Jan 16, 2024 •

edited

jkbk2004 commented Jan 17, 2024 •

edited

DeniseWorthen commented Jun 5, 2024 •

edited

zach1221 commented Jun 5, 2024 •

edited