Hpc stack update (Issue 1465) #1596

zach1221 · 2023-02-02T15:22:54Z

Description

Updated module paths added to modulefiles for new HPC-Stack locations created on tier-1 on-prem HPCs.
Testing of the new HPC-Stack locations has been conducted successfully on Hera, Jet, Orion, Cheyenne intel/gnu and Hera intel/gnu. Updates also made for autoRT baseline capability and ORT scripts.

Related issue to HPC-Stack updates #1465

New features/additions being added through this PR.
New HPC-Stack module locations added to modulefiles for the below HPCs.

Jet intel and intel debug
Hera intel and intel debug
Hera gnu and gnu debug
Cheyenne intel and intel debug
Cheyenne gnu and gnu debug
Orion intel and intel debug

Updates to the below files for autoRT baseline capability. (Adding new workdir and blstore paths)

bl.py

Updates to the below files to correct issues with ORT jenkins-ci pipeline. (Update ORT scripts to correct logic related to TASKS variables and cmp_proc_bind.sh)

wrt_env.sh, dcp.sh, mpi.sh, rst.sh, std.sh, thr.sh
opnReqTest, run_test.sh
jenkinsfile

Top of commit queue on: TBD

Input data additions/changes

No changes are expected to input data.

Anticipated changes to regression tests:

No Baseline Change

Subcomponents involved:

Combined with PR's (If Applicable):

Commit Queue Checklist:

Link PR's from all sub-components involved
Confirm reviews completed in sub-component PR's
Add all appropriate labels to this PR.
Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

RDHPCS
- Intel
  - Hera
  - Orion
  - Jet
  - Gaea
  - Cheyenne
- GNU
  - Hera
  - Cheyenne
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

jkbk2004 · 2023-02-02T16:02:42Z

need to revert hera/gnu to 9.2 and cheyenne/gnu to 10.1

zach1221 · 2023-02-02T16:51:24Z

@jkbk2004 I have reverted hera/gnu to 9.2 and cheyenne/gnu to 10.1.

jkbk2004 · 2023-02-02T17:05:16Z

@jkbk2004 I have reverted hera/gnu to 9.2 and cheyenne/gnu to 10.1.

@natalie-perlin @zach1221 can we make sure about these reverting cases?

jkbk2004 · 2023-02-02T19:15:12Z

cheyenne.gnu has some issues. Build crashes:
`Error: Type mismatch between actual argument at (1) and actual argument at (2) (REAL(4)/REAL(8)).
/glade/scratch/jongkim/UFS-RT-tests/rt-1595-gnu/HYCOM-interface/HYCOM/mod_xc_mp.h:1196:25:

1196 | call mpi_bcast(ga(1,j0_pe(mpe_1(np),np)+1), &
| 1`

jkbk2004 · 2023-02-02T19:25:37Z

Orion needs update fms/g2tmpl:
Lmod has detected the following error: The following module(s) are unknown: "fms/2022.04" "g2tmpl/1.10.2"

jkbk2004 · 2023-02-02T20:01:03Z

On gaea, some issues as well:
Lmod has detected the following error: /lustre/f2/pdata/ncep/Jong.Kim/rt-1595/modulefiles/ufs_gaea.intel: (ufs_gaea.intel): invalid command name "export"

jkbk2004 · 2023-02-02T20:15:00Z

@natalie-perlin can you address the above issues?

jkbk2004 · 2023-02-03T14:07:50Z

@natalie-perlin Build is not going thru on Orion. Please, take a look at: /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006

jkbk2004 · 2023-02-03T14:09:23Z

@natalie-perlin still same build trouble on gaea: /lustre/f2/scratch/Jong.Kim/FV3_RT/rt_7709/compile_001

natalie-perlin · 2023-02-03T14:13:40Z

Updated fms/2022.04 and g2tmpl/1.10.2 on Orion

jkbk2004 · 2023-02-03T14:23:42Z

Updated fms/2022.04 and g2tmpl/1.10.2 on Orion
It's still not working on orion.
+ cmake /work/noaa/epic-ps/jongkim/UFS-RT-tests/rt-1595 -DAPP=S2S -DCCPP_SUITES=FV3_GFS_v17_coupled_p8_sfcocn -DCMEPS_AOFLUX=ON -DMPI=ON -DCMAKE_BUILD_TYPE=Release -DMOM6SOLO=ON CMake Error at /apps/cmake-3.22.1/share/cmake-3.22/Modules/CMakeDetermineCCompiler.cmake:49 (message): Could not find compiler set in environment variable CC:

natalie-perlin · 2023-02-03T14:42:04Z

On Orion: Cmake complains about the following env. variables not set:
(See /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006/err)

CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage

Suggest to add the following to ./modulefiles/ufs_orion.intel.lua (after L29):

setenv("CMAKE_C_COMPILER", "mpiicc")
setenv("CMAKE_CXX_COMPILER", "mpiicpc")
setenv("CMAKE_Fortran_COMPILER", "mpiifort")

Similar env. variable settings to be added to ufs_orion.intel_debug.lua.

natalie-perlin · 2023-02-03T17:19:58Z

@jkbk2004, @zach1221 - regarding the issue on Gaea:
Please set the correct module environment for Gaea. Use the following EPIC-managed Lmod installed on Gaea in module-setup.sh , following the line that states elif [[ $MACHINE_ID = gaea* ]] ; then

      
    if ( ! eval module help > /dev/null 2>&1 ) ; then
      # Use EPIC-managed Lmod installed on Gaea
      export BASH_ENV="/lustre/f2/dev/role.epic/contrib/apps/lmod/lmod/init/bash"
      source $BASH_ENV
      export LMOD_SYSTEM_DEFAULT_MODULES="modules/3.2.11.4"
      module --initial_load --no_redirect restore
    fi

The following lines need then to be removed from compile.sh, as they likely overwrite the module environment initialization done in setup-modules.sh
# Activate lua environment for gaea
if [[ $MACHINE_ID == gaea.* ]] ; then
source /lustre/f2/pdata/esrl/gsd/contrib/lua-5.1.4.9/init/init_lmod.sh
fi

natalie-perlin · 2023-02-03T17:24:48Z

The module on Gaea needs to be converted to lua format. This is the modulefile ufs_gaea.intel.lua :

help([[
  This module loads libraries required for building and running UFS Weather Model 
  on the NOAA RDHPC machine Gaea using Intel-2022.1.2
]])

whatis([===[Loads libraries needed for building the UFS Weather Model on Gaea ]===])

prepend_path("MODULEPATH", "/lustre/f2/dev/role.epic/contrib/modulefiles")
load(pathJoin("miniconda3",os.getenv("miniconda_ver") or "4.12.0"))

load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))

prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-2021.3.0/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("intel", os.getenv("intel_ver") or "2021.3.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2021.3.0"))
load(pathJoin("hpc-cray-mpich", os.getenv("hpc_cray_mpich_ver") or "7.7.11"))
load(pathJoin("gcc", os.getenv("gcc_ver") or "8.3.0"))
load(pathJoin("libpng", os.getenv("libpng_ver") or "1.6.37"))

-- needed for WW3 build
load(pathJoin("gcc", os.getenv("gcc_ver") or "8.3.0"))
-- Needed at runtime:
load("alps")
load("rocoto")

load("ufs_common")

setenv("CC","cc")
setenv("FC","ftn")
setenv("CXX","CC")
setenv("CMAKE_C_COMPILER","cc")
setenv("CMAKE_CXX_COMPILER","CC")
setenv("CMAKE_Fortran_COMPILER","ftn")
setenv("CMAKE_Platform","gaea.intel")

natalie-perlin · 2023-02-03T17:26:30Z

The modulefile ufs_gaea.intel_debug.lua is similar to ufs_gaea.intel.lua, except for this line:
load("ufs_common_debug")

natalie-perlin · 2023-02-03T17:47:45Z

@zach1221 , @jkbk2004

@jkbk2004 I have reverted hera/gnu to 9.2 and cheyenne/gnu to 10.1.

What were the questions or issues with higher order compilers on Hera and Cheyenne?

zach1221 · 2023-02-03T17:59:04Z

@natalie-perlin I think it was just to recheck/confirm that there were no issues with the below compilers on these two HPCs.

zach1221 · 2023-02-03T18:50:34Z

@natalie-perlin I added the "setenv" variable for cmake compiler to the Orion module lua file but still failed with same "language not enabled error".

jkbk2004 · 2023-02-03T19:01:41Z

On Orion: Cmake complains about the following env. variables not set: (See /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006/err)
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage
Suggest to add the following to ./modulefiles/ufs_orion.intel.lua (after L29):
setenv("CMAKE_C_COMPILER", "mpiicc")
setenv("CMAKE_CXX_COMPILER", "mpiicpc")
setenv("CMAKE_Fortran_COMPILER", "mpiifort")
Similar env. variable settings to be added to ufs_orion.intel_debug.lua.

@natalie-perlin I tried but it still not working.

jkbk2004 · 2023-02-03T19:04:00Z

The module on Gaea needs to be converted to lua format. This is the modulefile ufs_gaea.intel.lua :

help([[
  This module loads libraries required for building and running UFS Weather Model 
  on the NOAA RDHPC machine Gaea using Intel-2022.1.2
]])

whatis([===[Loads libraries needed for building the UFS Weather Model on Gaea ]===])

prepend_path("MODULEPATH", "/lustre/f2/dev/role.epic/contrib/modulefiles")
load(pathJoin("miniconda3",os.getenv("miniconda_ver") or "4.12.0"))

load(pathJoin("cmake", os.getenv("cmake_ver") or "3.20.1"))

prepend_path("MODULEPATH","/lustre/f2/dev/role.epic/contrib/hpc-stack/intel-2021.3.0/modulefiles/stack")
load(pathJoin("hpc", os.getenv("hpc_ver") or "1.2.0"))
load(pathJoin("intel", os.getenv("intel_ver") or "2021.3.0"))
load(pathJoin("hpc-intel", os.getenv("hpc_intel_ver") or "2021.3.0"))
load(pathJoin("hpc-cray-mpich", os.getenv("hpc_cray_mpich_ver") or "7.7.11"))
load(pathJoin("gcc", os.getenv("gcc_ver") or "8.3.0"))
load(pathJoin("libpng", os.getenv("libpng_ver") or "1.6.37"))

-- needed for WW3 build
load(pathJoin("gcc", os.getenv("gcc_ver") or "8.3.0"))
-- Needed at runtime:
load("alps")
load("rocoto")

load("ufs_common")

setenv("CC","cc")
setenv("FC","ftn")
setenv("CXX","CC")
setenv("CMAKE_C_COMPILER","cc")
setenv("CMAKE_CXX_COMPILER","CC")
setenv("CMAKE_Fortran_COMPILER","ftn")
setenv("CMAKE_Platform","gaea.intel")

This should be separate feature. We are not using lua on gaea yet. We need to confirm your hpc stack installation work first.

natalie-perlin · 2023-02-03T19:17:49Z

On Orion: Cmake complains about the following env. variables not set: (See /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006/err)
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_Fortran_COMPILER not set, after EnableLanguage
Suggest to add the following to ./modulefiles/ufs_orion.intel.lua (after L29):
setenv("CMAKE_C_COMPILER", "mpiicc")
setenv("CMAKE_CXX_COMPILER", "mpiicpc")
setenv("CMAKE_Fortran_COMPILER", "mpiifort")
Similar env. variable settings to be added to ufs_orion.intel_debug.lua.
@natalie-perlin I tried but it still not working.

@jkbk2004 -
regarding Orion, where is the directory with the WM code that is being built?

natalie-perlin · 2023-02-03T19:23:01Z

@natalie-perlin I added the "setenv" variable for cmake compiler to the Orion module lua file but still failed with same "language not enabled error".

Let me know where is the modulefile is and the log with the error

jkbk2004 · 2023-02-03T19:41:48Z

@natalie-perlin can you read the crashing errors and paths above?

natalie-perlin · 2023-02-03T20:01:07Z

@natalie-perlin can you read the crashing errors and paths above?

@jkbk2004 @zach1221 - Not finding the log file with the "language not enabled error" message on Orion

jkbk2004 · 2023-02-03T20:04:47Z

Please, take a look /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006/err

natalie-perlin · 2023-02-03T20:19:04Z

@natalie-perlin I think it was just to recheck/confirm that there were no issues with the below compilers on these two HPCs.

Cheyenne gnu/10.1.0 is verified.

Hera gnu/9.2 in /scratch1/NCEPDEV/nems/role.epic/hpc-stack/libs/gnu-9.2/ is verified as well (with mpich/3.3.2)

jkbk2004 · 2023-02-03T20:53:41Z

Cheyenne.gnu passes ok!

natalie-perlin · 2023-02-03T21:11:19Z

Please, take a look /work/noaa/epic-ps/jongkim/rt-1595/stmp/jongkim/FV3_RT/rt_1450/compile_006/err

This log file with the error has a timestamp from earlier time today, before the modulefile ufs_orion.intel.lua changed. Are there any tests that have been run following the modulefile update?

UPD: ... Found new test locations, looking at them now

zach1221 · 2023-02-09T14:52:10Z

@jkbk2004 gaea modulefiles are reverted.

zach1221 · 2023-02-09T15:49:03Z

Jenkins-ci logs attached.
ufs-weather-model » ort-docker-pipeline » PR-1596 #9 Console [Jenkins].pdf

modulefiles/ufs_hera.gnu_debug.lua

tests/opnReqTests/dcp.sh

natalie-perlin · 2023-02-09T19:47:17Z

@jkbk2004 - Gaea transition has not been done yet.

Update ufs_orion.intel.lua

2a2554f

zach1221 added the jenkins-ci Jenkins CI: ORT build/test on docker container label Feb 3, 2023

Update ufs_gaea.intel_debug

d096257

zach1221 linked an issue Feb 9, 2023 that may be closed by this pull request

ORT script failure due to TASK-related variable changes #1586

Closed

zach1221 linked an issue Feb 9, 2023 that may be closed by this pull request

Updating hpc-stack modules and miniconda locations for Hera, Gaea, Cheyenne, Orion, Jet #1465

Closed

zach1221 marked this pull request as ready for review February 9, 2023 16:04

add hera intel.gnu RT logs: passed

623e802

jkbk2004 requested review from BrianCurtis-NOAA and jkbk2004 February 9, 2023 16:27

jkbk2004 added Waiting for Reviews The PR is waiting for reviews from associated component PR's. and removed jenkins-ci Jenkins CI: ORT build/test on docker container labels Feb 9, 2023

BrianCurtis-NOAA reviewed Feb 9, 2023

View reviewed changes

modulefiles/ufs_hera.gnu_debug.lua Outdated Show resolved Hide resolved

BrianCurtis-NOAA reviewed Feb 9, 2023

View reviewed changes

tests/opnReqTests/dcp.sh Outdated Show resolved Hide resolved

BrianCurtis-NOAA approved these changes Feb 9, 2023

View reviewed changes

zach1221 added 11 commits February 9, 2023 11:44

Update ufs_hera.gnu_debug.lua

1421d9a

Update ufs_cheyenne.gnu_debug.lua

d6dba60

Update ufs_cheyenne.intel_debug.lua

6b560da

Update ufs_hera.intel_debug.lua

a026038

Update ufs_jet.intel_debug.lua

fd00dac

Update ufs_orion.intel_debug.lua

3e84274

Update dcp.sh

e9040df

Update mpi.sh

9e5cd0c

Update rst.sh

fa0caa0

Update thr.sh

694a8bd

Update run_test.sh

42e5139

jkbk2004 approved these changes Feb 9, 2023

View reviewed changes

jkbk2004 merged commit 0c8e74c into ufs-community:develop Feb 9, 2023

natalie-perlin mentioned this pull request Feb 17, 2023

[develop:] update hpc-stack location for Gaea, miniconda3, Lmod initialization, Lua modulefiles #1617

Closed

35 tasks

Qingfu-Liu mentioned this pull request May 19, 2023

Error: out of memory #1759

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hpc stack update (Issue 1465) #1596

Hpc stack update (Issue 1465) #1596

zach1221 commented Feb 2, 2023 •

edited

Loading

jkbk2004 commented Feb 2, 2023

zach1221 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

zach1221 commented Feb 3, 2023

zach1221 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 •

edited

Loading

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 •

edited

Loading

zach1221 commented Feb 9, 2023

zach1221 commented Feb 9, 2023

natalie-perlin commented Feb 9, 2023 •

edited

Loading

Hpc stack update (Issue 1465) #1596

Hpc stack update (Issue 1465) #1596

Conversation

zach1221 commented Feb 2, 2023 • edited Loading

Top of commit queue on: TBD

Input data additions/changes

Anticipated changes to regression tests:

Subcomponents involved:

Combined with PR's (If Applicable):

Commit Queue Checklist:

Linked PR's and Issues:

Testing Day Checklist:

Testing Log (for CM's):

jkbk2004 commented Feb 2, 2023

zach1221 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 2, 2023

jkbk2004 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 • edited Loading

natalie-perlin commented Feb 3, 2023 • edited Loading

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

zach1221 commented Feb 3, 2023

zach1221 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 • edited Loading

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023

jkbk2004 commented Feb 3, 2023

natalie-perlin commented Feb 3, 2023 • edited Loading

zach1221 commented Feb 9, 2023

zach1221 commented Feb 9, 2023

natalie-perlin commented Feb 9, 2023 • edited Loading

zach1221 commented Feb 2, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 3, 2023 •

edited

Loading

natalie-perlin commented Feb 9, 2023 •

edited

Loading