Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build updates for WCOSS2 #1002

Merged
merged 26 commits into from
Aug 26, 2022
Merged

Build updates for WCOSS2 #1002

merged 26 commits into from
Aug 26, 2022

Conversation

KateFriedman-NOAA
Copy link
Member

@KateFriedman-NOAA KateFriedman-NOAA commented Aug 26, 2022

Description

This PR brings in changes to build the system on WCOSS2. It includes:

  • Add WCOSS2 stanzas to modulefiles/module-setup.csh.inc, modulefiles/module-setup.sh.inc, sorc/machine-setup.sh, and ush/load_fv3gfs_modules.sh.
  • Create new WCOSS2 LUA modulefiles: modulefiles/module_base.wcoss2.lua, modulefiles/modulefile.ww3.wcoss2.lua, and modulefiles/workflow_utils.wcoss2.lua.
  • Add WCOSS2 to sorc/link_workflow.sh.

With the above changes the develop branch will build out-of-the-box with or without the -g checkout option and the -a S2SW build option (not the current default of the S2SWA app). Run build_all.sh as: ./build_all.sh -a S2SW
The -u checkout option is not yet supported since the GDASapp does not yet support WCOSS2.

./checkout.sh [-g]
./build_all.sh -a S2SW
./link_workflow.sh emc wcoss2

This PR DOES NOT include any changes to run the system on WCOSS2, only build updates.

Type of change

  • New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Build tests on WCOSS2, including out-of-the-box with a fresh clone.

Checkout and build with -g checkout option and -a S2SW build option:

kate.friedman@clogin06:/lfs/h2/emc/global/noscrub/kate.friedman/git/feature_dev-wcoss2_build/sorc> ./checkout.sh -g
Receieved -g flag for optional checkout of GSI-based DA
Performing checkout of ufs_model
|-- Cloning from https://github.com/ufs-community/ufs-weather-model into ufs_model.fd
|-- Checking out Prototype-P8
|-- Updating submodules (if any)

Performing checkout of ufs_utils
|-- Cloning from https://github.com/ufs-community/UFS_UTILS.git into ufs_utils.fd
|-- Checking out ufs_utils_1_8_0
|-- Updating submodules (if any)

Performing checkout of verif-global
|-- Cloning from https://github.com/NOAA-EMC/EMC_verif-global.git into verif-global.fd
|-- Checking out c267780
|-- Updating submodules (if any)

Performing checkout of gsi_enkf
|-- Cloning from https://github.com/NOAA-EMC/GSI.git into gsi_enkf.fd
|-- Checking out 67f5ab4
|-- Updating submodules (if any)

Performing checkout of gsi_utils
|-- Cloning from https://github.com/NOAA-EMC/GSI-Utils.git into gsi_utils.fd
|-- Checking out 322cc7b
|-- Updating submodules (if any)

Performing checkout of gsi_monitor
|-- Cloning from https://github.com/NOAA-EMC/GSI-Monitor.git into gsi_monitor.fd
|-- Checking out acf8870
|-- Updating submodules (if any)

Performing checkout of gldas
|-- Cloning from https://github.com/NOAA-EMC/GLDAS.git into gldas.fd
|-- Checking out fd8ba62
|-- Updating submodules (if any)

kate.friedman@clogin06:/lfs/h2/emc/global/noscrub/kate.friedman/git/feature_dev-wcoss2_build/sorc> ./build_all.sh -a S2SW
Creating ../exec folder
 .... Building WW3 pre and post execs .... 
 .... Building forecast model .... 
 .... Building gsi and enkf .... 
 .... Building gsi utilities .... 
 .... Skip building GDASApp  .... 
 .... Building gsi monitor .... 
 .... Building UPP .... 
 .... Building ufs_utils .... 
 .... Building gldas .... 
 .... Building workflow_utils .... 
 .... Building gfs_util .... 

 .... Build system finished .... 
kate.friedman@clogin06:/lfs/h2/emc/global/noscrub/kate.friedman/git/feature_dev-wcoss2_build/sorc> 

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

Refs #419

- In move to WCOSS2 the transfer*list files were moved into a new
transfer folder under the upper-level parm folder.

Refs: #419
- The transfer*list files were updated in operations during the move
from WCOSS1 to WCOSS2.
- The beginning of the path within the transfer*list files was updated
from:
  com/gfs/_ENVIR_
to:
  _COMROOT_/gfs/_SHORTVER_

Refs: #419
- Add release notes for GFSv16.1 and GFSv16.2 versions that were added
into the operations branch but not yet in the develop branch.

Refs: #419
* develop:
  Updated GFS transfer*list files from operations (develop) (#918)
- Add /gempak subfolder where needed in gempak ush scripts.
- Remove unneeded commented out path settings from older iterations.

Refs: #419
- Cleaning out older GFS version release notes.
- Includes current GFSv16.2.1 release notes, will commit GFSv16.3
release notes with implementation this fall.
- Will then keep only the latest release notes moving forward.

Refs: #419
* develop:
  Add postsnd job when bufrsnd it on (#926)
* develop:
  Add GDASapp (first wave of JEDI changes) (#871)
* develop:
  WCOSS2 gempak ush scripts updates and cleanup of old release notes (#920)
* develop:
  change gdasechgres dependency to just gdasefcs01 instead of gdasefmn (#933)
  Remove obsolete platforms (WCOSS1, Dell, Cray, Theia) references. (#922)
  Optimize DA clone, build, and link (#931)
* develop:
  Initial commit of directory comparison tools (#934)
  Add preamble, convert to bash, and remove env (#929)
  minimal intervention to create a data-atmosphere xml (#936)
  Remove Cray, Dell, WCOSS1 from module-setup.sh.inc (#943)
  bring GDASApp jjobs and exscripts to global-workflow (#941)
* develop:
  Remove SDATE=CDATE IAU block in NCO config.base (#963)
  Updates for P8  (#946)
  Fix GLDAS j-job link (#954)
  Add ocean post to archive dependencies (#949)
- Add a WCOSS2 block to machine-setup.sh.
- Change the Hera and Orion blocks to use "module reset" instead of
"module purge"; includes hack from RDHPCS sysadmins for using the
"module reset" command on those machines successfully.

Refs: #419
* develop:
  Externals.cfg was stale and had drifted off (#965)
  Fix post comparison with zero-padded numbers (#964)
* develop:
  Add initial config files for global aerosol DA (#986)
  Update diag table to remove wav-ocn coupling fields (#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (#970)
- Change the path used to check if on WCOSS2 from "lfs/h2" to "/lfs/f1"

Refs: #419
* develop:
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (#1001)
  Fix preamble id (#996)
  Add missing "atmos" into job dependencies (#998)
  Bugfix in arch.sh to remove hardwired "htar" (#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (#990)
  Add GSI monitor scripts (#969)
  Fix product generation at some fcst hrs (#988)
- Pull out the changes to replace "module purge" with "module reset" for
the R&D machines in machine-setup.sh.
- Will commit separately after testing on R&Ds.
- Removing non-WCOSS2 changes from branch ahead of WCOSS2 build PR.

Refs: #419
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes look good.
one suggestion and some other comments TODO later.

modulefiles/modulefile.ww3.wcoss2.lua Show resolved Hide resolved
modulefiles/module-setup.csh.inc Show resolved Hide resolved
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@aerorahul aerorahul merged commit 392ca6f into develop Aug 26, 2022
@aerorahul aerorahul deleted the feature/dev-wcoss2 branch August 26, 2022 20:16
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Sep 6, 2022
* develop:
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update checkout.sh to remove git submodule update for upp checkout (NOAA-EMC#768)
  Add 06, 12, and 18 WCOSS2 operational GFS defs
  Add WCOSS2 operational GFS defs files
  Update repository name for EMC_post component to UPP
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 30, 2023
* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants