enable feature/coupled-crow to work on stampede2 #261

JianKuang-Intelsat · 2021-02-10T16:03:29Z

In addition to the stampede2 porting, this branch also switch to hpc-stack on Orion. Please try run this branch on both Orion and stampede2.

updated platform adaptation routine for orion

…w into feature/hpcstack

This will help eliminate ambiguity in the future

…re/hpcstack

Why not happening on Orion? The Orion shell keep operates even if setpdy.sh encountered this error (non-fatal for Orion); But it works differently on stampede2

…e/hpcstack Conflicts: sorc/build_fv3_coupled.sh sorc/checkout.sh sorc/link_fv3gfs.sh sorc/machine-setup.sh ush/load_fv3gfs_modules.sh

JessicaMeixner-NOAA

Is there an example directory for output with this change for orion? I'm guessing someone else will need to test for the UPP changes on hera as well. Do I need to do this or is someone else available to test this on hera?

I think the fix files need to be updated on Stampede.

JessicaMeixner-NOAA · 2021-02-10T16:07:09Z

modulefiles/module_base.orion_bk

@@ -0,0 +1,68 @@
+#%Module######################################################################


This file is not needed, is there an example test case on orion since the orion modules were updated?

The same case file for the trunk works here.

Should this be removed?

Agree this should be removed

This file needs to be removed before PR is merged.

sorc/link_fv3gfs.sh

JessicaMeixner-NOAA · 2021-02-10T16:08:24Z

sorc/machine-setup.sh_bk

@@ -0,0 +1,218 @@
+# Create a test function for sh vs. bash detection.  The name is


Is this file needed?

workflow/cases/coupled_free_forecast_wave_test.yaml

JianKuang-Intelsat · 2021-02-10T16:34:11Z

@JessicaMeixner-NOAA Thanks for these comments. I think since the primary purpose of this PR is to enable stampede2 running (stampede2 ONLY has hpc-stack modules, as of I know), and coupled-crow is functional on Hera as of now, we should leave hera-hpcstack adoption to a future PR.

JessicaMeixner-NOAA · 2021-02-10T16:41:56Z

@JessicaMeixner-NOAA Thanks for these comments. I think since the primary purpose of this PR is to enable stampede2 running (stampede2 ONLY has hpc-stack modules, as of I know), and coupled-crow is functional on Hera as of now, we should leave hera-hpcstack adoption to a future PR.

Agreed, however I think hera still needs to be tested for this PR because the version of ncep_post and how it was built is changed. Therefore, I think we need to ensure nothing was broken on hera. In addition, there are updates for orion in this PR, so we should make sure it's been tested there as well, even though that was not the main goal of this PR, which as you stated is the enable stampede2.

JianKuang-Intelsat · 2021-02-10T16:45:40Z

@JessicaMeixner-NOAA Yes. I agree.

remove excess files

JessicaMeixner-NOAA · 2021-02-10T21:14:43Z

@JianKuang-NOAA I noticed this branch is out of date when I was checking it out to test on hera, can you please merge the latest feature/coupled-crow so I can test this on hera -- if someone else is already running the test on hera, please let me know.

…e/hpcstack

JianKuang-Intelsat · 2021-02-10T21:17:59Z

@JessicaMeixner-NOAA Thanks for your mention. I just merged it.

ush/load_fv3gfs_modules.sh

JessicaMeixner-NOAA · 2021-02-11T14:32:08Z

@JianKuang-NOAA my attempt on hera failed, but the error seemed weird so I'm trying again with a fresh clone/build and will let you know how that goes. In the meantime my test on orion failed for ocean and the gridded wave post: /work/noaa/marine/jmeixner/p6stampede/test01/COMROOT/test01/logs/2013040100
It's my understanding that those currently work on orion, so hopefully those can be fixed before merging this PR in.

JessicaMeixner-NOAA

The re-test on hera went fine and @LydiaStefanova-NOAA checked the atm-post and the grib files were identical to what is in the current feature/coupled-crow so that's all fine. The orion issues (wave post and ocean post not running) need to be addressed before merging.

…e/hpcstack

WalterKolczynski-NOAA · 2021-03-02T16:23:49Z

@WalterKolczynski-NOAA I am waiting for the update for ocean post enabling it to run using hpc-stack.

reg2grb2 or something else? Because the PR migrating reg2grb2 to hpc-stack was merged last week (#273).

JianKuang-Intelsat · 2021-03-02T16:26:12Z

@WalterKolczynski-NOAA Yes. That's great. I will merge-in the updated coupled-crow and re-enable this PR later today. And invite you to review.

…e/hpcstack Conflicts: modulefiles/module_base.orion

WalterKolczynski-NOAA

Minor changes except for getting hpc-stack updated.

WalterKolczynski-NOAA · 2021-03-03T20:51:57Z

modulefiles/module_base.orion_bk

@@ -0,0 +1,68 @@
+#%Module######################################################################


Agree this should be removed

modulefiles/module_base.stampede

modulefiles/modulefile.reg2grb2.stampede

modulefiles/modulefile.ww3.stampede

sorc/build_ncep_post.sh

comment out contrib and noaatools in modulefile.ww3.stampede

WalterKolczynski-NOAA

Approved once the component updates are in.

…e/hpcstack

JianKuang-Intelsat · 2021-03-18T14:22:02Z

@WalterKolczynski-NOAA This PR is now up-to-date. Please take a test on Hera and Orion.

JessicaMeixner-NOAA · 2021-03-18T14:24:34Z

sorc/link_fv3gfs.sh

-  cd ${pwd}/ufs_utils.fd/sorc
-  ./link_fixdirs.sh $RUN_ENVIR $machine
-fi
+#if [ -d ${pwd}/ufs_utils.fd ]; then


Why is this commented out?

This is for debug purpose. I am restoring it now.

JessicaMeixner-NOAA · 2021-03-18T14:25:10Z

modulefiles/module_base.orion_bk

@@ -0,0 +1,68 @@
+#%Module######################################################################


This file needs to be removed before PR is merged.

JianKuang-Intelsat · 2021-03-18T15:05:15Z

@JessicaMeixner-NOAA @WalterKolczynski-NOAA I addressed Jessica's comments. Please take a test and approve this PR if ready. Thanks.

WalterKolczynski-NOAA · 2021-03-18T17:26:08Z

I'm on leave until Monday, so I might not get to it until then.

WalterKolczynski-NOAA · 2021-03-25T17:28:00Z

@JessicaMeixner-NOAA Are all your concerns addressed? I'd like to get this in.

JessicaMeixner-NOAA

@WalterKolczynski-NOAA yes, my concerns were corrected. Thanks @JianKuang-NOAA for fixing those.

I do have two questions though: 1. I thought this was waiting for the latest commit of ufs-weather-model which ported things to stampede: ufs-community/ufs-weather-model@37f0da7 and this PR does not update what is checked out to point to that. 2. the modules used in that PR are for hpc stack 1.1.0 which is inconsistent with this.

That being said I have no objection to this going in as it doesn't effect hera/orion/etc.

WalterKolczynski-NOAA · 2021-03-25T17:42:51Z

Agree, both of those should be addressed. Then this PR can go in.

JianKuang-Intelsat · 2021-03-25T17:55:25Z

@WalterKolczynski-NOAA Could you make these changes? I am focusing on EWOK project now, plus I have no Hera & WCOSS access which means someone other than me need to test it there anyway.

WalterKolczynski-NOAA · 2021-03-25T18:30:25Z

Since I have to take over, I'm going to merge this PR (since it doesn't break anything), then start from my own clone to update the remaining bits. That is easier than making sure I have all the right permissions to update Jian's branch.

WalterKolczynski-NOAA · 2021-03-25T18:33:36Z

Actually, I just hadn't added Stampede's ssh key to GitHub yet. Oh well, already done. There will be a new PR for the cleanup.

WalterKolczynski-NOAA · 2021-03-25T20:30:04Z

This PR should've tagged Issue #280

* If ufs.cpld.cpl.r.* files do not reproduce, try nccmp * Modify/add to run cpld 35d bmark tests * Fix nccmp implementation. Fix a bug for 35d tests * Use env var NCCMP * Add hera.intel to rt_35 and rt_wave_35d * Use which to find nccmp path. Apply nccmp to all files that fail cmp. Fix a bug in fv3_ccpp_wrtGauss_netcdf_parallel * Change compare method from nccmp to compare_ncfile.py * Netcdf compare changes on Hera; skip-ci * Netcdf compare changes on WCOSS Dell P3; skip-ci * Modify comopare_ncfile.py * Netcdf compare changes on wcoss cray * Implement Dusan's ecflow fix NOAA-EMC#273 * Move miniconda3 to emc.nemspara on Hera and Orion. Minor change in default_vars.sh

…itial, ocn -> wat, merra2 threading (NOAA-EMC#279) * changed .gitmodules to point to merra2 ccpp/physics * remove GFDL_atmos_cubed_sphere and ccpp-framework from .git module * remove IPD gfsphysics * Update .gitmodules and submodule pointer for ccpp-physics for code review and testing * Remove interstitial zorl composites * Update .gitmodules and submodule pointer fpor ccpp-physics for code review and testing * Remove or replace references to IPD in comments in atmos_model.F90 * Initialize Sfcprop%zorlx to clear_val instead of huge * Update submodule pointer for ccpp-physics * Rename Fortran variables and CCPP standard names / long names of surface composites from ocean to water * Rename Sfcprop%zorlw to Sfcprop%zorlwav * Rename Sfcprop%zorlo to Sfcprop%zorlw * update submodule pointer for ccpp-physics * Revert change to .gitmodules and update submodule pointer for ccpp-physics Co-authored-by: anning.cheng <anning.cheng@noaa.gov>

Jian Kuang and others added 16 commits September 18, 2020 09:56

additional files for stampede platform adaptation

1806ce2

Orion hpc-stack support modification in machine-setup.sh

c4bbfa5

new platform file for stampede2;

08e49d9

updated platform adaptation routine for orion

Merge branch 'feature/hpcstack' of github.com:NOAA-EMC/global-workflo…

76d0f0e

…w into feature/hpcstack

update orion platform adaptation routine

3adfbda

This will help eliminate ambiguity in the future

Merge remote-tracking branch 'origin/feature/coupled-crow' into featu…

cc0b102

…re/hpcstack

stampede platform

a10133e

add COMROOT into config.base, avoid setpdy.sh failure on stampede2

010f3d1

Why not happening on Orion? The Orion shell keep operates even if setpdy.sh encountered this error (non-fatal for Orion); But it works differently on stampede2

link and case file update for stampede2

3ee525b

bugfix for Orion

1d0e251

update env file for stampede platform, using ibrun for forecast jobs.

2fb2404

update ncep_post to tag upp_v10.0.3 to use hpcstack

3ee689b

Merge remote-tracking branch 'trunk/feature/coupled-crow' into featur…

c0592ca

…e/hpcstack Conflicts: sorc/build_fv3_coupled.sh sorc/checkout.sh sorc/link_fv3gfs.sh sorc/machine-setup.sh ush/load_fv3gfs_modules.sh

add wgrib2 to modulefile

fe6d73b

point to updated hpcstack path on Orion

5cc72ef

update hpc-stack path in machine-setup

430e3da

JianKuang-Intelsat requested a review from JessicaMeixner-NOAA February 10, 2021 16:03

JessicaMeixner-NOAA reviewed Feb 10, 2021

View reviewed changes

update fix file path to v6

c04b55a

remove excess files

Merge remote-tracking branch 'trunk/feature/coupled-crow' into featur…

6872f60

…e/hpcstack

JessicaMeixner-NOAA reviewed Feb 10, 2021

View reviewed changes

ush/load_fv3gfs_modules.sh Outdated Show resolved Hide resolved

comment out debug setting

bb82fa0

JessicaMeixner-NOAA suggested changes Feb 12, 2021

View reviewed changes

Merge remote-tracking branch 'trunk/feature/coupled-crow' into featur…

bf006fe

…e/hpcstack

Merge remote-tracking branch 'trunk/feature/coupled-crow' into featur…

2a92e9e

…e/hpcstack Conflicts: modulefiles/module_base.orion

WalterKolczynski-NOAA requested changes Mar 3, 2021

View reviewed changes

jian.kuang and others added 5 commits March 4, 2021 07:17

fix comment mismatch

fb0cbc5

move "git submodule" to checkout script

ae37892

comment out contrib and noaatools in modulefile.ww3.stampede

build option add

cc285f9

update hpc-stack path

397ba32

link script modification,

6878952

WalterKolczynski-NOAA approved these changes Mar 16, 2021

View reviewed changes

WalterKolczynski-NOAA requested changes Mar 16, 2021

View reviewed changes

Merge remote-tracking branch 'trunk/feature/coupled-crow' into featur…

9269e13

…e/hpcstack

JessicaMeixner-NOAA suggested changes Mar 18, 2021

View reviewed changes

Jian Kuang added 2 commits March 18, 2021 09:26

remove _bk file

e820d21

restore comment session

9a0df2b

WalterKolczynski-NOAA approved these changes Mar 22, 2021

View reviewed changes

JessicaMeixner-NOAA approved these changes Mar 25, 2021

View reviewed changes

WalterKolczynski-NOAA merged commit d2662ca into NOAA-EMC:feature/coupled-crow Mar 25, 2021

JessicaMeixner-NOAA mentioned this pull request Mar 26, 2021

feature/coupled-crow installation broken on stampede2 #308

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enable feature/coupled-crow to work on stampede2 #261

enable feature/coupled-crow to work on stampede2 #261

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA left a comment

JessicaMeixner-NOAA Feb 10, 2021

JianKuang-Intelsat Feb 10, 2021

JessicaMeixner-NOAA Feb 10, 2021

WalterKolczynski-NOAA Mar 3, 2021

JessicaMeixner-NOAA Mar 18, 2021

JessicaMeixner-NOAA Feb 10, 2021

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 10, 2021

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 10, 2021

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 11, 2021

JessicaMeixner-NOAA left a comment

WalterKolczynski-NOAA commented Mar 2, 2021 •

edited

Loading

JianKuang-Intelsat commented Mar 2, 2021

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Mar 3, 2021

WalterKolczynski-NOAA left a comment

JianKuang-Intelsat commented Mar 18, 2021

JessicaMeixner-NOAA Mar 18, 2021

JianKuang-Intelsat Mar 18, 2021

JessicaMeixner-NOAA Mar 18, 2021

JianKuang-Intelsat commented Mar 18, 2021

WalterKolczynski-NOAA commented Mar 18, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

JessicaMeixner-NOAA left a comment

WalterKolczynski-NOAA commented Mar 25, 2021

JianKuang-Intelsat commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

		@@ -0,0 +1,68 @@
		#%Module######################################################################

		@@ -0,0 +1,218 @@
		# Create a test function for sh vs. bash detection. The name is

enable feature/coupled-crow to work on stampede2 #261

enable feature/coupled-crow to work on stampede2 #261

Conversation

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 10, 2021

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 10, 2021

JianKuang-Intelsat commented Feb 10, 2021

JessicaMeixner-NOAA commented Feb 11, 2021

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Mar 2, 2021 • edited Loading

JianKuang-Intelsat commented Mar 2, 2021

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

JianKuang-Intelsat commented Mar 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JianKuang-Intelsat commented Mar 18, 2021

WalterKolczynski-NOAA commented Mar 18, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

JessicaMeixner-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA commented Mar 25, 2021

JianKuang-Intelsat commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 25, 2021

WalterKolczynski-NOAA commented Mar 2, 2021 •

edited

Loading