part of cam6_3_056: PUMAS GPU regression test suite #577

sjsprecious · 2022-04-20T16:48:46Z

This PR introduces a new regression test suite for the GPU-enabled PUMAS codes, which works on Casper.

There is no source code change here but just a few xml files. It also requires the updated CICE5/CICE6/CIME/ccs_config modules to work properly.

This regression test suite includes five different ERP tests:

CAM run with MG2, default namelist
CAM run with MG3, default namelist
CAM run with MG3, change the MG logical variables from their default values to the opposite values
CAM run with MG3, default namelist, different PCOLS values
CAM run with MG3, default namelist, different number of GPUs per node

Previously I made some GPU changes and somehow the CAM run could finish but return wrong results. The latter two new tests are able to detect those unexpected NBFB changes quickly before doing an ECT test and therefore I add them to the test suite.

To generate a baseline for this GPU regression test suite on Casper, use the following commands:

cd /path_to_CAM_main_dir/test/system
module load python/3.7.9
env BL_TESTDIR='' CAM_ACCOUNT=YOUR_PROJECT_ID CAM_FC=nvhpc-gpu CIME_MACHINE=casper ./test_driver.sh --cesm casper_gpu --baseline-dir /path_to_save_baseline -f

To perform a GPU regression test against the baseline generated above, use the following commands:

env BL_TESTDIR=/path_to_save_baseline CAM_ACCOUNT=YOUR_PROJECT_ID CAM_FC=nvhpc-gpu CIME_MACHINE=casper ./test_driver.sh --cesm casper_gpu --no-baseline -f

The status of the regression test could be viewed through the following commands:

cd /glade/scratch/$user/casper_gpu
./cs.status.casper_gpu_nvhpc-gpu_xxxx, where xxxx is the time stamp for this test run

Ideally we should observe PASS for all the output from the command above. However, I saw FAIL for the COMPARE_base_rest and there were 4 different fields in the cpl restart file. I later ran the ERP test for the PUMAS CPU codes on Casper and got the same error. However, if I switched the ERP test to ERS test and re-ran it with PUMAS GPU codes, there was no FAIL in the output. Thus I think the error message here is not caused by the PUMAS GPU codes but I do not know how to fix this issue.

Fix #512.

Update cam_development branch

modified: Externals.cfg

modified: cime_config/testdefs/testlist_cam.xml new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg2_default/shell_commands renamed: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_casper/user_nl_cam -> cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg2_default/user_nl_cam renamed: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_casper/user_nl_clm -> cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg2_default/user_nl_clm renamed: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_casper/shell_commands -> cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_default/shell_commands new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_default/user_nl_cam new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_default/user_nl_clm new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_nondefault/shell_commands new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_nondefault/user_nl_cam new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_nondefault/user_nl_clm new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_pcols1536/shell_commands new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_pcols1536/user_nl_cam new file: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_pcols1536/user_nl_clm

modified: test/system/test_driver.sh

gold2718

Couple of comments but otherwise, looks good!

gold2718 · 2022-04-24T18:57:26Z

cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg2_default/shell_commands

+./xmlchange NTASKS_ATM=36
+./xmlchange NTHRDS_ATM=1
+./xmlchange ROOTPE_ATM='0'
+./xmlchange NTASKS_LND=36
+./xmlchange NTHRDS_LND=1
+./xmlchange ROOTPE_LND='0'
+./xmlchange NTASKS_ROF=36
+./xmlchange NTHRDS_ROF=1
+./xmlchange ROOTPE_ROF='0'
+./xmlchange NTASKS_ICE=36
+./xmlchange NTHRDS_ICE=1
+./xmlchange ROOTPE_ICE='0'
+./xmlchange NTASKS_OCN=36
+./xmlchange NTHRDS_OCN=1
+./xmlchange ROOTPE_OCN='0'
+./xmlchange NTASKS_GLC=36
+./xmlchange NTHRDS_GLC=1
+./xmlchange ROOTPE_GLC='0'
+./xmlchange NTASKS_WAV=36
+./xmlchange NTHRDS_WAV=1
+./xmlchange ROOTPE_WAV='0'
+./xmlchange NTASKS_CPL=36
+./xmlchange NTHRDS_CPL=1
+./xmlchange ROOTPE_CPL='0'


For future reference, xmlchange recognizes these variables as categories so you can replicate this behavior with:

Suggested change

./xmlchange NTASKS_ATM=36

./xmlchange NTHRDS_ATM=1

./xmlchange ROOTPE_ATM='0'

./xmlchange NTASKS_LND=36

./xmlchange NTHRDS_LND=1

./xmlchange ROOTPE_LND='0'

./xmlchange NTASKS_ROF=36

./xmlchange NTHRDS_ROF=1

./xmlchange ROOTPE_ROF='0'

./xmlchange NTASKS_ICE=36

./xmlchange NTHRDS_ICE=1

./xmlchange ROOTPE_ICE='0'

./xmlchange NTASKS_OCN=36

./xmlchange NTHRDS_OCN=1

./xmlchange ROOTPE_OCN='0'

./xmlchange NTASKS_GLC=36

./xmlchange NTHRDS_GLC=1

./xmlchange ROOTPE_GLC='0'

./xmlchange NTASKS_WAV=36

./xmlchange NTHRDS_WAV=1

./xmlchange ROOTPE_WAV='0'

./xmlchange NTASKS_CPL=36

./xmlchange NTHRDS_CPL=1

./xmlchange ROOTPE_CPL='0'

./xmlchange NTASKS=36

./xmlchange NTHRDS=1

./xmlchange ROOTPE='0'

Thanks @gold2718. I just updated all the shell_commands files with your suggestions.

gold2718 · 2022-04-24T18:59:40Z

test/system/test_driver.sh

@@ -251,7 +251,7 @@ case $hostname in
    mach_workspace="/glade/scratch"

    # Check for CESM baseline directory
-    if [ -n "{$BL_TESTDIR}" ] && [ ! -d "${BL_TESTDIR}" ]; then
+    if [ -n "${BL_TESTDIR}" ] && [ ! -d "${BL_TESTDIR}" ]; then


Nice catch! I guess folks were not running this with a blank value for BL_TESTDIR.

Thanks! That is exactly how I find out this issue and I hope I am not the only one using an empty BL_TESTDIR.

gold2718

On second thought, I would like a chance to see your ChangeLog entry (can be before you have final test data).

modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg2_default/shell_commands modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_default/shell_commands modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_nondefault/shell_commands modified: cime_config/testdefs/testmods_dirs/cam/outfrq9s_mg3_pcols1536/shell_commands

nusbaume

Looks good to me! Just one possible typo.

nusbaume · 2022-04-28T16:00:27Z

test/system/test_driver.sh

@@ -465,7 +520,12 @@ if [ "${cesm_test_suite}" != "none" -a -n "${cesm_test_mach}" ]; then
    fi

    ## Setup CESM work directory
-    cesm_testdir=$mach_workspace/$LOGNAME/$test_id
+    if [ "${hostname:0:6}" == "casper" ] || [ "${hostname:0:5}" == "crhtc" ]; then
+       ## Would fail to compile on Casper with long foler name


Possibly a typo here (foler)?

Thanks @nusbaume for catching it. Just fix the typo!

peverwhee · 2022-04-30T00:54:50Z

this was brought in with #581

Katetc and others added 4 commits April 19, 2022 11:28

Merge pull request #8 from ESCOMP/cam_development

662d158

Update cam_development branch

Update CICE/CIME/CCS_CONFIG to make ERP run work properly

be19d5a

modified: Externals.cfg

Update the test_driver for Casper

8586311

modified: test/system/test_driver.sh

sjsprecious requested review from gold2718, cacraigucar, andrewgettelman and johnmauff April 20, 2022 16:48

sjsprecious assigned cacraigucar Apr 20, 2022

sjsprecious added enhancement New feature or request BFB bit for bit tag externals externals updating issue or PR labels Apr 20, 2022

sjsprecious marked this pull request as draft April 20, 2022 16:51

sjsprecious mentioned this pull request Apr 20, 2022

Add regression test suite for GPU-enabled PUMAS run on Casper ESCOMP/PUMAS#31

Closed

sjsprecious removed the BFB bit for bit tag label Apr 22, 2022

cacraigucar mentioned this pull request Apr 22, 2022

cam6_3_056: Update externals to support CICE6 #581

Merged

sjsprecious marked this pull request as ready for review April 23, 2022 03:27

gold2718 approved these changes Apr 24, 2022

View reviewed changes

gold2718 requested changes Apr 24, 2022

View reviewed changes

cacraigucar requested review from nusbaume and peverwhee April 26, 2022 15:47

cacraigucar approved these changes Apr 28, 2022

View reviewed changes

nusbaume approved these changes Apr 28, 2022

View reviewed changes

peverwhee approved these changes Apr 28, 2022

View reviewed changes

peverwhee pushed a commit to peverwhee/CAM that referenced this pull request Apr 28, 2022

add updates for ESCOMP#573, ESCOMP#577, ESCOMP#566

0cd1d89

sjsprecious removed request for andrewgettelman and johnmauff April 28, 2022 16:31

fix a typo in the comment

10830e9

cacraigucar changed the title ~~PUMAS GPU regression test suite~~ part of cam6_3_056: PUMAS GPU regression test suite Apr 29, 2022

peverwhee closed this Apr 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

part of cam6_3_056: PUMAS GPU regression test suite #577

part of cam6_3_056: PUMAS GPU regression test suite #577

sjsprecious commented Apr 20, 2022 •

edited

Loading

gold2718 left a comment

gold2718 Apr 24, 2022

sjsprecious Apr 24, 2022

gold2718 Apr 24, 2022

sjsprecious Apr 24, 2022

gold2718 left a comment

nusbaume left a comment

nusbaume Apr 28, 2022

sjsprecious Apr 28, 2022

peverwhee commented Apr 30, 2022

part of cam6_3_056: PUMAS GPU regression test suite #577

part of cam6_3_056: PUMAS GPU regression test suite #577

Conversation

sjsprecious commented Apr 20, 2022 • edited Loading

gold2718 left a comment

Choose a reason for hiding this comment

gold2718 Apr 24, 2022

Choose a reason for hiding this comment

sjsprecious Apr 24, 2022

Choose a reason for hiding this comment

gold2718 Apr 24, 2022

Choose a reason for hiding this comment

sjsprecious Apr 24, 2022

Choose a reason for hiding this comment

gold2718 left a comment

Choose a reason for hiding this comment

nusbaume left a comment

Choose a reason for hiding this comment

nusbaume Apr 28, 2022

Choose a reason for hiding this comment

sjsprecious Apr 28, 2022

Choose a reason for hiding this comment

peverwhee commented Apr 30, 2022

sjsprecious commented Apr 20, 2022 •

edited

Loading