Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Incorporate csawMG update and pass both intel and GNU RT tests #2180

Merged
merged 21 commits into from
Mar 14, 2024
Merged

Incorporate csawMG update and pass both intel and GNU RT tests #2180

merged 21 commits into from
Mar 14, 2024

Conversation

grantfirl
Copy link
Collaborator

@grantfirl grantfirl commented Mar 7, 2024 •
edited
Loading

Commit Queue Requirements:

  • Fill out all sections of this template.
  • All sub component pull requests have been reviewed by their code managers.
  • Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
  • Commit 'test_changes.list' from previous step

Description:

This is work from @AnningCheng-NOAA. This replaces #2130 for convenience of testing/updating/merging.

CSAW missed a major update and cannot pass RTs for GNU compiler. Both are fixed in this PR

Commit Message:

* UFSWM - Update the CS convection scheme
  * AQM - 
  * CDEPS - 
  * CICE - 
  * CMEPS - 
  * CMakeModules - 
  * FV3 - Update the CS convection scheme
    * ccpp-physics - Update the CS convection scheme
    * atmos_cubed_sphere - 
  * GOCART - 
  * HYCOM - 
  * MOM6 - 
  * NOAHMP - 
  * WW3 - 
  * stochastic_physics -

Priority:

  • Normal

Git Tracking

UFSWM:

Sub component Pull Requests:

UFSWM Blocking Dependencies:

  • None

Changes

Regression Test Changes (Please commit test_changes.list):

  • PR Adds New Tests/Baselines.

Input data Changes:

  • None.

Library Changes/Upgrades:

  • No Updates

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Derecho
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
  • opnReqTest (complete task if unnecessary)

@grantfirl grantfirl mentioned this pull request Mar 7, 2024
Closed
41 tasks
@grantfirl
Copy link
Collaborator Author

I'm running RTs on Hera now. I'll upload logs when finished.

@zach1221
Copy link
Collaborator

@grantfirl I think we're ready to begin testing this PR. Can you please sync up your branch?
@FernandoAndrade-NOAA @jkbk2004

@grantfirl
Copy link
Collaborator Author

@grantfirl I think we're ready to begin testing this PR. Can you please sync up your branch? @FernandoAndrade-NOAA @jkbk2004

@zach1221 Finished syncing.

@zach1221 zach1221 added Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. labels Mar 12, 2024
@zach1221
Copy link
Collaborator

Fernando and I are getting failures with the debug cases on Hera and orion, control_csawmg_debug and control_csawmgt_debug, both intel and gnu. Err log has: "subscript #2 of the array SIGMA has value 128 which is greater than the upper bound of 127"

/work/noaa/stmp/zshrader/stmp/zshrader/FV3_RT/rt_279127/control_csawmgt_debug_intel

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Mar 12, 2024 via email

@zach1221
Copy link
Collaborator

@grantfirl are you able to make the changes mentioned above?

@BrianCurtis-NOAA
Copy link
Collaborator

WCOSS2:

Found Python: /usr/bin/python3.6
Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_GFS_v16_csawmg,FV3_GFS_v16_ras ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_csawmg_intel/build_fv3_csawmg_intel/FV3/ccpp/ccp
p_prebuild.{out,err}

and

Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_HRRR,FV3_GFS_v16,FV3_GFS_v16_csawmg,FV3_GFS_v16_ras,FV3_GFS_v17_p8,FV3_GFS_v15_thompson_mynn_lam3km,FV3_RAP,FV3_RAP_unified_ugwp,FV3_RAP_cires_ugwp,FV3_RAP_flake,FV3_RAP_clm_lake,FV3_RAP_noah,FV3_RAP_sfcdiff,FV3_RAP_noah_sfcdiff_cires_ugwp,FV3_RRFS_v1beta,FV3_HRRR_c3,FV3_HRRR_gf,FV3_global_nest_v1 ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_atm_debug_dyn32_intel/build_fv3_atm_debug_dyn32_intel/FV3/ccpp/ccpp_prebuild.{out,err}

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Mar 13, 2024 via email

@jkbk2004
Copy link
Collaborator

"vertical_interface_dimension" should have been used instead of "vertical_layer_dimension" in cs_conv.meta, cs_conv_post.meta, and FV3/ccpp/data/CCPP_typedefs.meta as : [sigma] (sigmatot in cs_conv_post.meta and FV3/ccpp/data/CCPP_typedefs.meta) standard_name = convective_updraft_area_fraction_at_model_interfaces long_name = convective updraft area fraction at model interfaces units = frac dimensions = (horizontal_loop_extent,vertical_interface_dimension) The err should be fixed now.
…
On Tue, Mar 12, 2024 at 10:04 PM Brian Curtis @.> wrote: WCOSS2: Found Python: /usr/bin/python3.6 Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_GFS_v16_csawmg,FV3_GFS_v16_ras ... CMake Error at FV3/ccpp/CMakeLists.txt:40 (message): An error occured while running ccpp_prebuild.py, check /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_csawmg_intel/build_fv3_csawmg_intel/FV3/ccpp/ccp p_prebuild.{out,err} and Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_HRRR,FV3_GFS_v16,FV3_GFS_v16_csawmg,FV3_GFS_v16_ras,FV3_GFS_v17_p8,FV3_GFS_v15_thompson_mynn_lam3km,FV3_RAP,FV3_RAP_unified_ugwp,FV3_RAP_cires_ugwp,FV3_RAP_flake,FV3_RAP_clm_lake,FV3_RAP_noah,FV3_RAP_sfcdiff,FV3_RAP_noah_sfcdiff_cires_ugwp,FV3_RRFS_v1beta,FV3_HRRR_c3,FV3_HRRR_gf,FV3_global_nest_v1 ... CMake Error at FV3/ccpp/CMakeLists.txt:40 (message): An error occured while running ccpp_prebuild.py, check /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_atm_debug_dyn32_intel/build_fv3_atm_debug_dyn32_intel/FV3/ccpp/ccpp_prebuild.{out,err} — Reply to this email directly, view it on GitHub <#2180 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALQPMIOGJXZOJTGSK4YTKIDYX6X3TAVCNFSM6AAAAABELUFRCKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJTGEYTGNJVG4 . You are receiving this because you were mentioned.Message ID: @.>

@AnningCheng-NOAA You mean a fix is needed (replace vertical_layer_dimension with vertical_interface_dimension) at https://github.com/NOAA-EMC/fv3atm/blob/062f7958ccd9bd97b8737432f75b06ec734777a7/ccpp/data/CCPP_typedefs.meta#L1955 ? @grantfirl @zach1221 @FernandoAndrade-NOAA FYI

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Mar 13, 2024 via email

@zach1221
Copy link
Collaborator

@grantfirl following up here, could you please make the additional changes or provide Jong/myself permission to commit directly to the repo?

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Mar 13, 2024 via email

@grantfirl
Copy link
Collaborator Author

@grantfirl following up here, could you please make the additional changes or provide Jong/myself permission to commit directly to the repo?

Yes, I'll make that change. Stand by.

@grantfirl
Copy link
Collaborator Author

@zach1221 @AnningCheng-NOAA @jkbk2004 The change has been made and tests should be ready to resume.

@AnningCheng-NOAA
Copy link
Contributor

AnningCheng-NOAA commented Mar 13, 2024 via email

@grantfirl
Copy link
Collaborator Author

WCOSS2:

Found Python: /usr/bin/python3.6
Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_GFS_v16_csawmg,FV3_GFS_v16_ras ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_csawmg_intel/build_fv3_csawmg_intel/FV3/ccpp/ccp
p_prebuild.{out,err}

and

Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_HRRR,FV3_GFS_v16,FV3_GFS_v16_csawmg,FV3_GFS_v16_ras,FV3_GFS_v17_p8,FV3_GFS_v15_thompson_mynn_lam3km,FV3_RAP,FV3_RAP_unified_ugwp,FV3_RAP_cires_ugwp,FV3_RAP_flake,FV3_RAP_clm_lake,FV3_RAP_noah,FV3_RAP_sfcdiff,FV3_RAP_noah_sfcdiff_cires_ugwp,FV3_RRFS_v1beta,FV3_HRRR_c3,FV3_HRRR_gf,FV3_global_nest_v1 ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_atm_debug_dyn32_intel/build_fv3_atm_debug_dyn32_intel/FV3/ccpp/ccpp_prebuild.{out,err}

@BrianCurtis-NOAA Could you post the error written out to ccpp_prebuild.err in those instances?

@grantfirl
Copy link
Collaborator Author

@zach1221 @jkbk2004 I also added you both as collaborators to my ccpp-physics, fv3atm, and ufs-weather-model forks so that you can add changes if necessary if I'm away from my machine or in meetings and there is something urgent.

@BrianCurtis-NOAA
Copy link
Collaborator

WCOSS2:

Found Python: /usr/bin/python3.6
Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_GFS_v16_csawmg,FV3_GFS_v16_ras ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_csawmg_intel/build_fv3_csawmg_intel/FV3/ccpp/ccp
p_prebuild.{out,err}

and

Calling CCPP code generator (ccpp_prebuild.py) for suites --suites=FV3_HRRR,FV3_GFS_v16,FV3_GFS_v16_csawmg,FV3_GFS_v16_ras,FV3_GFS_v17_p8,FV3_GFS_v15_thompson_mynn_lam3km,FV3_RAP,FV3_RAP_unified_ugwp,FV3_RAP_cires_ugwp,FV3_RAP_flake,FV3_RAP_clm_lake,FV3_RAP_noah,FV3_RAP_sfcdiff,FV3_RAP_noah_sfcdiff_cires_ugwp,FV3_RRFS_v1beta,FV3_HRRR_c3,FV3_HRRR_gf,FV3_global_nest_v1 ...
CMake Error at FV3/ccpp/CMakeLists.txt:40 (message):
  An error occured while running ccpp_prebuild.py, check
  /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_169315/compile_atm_debug_dyn32_intel/build_fv3_atm_debug_dyn32_intel/FV3/ccpp/ccpp_prebuild.{out,err}

@BrianCurtis-NOAA Could you post the error written out to ccpp_prebuild.err in those instances?

I know Anning addressed the issue that showed up there, but I will definitely gather that files output if I see ccpp_physics build issues in the future. Thanks.

@zach1221 zach1221 added gaea-RT and removed gaea-RT labels Mar 13, 2024
@zach1221 zach1221 added derecho-RT Run regression tests on Derecho gaea-RT and removed derecho-RT Run regression tests on Derecho gaea-RT labels Mar 14, 2024
@zach1221 zach1221 added gaea-RT hercules-RT Run Hera regression testing and removed hercules-RT Run Hera regression testing gaea-RT labels Mar 14, 2024
@zach1221
Copy link
Collaborator

@grantfirl testing is complete here, we can begin the merging process.

@zach1221 zach1221 mentioned this pull request Mar 14, 2024
Merged
@grantfirl
Copy link
Collaborator Author

@grantfirl testing is complete here, we can begin the merging process.

Thanks, @zach1221 . I've asked the other reviewers to review again via email, in case the GitHub emails get lost in their inboxes (which often happens if you watch many repos). Worst case scenario, if we can't get an approval, I can force a merge. It is mostly Anning's work and I just did some cleaning up, so I'm somewhat impartial enough to review.

@zach1221
Copy link
Collaborator

@grantfirl fv3atm pr is merged. hash: NOAA-EMC/fv3atm@21744e2

@zach1221 zach1221 merged commit e4961eb into ufs-community:develop Mar 14, 2024
@FernandoAndrade-NOAA FernandoAndrade-NOAA mentioned this pull request Mar 14, 2024
Merged
14 tasks
@grantfirl grantfirl mentioned this pull request Mar 29, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jkbk2004 jkbk2004 jkbk2004 approved these changes

@BrianCurtis-NOAA BrianCurtis-NOAA BrianCurtis-NOAA approved these changes

Assignees
No one assigned
Labels
Baseline Updates Current baselines will be updated. Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

ufs_noaacloud modulefile should be upgraded to spack-stack-1.5.1 Add csawmg debug test, add to rt_gnu.conf
7 participants
@grantfirl @zach1221 @AnningCheng-NOAA @BrianCurtis-NOAA @jkbk2004 @grant @FernandoAndrade-NOAA