Modify mpas framework to match mpasa in EW v2.1

[dazlich]

EW v2.1 broke the coupled model because of inconsistencies between the mpas framework in the atmosphere and in the ocean.

This PR restores the consistency between the two frameworks.

[dazlich]

@gdicker1 - what's the tag that will be assigned to this? And do we want this to get into the main branch as well?

[gdicker1]

I'm thinking this change (once merged) will be mpasfrwk-ew2.1.001. I agree this should be merged onto main ASAP maybe as a patch release for EarthWorks overall. I'm doing my tests now to approve.

[gdicker1]

For now though, shouldn't this PR go to the ew-develop branch? We can merge or fast-forward the change onto ew-main pretty easily.

[dazlich]

Did I not do that? My bad On Feb 20, 2024, at 11:54 AM, G. Dylan Dickerson ***@***.***> wrote: ** Caution: EXTERNAL Sender ** @gdicker1 requested changes on this pull request. For now though, shouldn't this PR go to the ew-develop branch? We can merge or fast-forward the change onto ew-main pretty easily. — Reply to this email directly, view it on GitHub<#6 (review)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADS3XWHG4OALLOFBS6E3ADLYUTWQFAVCNFSM6AAAAABDP3HEYWVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTQOJRGI3TMNRZGQ>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[dazlich]

I'm stumped figuring out how to change this from ew-main to ew-develop

[gdicker1]

I'm stumped figuring out how to change this from ew-main to ew-develop

I can change it then!

[gdicker1]

FYI: I can do this by clicking the "edit" button next to the PR title, I then see the text below the title change from "... merge 14 commits to ew-main ..." to "... merge 14 commits to base:ew-main ...". I can click on base:ew-main to get a drop down menu and switch to ew-develop. Then a "change base" pop-up appears; clicking the green button in the pop-up finishes the base change.

Changing the other branch is a different deal! (Basically create a new PR or do some rebasing+force-pushing of that branch)

[dazlich]

I see it now, thanks! On Feb 20, 2024, at 1:58 PM, G. Dylan Dickerson ***@***.***> wrote: ** Caution: EXTERNAL Sender ** FYI: I can do this by clicking the "edit" button next to the PR title, I then see the text below the title change from "... merge 14 commits to ew-main ..." to "... merge 14 commits to base:ew-main ...". I can click on base:ew-main to get a drop down menu and switch to ew-develop. Then a "change base" pop-up appears; clicking the green button in the pop-up finishes the base change. Changing the other branch is a different deal! (Basically create a new PR or do some rebasing+force-pushing of that branch) — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADS3XWEI7H2D3CE77R6RM6LYUUE63AVCNFSM6AAAAABDP3HEYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGA4DIMRRGA>. You are receiving this because you authored the thread.Message ID: ***@***.***>

[gdicker1]

It looks like this allows things to work on the CPU side, but breaks specifically GPU FullyCoupled builds still fail.

---

From "/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/cesm.bldlog.240220-113659"

nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_3d_real_acc_8350_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_3d_real_acc_8308_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_2d_real_acc_7433_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_2d_real_acc_7393_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_1d_real_acc_6519_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink error   : Multiple definition of 'mpas_dmpar_mpas_dmpar_exch_halo_1d_real_acc_6481_gpu' in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libocn.a:mpas_dmpar.o', first defined in '/glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/lib/libice.a:mpas_dmpar.o'
nvlink fatal   : merge_elf failed
pgacclnk: child process exit status 2: /glade/u/apps/common/23.04/spack/opt/spack/nvhpc/23.5/Linux_x86_64/23.5/compilers/bin/tools/nvdd
gmake: *** [/glade/work/gdicker/EarthWorks/EWRepo_PullRequests/2024Feb20_MPASfrwk6/cases/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/Tools/Makefile:978: /glade/derecho/scratch/gdicker/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc/bld/cesm.exe] Error 2

This was created with the following command:

> /glade/work/gdicker/EarthWorks/EWRepo_PullRequests/2024Feb20_MPASfrwk6/EarthWorks/cime/scripts/create_newcase --case /glade/work/gdicker/EarthWorks/EWRepo_PullRequests/2024Feb20_MPASfrwk6/cases/2024Feb20_113609_EWMTesting_FullyCoupled.mpasa120.derecho.nvhpc --project UCSU0085 --compiler nvhpc --res mpasa120_oQU120 --compset 2000_CAM60_CLM50%SP_MPASSI_MPASO_SROF_SGLC_SWAV --driver nuopc --run-unsupported -i /glade/campaign/univ/ucsu0085/inputdata --ngpus-per-node 4 --gpu-type a100 --gpu-offload openacc --pecount 128

Further record of 120km and 60km GPU FullyCoupled case creation/setup on Derecho: "/glade/work/gdicker/EarthWorks/EWRepo_PullRequests/2024Feb20_MPASfrwk6/EarthWorks/tools/Earthworks_scripts/EWv2_CreateBuildRun/log.setup.2024Feb20_113609_EWMTesting.FullyCoupled.txt"
Further record of attempt to build cases: "/glade/work/gdicker/EarthWorks/EWRepo_PullRequests/2024Feb20_MPASfrwk6/EarthWorks/tools/Earthworks_scripts/EWv2_CreateBuildRun/log.buildrun.2024Feb20_113609_EWMTesting.FullyCoupled.txt"

[gdicker1]

Sorry @dazlich, this was a premature approval, when I was trying to hit other buttons. Mind if I hit the "revert"?

[dazlich]

github is nothing if not full of buttons On Feb 20, 2024, at 2:28 PM, G. Dylan Dickerson ***@***.***> wrote: ** Caution: EXTERNAL Sender ** Sorry @dazlich<https://github.com/dazlich>, this was a premature approval, when I was trying to hit other buttons. — Reply to this email directly, view it on GitHub<#6 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADS3XWFTAONVWFUPWUTPTNLYUUIQLAVCNFSM6AAAAABDP3HEYWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVGEZTGOJRGM>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[gdicker1]

Welp now I'm stumped on reverting this without another PR. We'll keep these changes and I'll link the error in this comment back to an issue page, since it shows we still have GPU issues.