fix angles on Bu for use by CICE #922

DeniseWorthen · 2024-03-15T18:36:10Z

Remove find_anq routine and instead calculate Bu angles using the Ct angle on the opposite side of the seam.
Add angchk to verify CICE internally calculated Ct angle against both MOM6 and CICE model output
Remove resetting of longitudes to 0:360
Add tripole:tripole bilinear mapping for creation of CICE IC downscaling and add required Bu<->Ct mapping on tripole grid
Replace existing unit test on Bu angles by providing 5x5 blocks of input, finding anglet, angle and angchk and comparing known (good) output for MOM6 and CICE6.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules and WCOSS2). Done using 9caec86.
Compile branch on Hera using GNU. Done using 5da3dc6.
Compile branch in 'Debug' mode on WCOSS2. Done using 5da3dc6.
Run unit tests locally on any Tier 1 machine. Done on Hercules using 5da3dc6.
Run relevant consistency tests locally on all Tier 1 machine.

This PR will change baselines for all resolutions.

The tripole master file at each resolution will now contain two additional fields, the Bu angle and the angchk array.
The previous code added an increment of 360.00 in longitude to the MOM6 longitudes, which have values from -300 to 60. This was done only for convenience, but it was found to produce a slight anomaly in the right-hand pole longitude value.
Additional mapping files are generated in order to address the requirement of downscaling CICE ICs from higher to lower resolutions.

A test baseline was created using a feature branch containing only the additional angle variables and the removal of the 360deg increment. The reg tests were run for the feature/fixangleBu branch against this test baseline. The following files did not compare due to the change in the Bu angle fields.

20:Comparing grid_cice_NEMS_mx025.nc........NOT OK
38:Comparing tripole.mx025.nc........NOT OK
58:Comparing grid_cice_NEMS_mx050.nc........NOT OK
74:Comparing tripole.mx050.nc........NOT OK
94:Comparing grid_cice_NEMS_mx100.nc........NOT OK
107:Comparing tripole.mx100.nc........NOT OK
128:Comparing grid_cice_NEMS_mx500.nc........NOT OK
138:Comparing tripole.mx500.nc........NOT OK

DOCUMENTATION:

Documentation has been updated. The doxygen was generated on Cactus using 5da3dc6. There we no warnings. It was visualized on rzdm and looked good.

ISSUE:

fixes rotation angles on corner grid points are wrong in cpld_gridgen #921

* remove find_anq routine and instead calculate Bu angles using the Ct angle on the opposite side of the seam. * add angchk to verify CICE internally calculated Ct angle against both MOM6 and CICE model output * remove resetting of longitudes to 0:360 * add tripole:tripole bilinear mapping for creation of CICE IC downscaling and add required Bu<->Ct mapping on tripole grid

DeniseWorthen · 2024-03-18T21:46:05Z

@GeorgeGayno-NOAA I have to rebuild my unit test since I'v re-written the routine it tested.

GeorgeGayno-NOAA · 2024-03-22T13:35:18Z

@DeniseWorthen is there anything I can do to help with this PR?

DeniseWorthen · 2024-03-22T13:45:07Z

No, thanks. I'm just trying to get a new unit test running since I've basically replaced the code that the previous unit test was using. I've almost got it working now.

* tripole, grid_cice and

* grid cice and tripole files are not b4b

* need to allow for non-full dimensioned arrays being sent to angles routines

* only cice grid files, tripole files change

* need to clean up messaging and angchk test

* 5x5 blocks of grid points are used to calc angle variables and checked against known output from mom6 and cice6 history files

* send array bounds to angles routines. This allows non-calculated values within the unit test to return as values 0.0 instead of non-sensical values

* baselines pass against a bl created from addanglevar branch * only cice grid and tripole files differ

DeniseWorthen · 2024-03-25T17:57:58Z

@GeorgeGayno-NOAA This PR will require all new baselines for all platforms, but it is ready for review.

GeorgeGayno-NOAA · 2024-03-26T13:30:26Z

Just a minor comment about an unused variable:

/work2/noaa/da/ggayno/save/UFS_UTILS.denise/sorc/cpld_gridgen.fd/gen_fixgrid.F90(52): remark #7712: This variable has not been used.   [JJ]
  integer :: ii,jj
----------------^

tests/cpld_gridgen/ftst_find_angq.F90

GeorgeGayno-NOAA · 2024-03-26T16:58:22Z

@DeniseWorthen do you have a new set of baseline files or should I create them from your branch?

DeniseWorthen · 2024-03-26T17:35:55Z

I need to generate them. I probably have them on hera, but I'd prefer to regenerate.

DeniseWorthen · 2024-03-28T18:25:24Z

@GeorgeGayno-NOAA I've generated the following baselines

WCOSS2: /lfs/h2/emc/stmp/denise.worthen/CPLD_GRIDGEN/BASELINE
HERA: /scratch1/NCEPDEV/stmp4/Denise.Worthen/CPLD_GRIDGEN/BASELINE
HERCULES: /work/noaa/stmp/dworthen/CPLD_GRIDGEN/BASELINE
ORION: /work/noaa/stmp/dworthen/CPLD_GRIDGEN/BASELINE

For Jet, I got a failure the first time which I seemed like a system error? I'm re-running Jet now.

DeniseWorthen · 2024-03-28T18:41:36Z

Baselines on Jet were successful on 2nd try

/lfs4/HFIP/h-nems/Denise.Worthen/CPLD_GRIDGEN/BASELINE/

GeorgeGayno-NOAA · 2024-03-28T18:55:11Z

Baselines on Jet were successful on 2nd try

/lfs4/HFIP/h-nems/Denise.Worthen/CPLD_GRIDGEN/BASELINE/

Great. I will host the files in the official location, then merge.

* gnu and gnu debug triggered several errors previously not caught using intel and intel debug

DeniseWorthen · 2024-04-02T22:53:44Z

@GeorgeGayno-NOAA I've pushed a couple of minor fixes from testing w/ GNU on hercules. I've also created a new baseline on hercules at /work/noaa/stmp/dworthen/CPLD_GRIDGEN/20240402.BASELINE. I've run against that baseline twice and was able to reproduce.

I spot-checked the mx100 and mx050 against the ufs_utils.hercules/reg_tests/cpld_gridgen files and I do see some differences, but small. Could you see if you have any trouble reproducing the baseline I created?

GeorgeGayno-NOAA · 2024-04-03T12:48:24Z

@GeorgeGayno-NOAA I've pushed a couple of minor fixes from testing w/ GNU on hercules. I've also created a new baseline on hercules at /work/noaa/stmp/dworthen/CPLD_GRIDGEN/20240402.BASELINE. I've run against that baseline twice and was able to reproduce.

I spot-checked the mx100 and mx050 against the ufs_utils.hercules/reg_tests/cpld_gridgen files and I do see some differences, but small. Could you see if you have any trouble reproducing the baseline I created?

I am able to reproduce your new baseline! Please recreate your baselines on Orion, Jet, Hera and WCOSS2. Then, I can do a final test., then merge.

DeniseWorthen · 2024-04-03T12:53:56Z

Great. I'm doing one final consistency check and then I'll start the remaining platforms.

DeniseWorthen · 2024-04-03T14:28:36Z

@GeorgeGayno-NOAA I've satisfied myself that the angchk, anglet and angle variables for the new baselines are as expected compared to the current develop branch. So I'll go ahead and start generating on the other platforms.

DeniseWorthen · 2024-04-03T15:34:49Z

Hera baseline : /scratch1/NCEPDEV/stmp4/Denise.Worthen/CPLD_GRIDGEN/BASELINE
Orion baseline : /work/noaa/stmp/dworthen/CPLD_GRIDGEN/BASELINE

DeniseWorthen · 2024-04-03T15:40:00Z

JET: /lfs4/HFIP/h-nems/Denise.Worthen/CPLD_GRIDGEN/BASELINE
WCOSS2: /lfs/h2/emc/stmp/denise.worthen/CPLD_GRIDGEN/BASELINE/

GeorgeGayno-NOAA · 2024-04-05T19:26:43Z

Using 993c670 I am able to reproduce your baselines on Hercules, WCOSS2 and Hera. However, I am still getting those annoying small differences on Jet and Orion. And on Jet, I get occasional seg faults.

Location of your branch:

Jet: /lfs4/HFIP/emcda/George.Gayno/ufs_utils.git/UFS_UTILS.denise/reg_tests/cpld_gridgen
Orion: /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.denise/reg_tests/cpld_gridgen

I moved your baselines here:

Jet: /lfs4/HFIP/hfv3gfs/emc.nemspara/role.ufsutils/ufs_utils/reg_tests/cpld_gridgen/baseline_data/*.new
Orion: /work/noaa/nems/role-nems/ufs_utils/reg_tests/cpld_gridgen/baseline_data/*new

DeniseWorthen · 2024-04-05T20:16:17Z

@GeorgeGayno-NOAA This is puzzling. Let me do some work on Orion and Jet and see if I can reproduce on those platforms.

DeniseWorthen · 2024-04-08T16:08:34Z

@GeorgeGayno-NOAA I'm seeing a lot of intermittent issues. For example, on Orion even if the baseline for one resolution passes, the following job, at the next lower resolution, is never submitted. It seems to disappear into the ether. But by launching the rt.sh again, the job succeeds. On Jet, the first time I ran this morning the job hung.

Right now, I'm testing bumping the memory up on Jet and increasing the wall clock time everywhere. I don't see that either of these should really be necessary, but at least on Jet I've been able to run against the baseline several times now. Is this all down to Rocky8? I'll keep you posted.

DeniseWorthen · 2024-04-08T17:08:00Z

@GeorgeGayno-NOAA Just to be sure, the mom6 fix files used right now are set in the cpld_gridgen.sh rt.sh script. For example,

export MOM6_FIXDIR=/scratch1/NCEPDEV/global/glopara/fix/mom6/20220805

There is a second directory there (20231219) but those files should not be any different. I don't think that is causing any issue, but I'm wondering if I should update those to the later directory.

DeniseWorthen · 2024-04-08T18:02:52Z

On Jet, I have been able to reproduce the baseline at least 4 times using

diff --git a/reg_tests/cpld_gridgen/rt.sh b/reg_tests/cpld_gridgen/rt.sh
index 70d11c04..63b0fcc9 100755
--- a/reg_tests/cpld_gridgen/rt.sh
+++ b/reg_tests/cpld_gridgen/rt.sh
@@ -108,9 +108,9 @@ TESTS_FILE="$PATHRT/rt.conf"
 export TEST_NAME=

 # for C3072 on hera, use WLCLK=60 and MEM="--exclusive"
-WLCLK_dflt=20
+WLCLK_dflt=35
 export WLCLK=$WLCLK_dflt
-MEM_dflt="--mem=12g"
+MEM_dflt="--mem=16g"
 export MEM=$MEM_dflt

 cd $PATHRT
@@ -170,7 +170,7 @@ elif [[ $target = jet ]]; then
     NCCMP=nccmp
     PARTITION=xjet
     ulimit -s unlimited
-    WLCLK=25
+    WLCLK=35
     SBATCH_COMMAND="./cpld_gridgen.sh"
 fi

failures

DeniseWorthen · 2024-04-08T19:10:49Z

I am going to see if I can reproduce everywhere using the bump in memory and time.

DeniseWorthen · 2024-04-08T19:55:44Z

@GeorgeGayno-NOAA On WCOSS2, I don't think the new baselines haven't been staged correctly. They look to be pointing to the Dec 20 baselines.

lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 025 -> 025.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 050 -> 050.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 100 -> 100.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:35 500 -> 500.8639547

GeorgeGayno-NOAA · 2024-04-08T20:49:41Z

@GeorgeGayno-NOAA On WCOSS2, I don't think the new baselines haven't been staged correctly. They look to be pointing to the Dec 20 baselines.
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 025 -> 025.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 050 -> 050.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:34 100 -> 100.8639547
lrwxrwxrwx 1 emc.nems nems   11 Apr  5 19:35 500 -> 500.8639547

Try it now.

GeorgeGayno-NOAA · 2024-04-10T13:21:07Z

I just retested your branch (9caec86) on Jet and Orion. I still get these annoying differences. The location of your checked out branch is:

Jet: /lfs4/HFIP/emcda/George.Gayno/ufs_utils.git/UFS_UTILS.denise/reg_tests/cpld_gridgen
Orion: /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.denise/reg_tests/cpld_gridgen

I would like to point to your executable from my work space. And I wonder if we are loading different libraries at runtime.

DeniseWorthen · 2024-04-10T14:01:57Z

I've been running a sanity check this morning everywhere using a clean checkout of the branch. I've gotten jet,orion and hercules and hera to pass. I haven't started WCOSS2 yet.

On orion, my latest checkout and test are in:

/work/noaa/marine/dworthen/utils_dw
/work/noaa/stmp/dworthen/CPLD_GRIDGEN/rt_106867

Both Hercules and Orion write tests to the same location. So my latest Hercules test is

/work/noaa/nems/dworthen/utils_dw/
/work/noaa/stmp/dworthen/CPLD_GRIDGEN/rt_1639913

DeniseWorthen · 2024-04-10T14:44:41Z

@GeorgeGayno-NOAA So the differences are in the masked ocean mask files. Are we pointing to different orog fix files?

In /work/noaa/stmp/ggayno/CPLD_GRIDGEN/rt_443522/025, grid.nml shows fv3dir=/work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.denise/fix/orog'

which is

ls -l /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.denise/fix/orog
lrwxrwxrwx 1 ggayno da 43 Feb 15  2023 /work/noaa/da/ggayno/save/ufs_utils.git/UFS_UTILS.denise/fix/orog -> /work/noaa/global/glopara/fix/orog/20220805

In my /work/noaa/stmp/dworthen/CPLD_GRIDGEN/rt_106867/025, grid.nml shows
fv3dir='/work/noaa/marine/dworthen/utils_dw/fix/orog'

which is

ls -l /work/noaa/marine/dworthen/utils_dw/fix/orog
lrwxrwxrwx 1 dworthen marine 43 Apr 10 06:48 /work/noaa/marine/dworthen/utils_dw/fix/orog -> /work/noaa/global/glopara/fix/orog/20231027

GeorgeGayno-NOAA · 2024-04-10T18:54:57Z

@DeniseWorthen I was pointing to some wrong 'fix' files. Everything passes on all machines for me. Sorry for the confusion. I will merge.

GeorgeGayno-NOAA

Able to successfully run the regression tests against the new baselines. Everything looks good. Will merge.

DeniseWorthen · 2024-04-10T19:04:50Z

@GeorgeGayno-NOAA Great. Thanks for working through it with me.

GeorgeGayno-NOAA self-requested a review March 18, 2024 12:22

DeniseWorthen and others added 3 commits March 18, 2024 09:15

clean up log messages, remove duplicate weight file

0b514b5

Merge branch 'ufs-community:develop' into feature/fixangleBu

2ed3a28

update documentation

4fa58c1

Merge branch 'ufs-community:develop' into feature/fixangleBu

61b8bef

DeniseWorthen and others added 9 commits March 22, 2024 10:43

test against bl.addanglevar.1e4e33c4

d6ce6a7

* tripole, grid_cice and

test against bl.addanglevar.1e4e33c4

1979f6d

* grid cice and tripole files are not b4b

modifications for new unit test

bec20a5

* need to allow for non-full dimensioned arrays being sent to angles routines

confirm results against bl.addanglevar.1e4e33c4

202d311

* only cice grid files, tripole files change

partially working unit test

068d6bb

* need to clean up messaging and angchk test

commit new unit test

76ff0a3

* 5x5 blocks of grid points are used to calc angle variables and checked against known output from mom6 and cice6 history files

Merge branch 'ufs-community:develop' into feature/fixangleBu

ec38343

update unit test

2a5deaf

* send array bounds to angles routines. This allows non-calculated values within the unit test to return as values 0.0 instead of non-sensical values

baseline test

53b8a94

* baselines pass against a bl created from addanglevar branch * only cice grid and tripole files differ

DeniseWorthen marked this pull request as ready for review March 25, 2024 17:45

remove unused variable

677fadd

GeorgeGayno-NOAA reviewed Mar 26, 2024

View reviewed changes

tests/cpld_gridgen/ftst_find_angq.F90 Show resolved Hide resolved

fix unit test

5da3dc6

GeorgeGayno-NOAA approved these changes Mar 28, 2024

View reviewed changes

fixes from testing w/ hercules gnu12

993c670

* gnu and gnu debug triggered several errors previously not caught using intel and intel debug

set wall clock and memory higher in attempt to resolve test

f21f0b1

failures

DeniseWorthen added 2 commits April 8, 2024 19:49

add hera log

aad5ef1

add jet log

7f2df8f

DeniseWorthen added 3 commits April 8, 2024 21:23

add wcoss2 log

a99e6b1

add orion log

5148e60

add hercules log

9caec86

GeorgeGayno-NOAA approved these changes Apr 10, 2024

View reviewed changes

GeorgeGayno-NOAA merged commit 8781eb8 into ufs-community:develop Apr 10, 2024
4 checks passed

DeniseWorthen mentioned this pull request Apr 12, 2024

update inputdata for Bu angle fix; re-enable cpld C48 warmstart test ufs-community/ufs-weather-model#2235

Merged

14 tasks

DeniseWorthen mentioned this pull request May 6, 2024

create two additional remapping weights using cpld_gridgen for use by ocnice prep; deprecate two weights files and regresssion test dependency #943

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix angles on Bu for use by CICE #922

fix angles on Bu for use by CICE #922

DeniseWorthen commented Mar 15, 2024 •

edited by GeorgeGayno-NOAA

DeniseWorthen commented Mar 18, 2024

GeorgeGayno-NOAA commented Mar 22, 2024

DeniseWorthen commented Mar 22, 2024

DeniseWorthen commented Mar 25, 2024

GeorgeGayno-NOAA commented Mar 26, 2024

GeorgeGayno-NOAA commented Mar 26, 2024

DeniseWorthen commented Mar 26, 2024

DeniseWorthen commented Mar 28, 2024

DeniseWorthen commented Mar 28, 2024

GeorgeGayno-NOAA commented Mar 28, 2024

DeniseWorthen commented Apr 2, 2024

GeorgeGayno-NOAA commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

GeorgeGayno-NOAA commented Apr 5, 2024

DeniseWorthen commented Apr 5, 2024

DeniseWorthen commented Apr 8, 2024 •

edited

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

GeorgeGayno-NOAA commented Apr 8, 2024

GeorgeGayno-NOAA commented Apr 10, 2024

DeniseWorthen commented Apr 10, 2024

DeniseWorthen commented Apr 10, 2024 •

edited

GeorgeGayno-NOAA commented Apr 10, 2024

GeorgeGayno-NOAA left a comment

DeniseWorthen commented Apr 10, 2024

fix angles on Bu for use by CICE #922

fix angles on Bu for use by CICE #922

Conversation

DeniseWorthen commented Mar 15, 2024 • edited by GeorgeGayno-NOAA

TESTS CONDUCTED:

DOCUMENTATION:

ISSUE:

DeniseWorthen commented Mar 18, 2024

GeorgeGayno-NOAA commented Mar 22, 2024

DeniseWorthen commented Mar 22, 2024

DeniseWorthen commented Mar 25, 2024

GeorgeGayno-NOAA commented Mar 26, 2024

GeorgeGayno-NOAA commented Mar 26, 2024

DeniseWorthen commented Mar 26, 2024

DeniseWorthen commented Mar 28, 2024

DeniseWorthen commented Mar 28, 2024

GeorgeGayno-NOAA commented Mar 28, 2024

DeniseWorthen commented Apr 2, 2024

GeorgeGayno-NOAA commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

DeniseWorthen commented Apr 3, 2024

GeorgeGayno-NOAA commented Apr 5, 2024

DeniseWorthen commented Apr 5, 2024

DeniseWorthen commented Apr 8, 2024 • edited

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

DeniseWorthen commented Apr 8, 2024

GeorgeGayno-NOAA commented Apr 8, 2024

GeorgeGayno-NOAA commented Apr 10, 2024

DeniseWorthen commented Apr 10, 2024

DeniseWorthen commented Apr 10, 2024 • edited

GeorgeGayno-NOAA commented Apr 10, 2024

GeorgeGayno-NOAA left a comment

Choose a reason for hiding this comment

DeniseWorthen commented Apr 10, 2024

DeniseWorthen commented Mar 15, 2024 •

edited by GeorgeGayno-NOAA

DeniseWorthen commented Apr 8, 2024 •

edited

DeniseWorthen commented Apr 10, 2024 •

edited