Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coupled IAU RT test #1916

Merged
merged 18 commits into from
Oct 12, 2023
Merged

Conversation

junwang-noaa
Copy link
Collaborator

@junwang-noaa junwang-noaa commented Sep 21, 2023

PR Author Checklist:

  • I have linked PR's from all sub-components involved in section below.
  • I am confirming reviews are completed in ALL sub-component PR's.
  • I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
    • LOG:
  • I have added the list of all failed regression tests to "Anticipated changes" section.
  • I have filled out all sections of the template.

Description

A coupled IAU RT test cpld_control_gfsv17_iau is created for GFSV17 testing.

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Subcomponent Pull Requests

None

Blocking Dependencies

None

Subcomponents involved:

  • AQM
  • CDEPS
  • CICE
  • CMEPS
  • CMakeModules
  • FV3
  • GOCART
  • HYCOM
  • MOM6
  • NOAHMP
  • WW3
  • stochastic_physics
  • none

Anticipated Changes

Input data

  • No changes are expected to input data.
  • Changes are expected to input data:
    • New input data.
    • Updated input data.

Regression Tests:

  • No changes are expected to any regression test: baseline is needed for the new test, no results change for current tests
  • Changes are expected to the following tests:
Tests effected by changes in this PR:

Libraries

  • Not Needed
  • Needed
    • Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
    • Add issue link from JCSDA/spack-stack following this item
Code Managers Log
  • This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
  • Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
    • N/A

Testing Log:

  • RDHPCS
    • Hera
    • Orion
    • Hercules
    • Jet
    • Gaea
    • Cheyenne
  • WCOSS2
    • Dogwood/Cactus
    • Acorn
  • CI
    • Completed
  • opnReqTest
    • N/A
    • Log attached to comment

@junwang-noaa
Copy link
Collaborator Author

The full RT passed except the missing baseline for the new test. The RT log is at:

/scratch1/NCEPDEV/nems/Jun.Wang/nems/vlab/20230817/ufs-weather-model/tests/logs

@junwang-noaa
Copy link
Collaborator Author

@jiandewang would you please check if the mom6 output from the IAU test at:

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel ? Thanks

@jiandewang
Copy link
Collaborator

@jiandewang would you please check if the mom6 output from the IAU test at:

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel ? Thanks

@junwang-noaa I had a quick look on ocean output, nothing strange

@JessicaMeixner-NOAA
Copy link
Collaborator

@jiandewang @NeilBarton-NOAA - This PR is testing the IAU capability which includes wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3. Can you confirm if the ocean/ice models are correctly doing this zero-ing out or if this remain open issues?

@jiandewang
Copy link
Collaborator

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel

@jiandewang @NeilBarton-NOAA - This PR is testing the IAU capability which includes wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3. Can you confirm if the ocean/ice models are correctly doing this zero-ing out or if this remain open issues?

@JessicaMeixner-NOAA I am not quite understanding what you want me to check. All I see is there are two ocean outout files:
ocn_2021_03_22_12.nc and ocn_2021_03_22_18.nc. Model started at 6z and saved every 6hr output

@jiandewang
Copy link
Collaborator

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel

@jiandewang @NeilBarton-NOAA - This PR is testing the IAU capability which includes wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3. Can you confirm if the ocean/ice models are correctly doing this zero-ing out or if this remain open issues?

@JessicaMeixner-NOAA I am not quite understanding what you want me to check. All I see is there are two ocean outout files: ocn_2021_03_22_12.nc and ocn_2021_03_22_18.nc. Model started at 6z and saved every 6hr output

what is "wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3." ?

@jiandewang
Copy link
Collaborator

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel

@jiandewang @NeilBarton-NOAA - This PR is testing the IAU capability which includes wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3. Can you confirm if the ocean/ice models are correctly doing this zero-ing out or if this remain open issues?

@JessicaMeixner-NOAA I am not quite understanding what you want me to check. All I see is there are two ocean outout files: ocn_2021_03_22_12.nc and ocn_2021_03_22_18.nc. Model started at 6z and saved every 6hr output

what is "wanting the averaging buckets to be zerod out at FHR=0 when we have FHROT=3." ?

I guess this test is on FV3 side it is doing this kind of IAU but not on ocean side, @junwang-noaa can you confirm on this ?

@DeniseWorthen
Copy link
Collaborator

@junwang-noaa For CICE, my initial read is that this is doing what you want for the CICE history files. In your run, in ice_diag.d, I see that the first history file iceh_06h.2021-03-22-43200.nc is written after 15 timesteps and 15*720secs = 3 hours. The next file iceh_06h.2021-03-22-64800.nc is written after 30 timesteps (6 hours).

I will do a test and see if the fields in the history files appear to be 3 and 6 hour averages as expected.

@JessicaMeixner-NOAA
Copy link
Collaborator

That's great news @DeniseWorthen

@jiandewang we're basically expecting to see what @DeniseWorthen mentioned. FHROT means that we change when we start the forecast relative to the forecast hour 0. We can hop on a quick call tomorrow tand I can try to explain more if needed.

@jiandewang
Copy link
Collaborator

That's great news @DeniseWorthen

@jiandewang we're basically expecting to see what @DeniseWorthen mentioned. FHROT means that we change when we start the forecast relative to the forecast hour 0. We can hop on a quick call tomorrow tand I can try to explain more if needed.

now I know what's your purpose. But for ocean all I see is 12z and 18z output. I think diag_table doesn't have this capability but ice has a way to control that.

@junwang-noaa
Copy link
Collaborator Author

@DeniseWorthen Thanks for confirmation. I think ice is OK. We do not expect users to use any averaged fields in the first ice output iceh_06h.2021-03-22-43200.nc, which is corresponding to atm f00 at 12z cycle. The next history file iceh_06h.2021-03-22-64800.nc has correct 6 hour average.

@junwang-noaa
Copy link
Collaborator Author

@jiandewang if I understand correct, the ocn files should also doing the same thing as ice does. But please check the SST file:

/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_248142/cpld_control_gfsv17_iau_intel/SST_2021_03_23.nc

which is a daily mean. I assume we will not output it in marine DA (@guillaumevernieres) step, but will be output in GFSv17 forecast and this daily mean starts from the initial forecast time (start time), which is 6 hourly before the current cycle. E.g. in this case, the SST mean is from 2023062206- 2023062318, while this is a 12Z cycle run and we need it to be 2023062212-2023022306.

@junwang-noaa
Copy link
Collaborator Author

@JessicaMeixner-NOAA would you please confirm the wave fields coming out correctly? Thanks

@JessicaMeixner-NOAA
Copy link
Collaborator

The wave output is as requested which is instantaneous point and gridded files every 3 hours.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Sep 29, 2023

@jiandewang @junwang-noaa The SST_2021_03_23.nc file contains all missing values for the SST. I think this is because FMS does not have a complete "24 hour" set of values to accumulate and average. I'm not sure if this file is actually needed.

I can't remember how the 3-d files are named. Is the timestamp the end of the averaging period, so that ocn_2021_03_22_18.nc contain the 6-hour mean of hours 13,14,15,16,17,18?

@jiandewang
Copy link
Collaborator

@DeniseWorthen you are right, this setting only run 18hr so SST file has all missing value. For 6hr 3D the file name timestamp is using end of time.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Sep 29, 2023

@jiandewang Thanks for the clarification. In that case, I am not confident that we're getting the expected values in the ocean file but I can't yet be entirely sure. What I did was write mediator history file 1-hour averages, averaged them over the intervals and compared that to what the model has for the same interval. For the ice model, I saw differences O~10-8 but I'm seeing differences more like O~10-1 w/ the MOM6 file for SST.

@junwang-noaa
Copy link
Collaborator Author

@DeniseWorthen @jiandewang Thanks. I think the ocn file is in the mid-time. Please see ocn output from 24 hr forecast with a cold start:
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_182886/cpld_control_gfsv17_intel

@jiandewang
Copy link
Collaborator

@DeniseWorthen @jiandewang Thanks. I think the ocn file is in the mid-time. Please see ocn output from 24 hr forecast with a cold start: /scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_182886/cpld_control_gfsv17_intel

sorry my bad. ocn file is the middle time. In g-w Prototype and HR runs we rename them to ending time in the script. In rt.sh we don't do this renaming step. I mixed rt.sh and g-w.

@DeniseWorthen
Copy link
Collaborator

@jiandewang Ok, my brain isn't working today! What hours are used to create the ocn output that is labeled "18"?

@jiandewang
Copy link
Collaborator

@jiandewang Ok, my brain isn't working today! What hours are used to create the ocn output that is labeled "18"?

19z-24z

@zach1221 zach1221 added No Baseline Change No Baseline Change and removed Baseline Updates Current baselines will be updated. labels Oct 11, 2023
@zach1221
Copy link
Collaborator

My first test run failed on Hera for the new test because of missing baselines. Do we need to change the directories in the cpld_control_run.IN file so it points to the new locations for the new input data in input-data-20221101? Currently still points to Jun's scratch directory.

@DeniseWorthen
Copy link
Collaborator

DeniseWorthen commented Oct 11, 2023

@zach1221 Yes, assuming the file has been added to the inputdata on all platforms, @junwang-noaa needs to make that update in cpld_control_run.IN. Or, can you push to her branch?

@zach1221
Copy link
Collaborator

@zach1221 Yes, assuming the file has been added to the inputdata on all platforms, @junwang-noaa needs to make that update in cpld_control_run.IN. Or, can you push to her branch?

Yes, the new input data has been distributed across the platforms. Except for wcoss2 I guess.

@jkbk2004
Copy link
Collaborator

@zach1221 Yes, assuming the file has been added to the inputdata on all platforms, @junwang-noaa needs to make that update in cpld_control_run.IN. Or, can you push to her branch?

Yes, the new input data has been distributed across the platforms. Except for wcoss2 I guess.

Increment files were rsynced ok acorss all RDHPCS.

@DeniseWorthen
Copy link
Collaborator

@zach1221 Jun asked me to update the directory in the PR. Can you please try to generate a baseline for the new test now? Hopefully I got it right, but I did not test first.

@zach1221
Copy link
Collaborator

@zach1221 Jun asked me to update the directory in the PR. Can you please try to generate a baseline for the new test now? Hopefully I got it right, but I did not test first.

Ok, thanks. I wasn't certain what it should be, so was testing a change on my end. I'll try yours and let you know how it goes.

aerorahul
aerorahul previously approved these changes Oct 11, 2023
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes look good to me.
Thank you for synchronizing the MOM_input_templates from the global-workflow.
Please link the issue #1613 with this PR. That issue will be resolved with this merge.

@zach1221
Copy link
Collaborator

@junwang-noaa we're now finished with testing. If you're able to close the two unresolved conversations, then we can begin the merging process.

@junwang-noaa
Copy link
Collaborator Author

Done

@DeniseWorthen
Copy link
Collaborator

@junwang-noaa I meant to ask---I know at some point you were testing an updated FMS in relation to this PR? Is an updated FMS required to have the IAU working properly for MOM6 output?

@junwang-noaa
Copy link
Collaborator Author

junwang-noaa commented Oct 12, 2023

That's right. I am waiting for an FMS release so that we can get correct MOM6 output in IAU configuration. It may take some time, I am still checking the timeline of the FMS release

@zach1221 zach1221 merged commit 68050e5 into ufs-community:develop Oct 12, 2023
@junwang-noaa junwang-noaa deleted the couplediau branch February 5, 2024 17:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
jenkins-ci Jenkins CI: ORT build/test on docker container No Baseline Change No Baseline Change Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

create coupled IAU RT test case Update MOM_input templates to be usable for global applications GFS and GEFS
10 participants