Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move fieldmetadata to inside version control rather than glopara fix #537

Merged
merged 7 commits into from
Jul 14, 2023

Conversation

CoryMartin-NOAA
Copy link
Contributor

I think out of convenience before, the glopara/fix/gdas directories contain several text files that do not need to be maintained by the glopara group, and can instead store them in GDASApp or global-workflow.

This PR is the first step towards that, by moving the FV3-JEDI IO fieldmetadata files into GDASApp. Subsequent work will need to determine if things like FV3 diag tables can be shared between FV3-JEDI and the UFS.

@CoryMartin-NOAA CoryMartin-NOAA self-assigned this Jul 14, 2023
@CoryMartin-NOAA CoryMartin-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 14, 2023
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 14, 2023
@CoryMartin-NOAA
Copy link
Contributor Author

A companion PR after this one will need to be made to global-workflow to update to this hash + modify scripts to look in the new location.

DavidNew-NOAA
DavidNew-NOAA previously approved these changes Jul 14, 2023
Copy link
Collaborator

@DavidNew-NOAA DavidNew-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

aerorahul
aerorahul previously approved these changes Jul 14, 2023
Copy link
Contributor

@aerorahul aerorahul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me.

@emcbot
Copy link

emcbot commented Jul 14, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Jul 14 16:50:27 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul 14 17:56:39 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Jul 14 18:07:12 UTC 2023
Tests: 90% tests passed, 5 tests failed out of 49
	1492 - test_gdasapp_land_letkfoi_snowda (Failed)
	1502 - test_gdasapp_atm_jjob_var_run (Failed)
	1503 - test_gdasapp_atm_jjob_var_final (Failed)
	1505 - test_gdasapp_atm_jjob_ens_run (Failed)
	1506 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/537/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 14, 2023
@emcbot
Copy link

emcbot commented Jul 14, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Jul 14 11:47:29 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Fri Jul 14 13:08:00 CDT 2023
Build: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/537/global-workflow/sorc/log.build

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jul 14, 2023
Copy link
Collaborator

@jiaruidong2017 jiaruidong2017 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One concern is that all the field meta data were merged into a single file: fv3jedi_fieldmetadata_restart.yaml. Once there is any change to the meta data from fv3-jedi, how does GDASApp make the updates?

@CoryMartin-NOAA
Copy link
Contributor Author

@jiaruidong2017 we just edit the file that I've added here. Not sure I understand the concern (unless I'm missing something)

@jiaruidong2017
Copy link
Collaborator

@CoryMartin-NOAA What I mean is, do we have a tool which can monitor the changes from fv3-jedi and update this file automatically? For example, when the GDASApp was built, the meta data were already updated in the directory: $(project_source_dir)/fv3-jedi/test/Data/fieldmetadata/. Can we simply use or link these files so that we don't need to monitor their changes?

@CoryMartin-NOAA
Copy link
Contributor Author

@jiaruidong2017 I think we want to know when this changes though, not anything automatic. Because this file contains the information on how to read the model state into JEDI. If anything changes here, we would need the model forecasts to change appropriately. We have the tests so that we know if it starts failing that something changed.

@jiaruidong2017
Copy link
Collaborator

@CoryMartin-NOAA Okay, I understand what you mean inside version control from your title. Thanks for answer my questions.

@CoryMartin-NOAA
Copy link
Contributor Author

Tests will fail until the workflow is updated. I have tested this manually on Hera and all tests pass:

      Start 1501: test_gdasapp_atm_jjob_var_init
43/49 Test #1501: test_gdasapp_atm_jjob_var_init ........................   Passed   44.30 sec
      Start 1502: test_gdasapp_atm_jjob_var_run
44/49 Test #1502: test_gdasapp_atm_jjob_var_run .........................   Passed   74.40 sec
      Start 1503: test_gdasapp_atm_jjob_var_final
45/49 Test #1503: test_gdasapp_atm_jjob_var_final .......................   Passed   10.10 sec
      Start 1504: test_gdasapp_atm_jjob_ens_init
46/49 Test #1504: test_gdasapp_atm_jjob_ens_init ........................   Passed   43.70 sec
      Start 1505: test_gdasapp_atm_jjob_ens_run
47/49 Test #1505: test_gdasapp_atm_jjob_ens_run .........................   Passed  170.86 sec
      Start 1506: test_gdasapp_atm_jjob_ens_final
48/49 Test #1506: test_gdasapp_atm_jjob_ens_final .......................   Passed   42.79 sec
      Start 1507: test_gdasapp_aero_gen_3dvar_yaml
49/49 Test #1507: test_gdasapp_aero_gen_3dvar_yaml ......................   Passed    0.25 sec

100% tests passed, 0 tests failed out of 49

This will need to be approved and merged before the workflow tests will pass, which will require a global-workflow PR, which will be a bit of a chicken/egg problem.

Copy link
Collaborator

@jiaruidong2017 jiaruidong2017 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me now.

@CoryMartin-NOAA CoryMartin-NOAA merged commit edd52d0 into develop Jul 14, 2023
5 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/fieldmetadata branch July 14, 2023 19:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants