Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update support for the GSI on Jet #215

Closed
DavidHuber-NOAA opened this issue Sep 21, 2021 · 4 comments · Fixed by #232
Closed

Update support for the GSI on Jet #215

DavidHuber-NOAA opened this issue Sep 21, 2021 · 4 comments · Fixed by #232
Assignees

Comments

@DavidHuber-NOAA
Copy link
Collaborator

As part of the global workflow support on Jet (NOAA-EMC/global-workflow#357), the Jet build/run scripts and regression tests for the GSI should be updated. The GSI has successfully built and run using the global workflow based on commit 9c1fc15d on port_2_jet. Additional work will be required to incorporate these changes into the master and to support regression tests on Jet, including adding the test case dataset.

@MichaelLueken-NOAA I will add the changes to incorporate the build/run script changes. I'm also willing to make the code changes to enable regression testing on Jet. Do you know if the regression test data is already hosted on Jet?

@MichaelLueken
Copy link
Contributor

@DavidHuber-NOAA I don't have an official allocation on Jet, just my home directory, so there are no regression test canned data available on Jet.

@DavidHuber-NOAA
Copy link
Collaborator Author

I've successfully ported the regression tests and they all pass except global_fv3_4denvar_C192 and global_C96_fv3aero.

For global_fv3_4denvar_C192, it failed because I failed to add add the path to ncdump. This test is rerunning now.

For global_C96_fv3aero, it failed for two reasons:

  1. the time it took to run global_C96_fv3aero_hiproc_updat was ~65s, with a threshold of ~39s
  2. related, the scalability test failed as the time difference between global_C96_fv3aero_hiproc_updat and global_C96_fv3aero_loproc_updat differed by ~34s, which is greater than the threshold of 4.6s.

@MichaelLueken-NOAA Assuming global_fv3_4denvar_C192 passes, may I proceed with the port or do these threshold violations require more attention?

@MichaelLueken
Copy link
Contributor

@DavidHuber-NOAA So long as the regression tests reproduce, then you should be fine moving forward with the port (wall time and scalability can pass with a resubmission of the tests, failures are generally due to system noise ).

@DavidHuber-NOAA
Copy link
Collaborator Author

The global_fv3_4denvar_C192 test failed on the second run as global_fv3_4denvar_C192_loproc_updat used 3.2GB of hardware memory, exceeding the threshold of 2.6GB. All reproducibility tests passed, so I am going to proceed.

All I have left at this point are the monitors, which I just need to test. I'm going to wrap in the port of the monitors to S4 in this issue as well.

DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 14, 2021
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 14, 2021
…ts made for the workflow.

The MinMon will now look for data files in the location specified by the
global global workflow vrfy.sh script.  The MinMon will also now copy
the images into the TANKDIR.
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 14, 2021
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 14, 2021
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 14, 2021
DavidHuber-NOAA added a commit to DavidHuber-NOAA/GSI that referenced this issue Oct 15, 2021
MichaelLueken added a commit that referenced this issue Oct 15, 2021
GitHub Issue #215. Update Jet support, port monitors to Jet and S4.
aerorahul pushed a commit to NOAA-EMC/GSI-Monitor that referenced this issue Jun 2, 2022
GitHub Issue NOAA-EMC/GSI#215. Update Jet support, port monitors to Jet and S4.
aerorahul pushed a commit to NOAA-EMC/GSI-utils that referenced this issue Jun 30, 2022
GitHub Issue NOAA-EMC/GSI#215. Update Jet support, port monitors to Jet and S4.
aerorahul pushed a commit to NOAA-EMC/global-workflow that referenced this issue Jul 12, 2022
GitHub Issue NOAA-EMC/GSI#215. Update Jet support, port monitors to Jet and S4.
aerorahul pushed a commit to NOAA-EMC/global-workflow that referenced this issue Jul 12, 2022
GitHub Issue NOAA-EMC/GSI#215. Update Jet support, port monitors to Jet and S4.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants