Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Applications for generating mom6 increment and hybrid-weights #524

Merged
merged 14 commits into from
Jul 17, 2023

Conversation

guillaumevernieres
Copy link
Contributor

@guillaumevernieres guillaumevernieres commented Jul 6, 2023

@guillaumevernieres guillaumevernieres added the orion-GW-RT Queue for automated testing with global-workflow on Orion label Jul 6, 2023
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 6, 2023
CoryMartin-NOAA
CoryMartin-NOAA previously approved these changes Jul 6, 2023
@emcbot
Copy link

emcbot commented Jul 6, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu Jul  6 14:47:06 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Jul  6 16:02:54 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu Jul  6 16:19:21 CDT 2023
Tests: 90% tests passed, 5 tests failed out of 52
	1489 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1490 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1491 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jul 6, 2023
@guillaumevernieres guillaumevernieres added hera-RT Queue for automated testing on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion and removed orion-GW-RT-Failed Automated testing with global-workflow failed on Orion labels Jul 6, 2023
@emcbot emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 6, 2023
@emcbot
Copy link

emcbot commented Jul 7, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu Jul  6 17:46:12 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Jul  6 18:43:50 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu Jul  6 19:02:57 CDT 2023
Tests: 90% tests passed, 5 tests failed out of 52
	1501 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1502 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1503 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion hera-RT-Running Automated testing running on Hera and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion hera-RT Queue for automated testing on Hera labels Jul 7, 2023
@emcbot
Copy link

emcbot commented Jul 7, 2023

Automated GDASApp Testing Results:
Machine: hera

Start: Fri Jul  7 00:45:20 UTC 2023 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul  7 01:26:41 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Jul  7 01:27:50 UTC 2023
Tests: 100% tests passed, 0 tests failed out of 28

@emcbot emcbot added hera-RT-Passed Automated testing successful on Hera and removed hera-RT-Running Automated testing running on Hera labels Jul 7, 2023
@CoryMartin-NOAA CoryMartin-NOAA added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Jul 7, 2023
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Jul 7, 2023
@emcbot
Copy link

emcbot commented Jul 14, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Jul 14 18:23:06 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul 14 18:32:10 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Jul 14 18:42:25 UTC 2023
Tests: 

Build: SUCCESS
Build: Completed at Fri Jul 14 18:47:17 UTC 2023

Tests: Failed
Tests: Failed at Fri Jul 14 19:47:03 UTC 2023
Tests:
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added the hera-GW-RT-Failed Automated testing with global-workflow failed on Hera label Jul 14, 2023
@emcbot
Copy link

emcbot commented Jul 14, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Jul 14 18:23:06 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul 14 18:32:10 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Fri Jul 14 18:42:25 UTC 2023
Tests: 

Build: SUCCESS
Build: Completed at Fri Jul 14 18:47:17 UTC 2023

Tests: Failed
Tests: Failed at Fri Jul 14 19:47:03 UTC 2023
Tests:
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

Tests:                                  *Failed*
Tests: Failed at Fri Jul 14 19:54:57 UTC 2023
Tests: 98% tests passed, 1 tests failed out of 52
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@CoryMartin-NOAA
Copy link
Contributor

@guillaumevernieres why are the emcbot messages' formatting now screwed up?

@guillaumevernieres
Copy link
Contributor Author

@guillaumevernieres why are the emcbot messages' formatting now screwed up?

Quantum ci, it's got both pass and failed status.
I wonder if a previous ci was still running when the next one was submitted.

@guillaumevernieres
Copy link
Contributor Author

I might revert the debug to batch change if the ci fails again next time around and isolate that change in a dedicated pr.
I wasn't submitting the jobs properly on orion, that's why it failed, but the hera issue is different, the atm_jjob_ens_run is timing out @RussTreadon-NOAA , did it happen in the past? I'm done for today.

@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion hera-GW-RT-Passed Automated testing with global-workflow successful on Hera labels Jul 14, 2023
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Jul 14, 2023
@emcbot
Copy link

emcbot commented Jul 15, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Fri Jul 14 18:31:02 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul 14 19:03:00 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Fri Jul 14 19:29:42 CDT 2023
Tests: 92% tests passed, 4 tests failed out of 53
	1518 - test_gdasapp_atm_jjob_var_run (Failed)
	1519 - test_gdasapp_atm_jjob_var_final (Failed)
	1521 - test_gdasapp_atm_jjob_ens_run (Failed)
	1522 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Jul 15, 2023
@emcbot
Copy link

emcbot commented Jul 15, 2023

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Fri Jul 14 23:30:42 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Jul 14 23:57:36 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Sat Jul 15 00:57:00 UTC 2023
Tests: 92% tests passed, 4 tests failed out of 53
	1518 - test_gdasapp_atm_jjob_var_run (Failed)
	1519 - test_gdasapp_atm_jjob_var_final (Failed)
	1522 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Jul 15, 2023
@guillaumevernieres
Copy link
Contributor Author

@RussTreadon-NOAA , any-chance you could have a look at the reason for the failed ctests:

	1518 - test_gdasapp_atm_jjob_var_run (Failed)
	1519 - test_gdasapp_atm_jjob_var_final (Failed)
	1521 - test_gdasapp_atm_jjob_ens_run (Failed)
	1522 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/524/global-workflow/sorc/gdas.cd/build/log.ctest

I tried to have them submitted to the batch qos, but I might have broken something in the process.

@guillaumevernieres
Copy link
Contributor Author

guillaumevernieres commented Jul 17, 2023

I wasn't paying attention. The atmos failure is being addressed here.
Could I get a review @AndrewEichmann-NOAA , @RussTreadon-NOAA , @CoryMartin-NOAA , @jiaruidong2017 ?

Copy link
Collaborator

@AndrewEichmann-NOAA AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nothing obviously wrong

@guillaumevernieres guillaumevernieres merged commit 113ecc8 into develop Jul 17, 2023
5 checks passed
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/oops_app branch July 17, 2023 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion
Projects
None yet
5 participants