Hotfix/restart build

[TerrenceMcGuinness-NOAA]

Description

Improved CI robustness for reverting back to CI-Ready from any given state
See a full review of the latest updated PR user's messaging from the CI Scripts (here)

New Features:

• Improved scancel routine (refactored into bash "subroutine")

• Improved messaging (see below) when ever a user changes state

• Any and all previous build scripts and running experiments are killed as a result of reset to Ready

  Resolves bug/fix on false positive for link phase fail NOAA-EMC/global-workflow#2060
  Resolves Add Kill Switch (Label) to CI NOAA-EMC/global-workflow#2067

• Bug fix (Resetting CI tests had lingering builds simultaneously running)

• New feature (Robust Resetting with messages)

• Maintenance (code refactor - use of BASH subroutines in main drivers)

Example of output when user updates state and request a restart:

CI Update on Hera at 11/28/23 09:33:11 PM
=================================================
PR:201 Reset to Hera-Ready by user and is now restarting CI tests
Driver PID: 302744 on hfe12 is no longer running this test
Driver PID: has restarted as 163669 on hfe12
No current experiments to cancel in PR: 201 on Hera

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO

How has this been tested?

Reset at all steps in dev Cron CI space with appropriate kills taking place on Hera and Orion

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[emcbot]

CI Update on Hera at 11/28/23 10:45:11 PM
============================================
Cloning and Building global-workflow PR: 203
with PID: 294296 on host: hfe12

[emcbot]

CI Update on Orion at 11/29/23 01:18:04 PM
=================================================
PR:203 Reset to Orion-Ready by user and is now restarting CI tests
Canceled all jobs for experiment C48_ATM_7e05961c in PR:203 on Orion
Canceled all jobs for experiment C48_S2SA_gefs_7e05961c in PR:203 on Orion
Canceled all jobs for experiment C48_S2SW_7e05961c in PR:203 on Orion
Canceled all jobs for experiment C96_atm3DVar_7e05961c in PR:203 on Orion
Canceled all jobs for experiment C96C48_hybatmDA_7e05961c in PR:203 on Orion

[emcbot]

CI Update on Orion at 11/29/23 01:18:47 PM
============================================
Cloning and Building global-workflow PR: 203
with PID: 337816 on host: Orion-login-1

[emcbot]

CI Update on Hera at 11/29/23 07:20:38 PM
============================================
Cloning and Building global-workflow PR: 203
with PID: 246466 on host: hfe12

[emcbot]

Automated global-workflow Testing Results:

Machine: Orion
Start: Wed Nov 29 13:18:59 CST 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout: Completed at 11/29/23 01:19:59 PM
Build: Completed at 11/29/23 02:12:31 PM
Case setup: Completed for experiment C48_ATM_063a95b4
Case setup: Completed for experiment C48_S2SWA_gefs_063a95b4
Case setup: Completed for experiment C48_S2SW_063a95b4
Case setup: Completed for experiment C96_atm3DVar_063a95b4
Case setup: Completed for experiment C96C48_hybatmDA_063a95b4

[emcbot]

Automated global-workflow Testing Results:

Machine: Hera
Start: Wed Nov 29 19:20:49 UTC 2023 on hfe12
---------------------------------------------------
Checkout: Completed at 11/29/23 07:23:08 PM
Build: Completed at 11/29/23 08:12:36 PM
Case setup: Completed for experiment C48_ATM_063a95b4
Case setup: Completed for experiment C48_S2SW_063a95b4
Case setup: Completed for experiment C48_S2SWA_gefs_063a95b4
Case setup: Completed for experiment C96C48_hybatmDA_063a95b4
Case setup: Completed for experiment C96_atm3DVar_063a95b4

[emcbot]

Experiment C48_ATM_063a95b4 SUCCESS on Hera at 11/29/23 09:21:11 PM

[emcbot]

Experiment C96C48_hybatmDA_063a95b4 SUCCESS on Hera at 11/29/23 10:27:16 PM

[emcbot]

Experiment C96C48_hybatmDA_063a95b4 SUCCESS on Orion at 11/29/23 04:51:14 PM

[emcbot]

All CI Test Cases Passed on Hera:

Experiment C48_ATM_063a95b4 *** SUCCESS *** at 11/29/23 09:21:11 PM
Experiment C96C48_hybatmDA_063a95b4 *** SUCCESS *** at 11/29/23 10:27:16 PM

NOTE: This is a "short" incomplete list of completed cases because of a discrepancy with both the dev crons and the productions crons pointing to the same build location with the check_ci.sh scripts. This will not happen in the future by simply having dev builds in a separate location going forward.

[emcbot]

All CI Test Cases Passed on Orion:

Experiment C96C48_hybatmDA_063a95b4 *** SUCCESS *** at 11/29/23 04:51:14 PM