Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Archiving cleanup #2621

Merged
merged 169 commits into from
Jun 1, 2024
Merged

Archiving cleanup #2621

merged 169 commits into from
Jun 1, 2024

Conversation

DavidHuber-NOAA
Copy link
Contributor

Description

  1. Adds a lot of comments to the jinja templates for archiving
  2. Rearranges the gdas and enkf templates to a more logical order
  3. Fixes a couple of bugs in the enkf archiving of increments and analyses
  4. Disables archiving for the half cycle
  5. Removes the FITSARC key from config.base and arcdir.yaml.j2, instead relying on DO_FIT2OBS
  6. Updates wxflow to add the option to not allow undefined variables when parsing jinja templates and invokes this feature when running archives

Resolves #2612

Type of change

  • Bug fix (fixes something broken)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO

How has this been tested?

Cycled test on Hera

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes

@DavidHuber-NOAA
Initial set of archiving yamls.
de29141
@DavidHuber-NOAA
rework gdas.yaml to Jinja template
404a46b
@DavidHuber-NOAA
Restart with a blank slate; rename gdas.yaml.
172dc68
@DavidHuber-NOAA
Create archive task class with initial gdas.yaml.j2
60b9cd4 
This also
- adds a missing COM variable to JGLOBAL_ARCHIVE
- Creates a test driver script and python program
@DavidHuber-NOAA
Create configure method
754c196
@DavidHuber-NOAA
Updates to archive task.
24dddf5
@DavidHuber-NOAA
Add htar features to archive task, update gdas template. (untested)
bdf66df
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
395bd62
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
4d2e64c
@DavidHuber-NOAA
Update archive protocol function, add templates.
e8cb2c1
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
4f14da5
@DavidHuber-NOAA
Archive protocol overhaul.
7b5a9cc 
- Use new hsi/htar classes in wxflow
- Assign cycle dates/hours in the job to avoid assigning in each template
- Rename templates to match current/EE2 naming conventions
- Write function to relative-ize COM paths
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
b51cceb
@DavidHuber-NOAA
Update snowda archiving to match develop.
cbc815d
@DavidHuber-NOAA
Address PEP8 issues.
a53c56a
@DavidHuber-NOAA
Add gdasocean_analysis template
aa204ed
@DavidHuber-NOAA
Add local (ARCDIR) archiving to archive.py.
fa98a28
@DavidHuber-NOAA
Fix PEP8 errors.
406c7a1
@DavidHuber-NOAA
Let archive.py handle tracker pslot renaming.
a99ecb5
@DavidHuber-NOAA
Rewrite archive job to use exglobal_archive.py.
1fcccf0
@DavidHuber-NOAA
Enabled GFS archiving and reworked some features
bd66212 
- Moved the cycle_YYYYMMDDHH, etc, variable declarations to templates
- Use COM directories inside of the templates (instead of _dir directories)
- Add for-looping for tile numbers
- Added several new templates needed for the GFS
- Improved ATARDIR archiving logic
@DavidHuber-NOAA
Update wxflow hash.
dc9004d
@DavidHuber-NOAA
Add enkf archiving.
a8cbe2b
@DavidHuber-NOAA
Remove now-defunct hpssarch_gen.sh.
8611bf8
@DavidHuber-NOAA
Address PEP8 warnings
533cccc
@DavidHuber-NOAA
Fix additional PEP8 warnings
05f54dc
@DavidHuber-NOAA
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
338bdd2
@DavidHuber-NOAA
Fixes for snow DA archiving.
31a1d9a
@DavidHuber-NOAA
Remove unused fhmin.
ec747f7
@DavidHuber-NOAA
Better explicitize enkf archiving
58840c4
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera label May 31, 2024
@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

Experiment C48_ATM_98051ebd SUCCESS on Wcoss2 at 05/31/24 03:03:37 PM

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

Experiment C48_S2SW_98051ebd SUCCESS on Wcoss2 at 05/31/24 03:51:13 PM

@emcbot
Copy link

emcbot commented May 31, 2024

Experiment C96C48_hybatmDA_98051ebd SUCCESS on Wcoss2 at 05/31/24 04:36:28 PM

@DavidHuber-NOAA
Copy link
Contributor Author

@TerrenceMcGuinness-NOAA It looks like the builds failed on Hera. Could you reboot CI on that platform?

@DavidHuber-NOAA
Copy link
Contributor Author

@TerrenceMcGuinness-NOAA Disregard, I see now that CI was restarted on Hera already.

@emcbot emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

CI Passed Hera at
Built and ran in directory /scratch1/NCEPDEV/global/CI/2621

@emcbot emcbot added CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully and removed CI-Hercules-Running **Bot use only** CI testing on Hercules for this PR is in-progress labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

CI Passed Hercules at
Built and ran in directory /work2/noaa/stmp/CI/HERCULES/2621

@emcbot
Copy link

emcbot commented May 31, 2024

Experiment C96_atm3DVar_extended_98051ebd SUCCESS on Wcoss2 at 05/31/24 10:36:34 PM

@emcbot emcbot added CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully and removed CI-Wcoss2-Running **Bot use only** CI testing on WCOSS for this PR is in-progress labels May 31, 2024
@emcbot
Copy link

emcbot commented May 31, 2024

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_98051ebd *** SUCCESS *** at 05/31/24 03:03:37 PM
Experiment C48_S2SW_98051ebd *** SUCCESS *** at 05/31/24 03:51:13 PM
Experiment C96C48_hybatmDA_98051ebd *** SUCCESS *** at 05/31/24 04:36:28 PM
Experiment C96_atm3DVar_extended_98051ebd *** SUCCESS *** at 05/31/24 10:36:34 PM

@emcbot emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Jun 1, 2024
@emcbot
Copy link

emcbot commented Jun 1, 2024

CI Passed Orion at
Built and ran in directory /work2/noaa/stmp/CI/ORION/2621

@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit c92bf41 into NOAA-EMC:develop Jun 1, 2024
9 of 10 checks passed
@DavidHuber-NOAA DavidHuber-NOAA deleted the feature/archive_cleanup branch June 3, 2024 15:46
danholdaway added a commit to danholdaway/global-workflow that referenced this pull request Jun 4, 2024
@danholdaway
Merge remote-tracking branch 'upsteam/develop' into feature/rename_atm
97a63a7 
* upsteam/develop:
  Archiving cleanup (NOAA-EMC#2621)
  Switch to Rocky 9 built external packages on Hercules (NOAA-EMC#2608)
  Add the capability to use slurm reservation nodes (NOAA-EMC#2627)
  Update forecast job to use COMIN/COMOUT (NOAA-EMC#2622)
  Update to add 1-deg global wave grid (NOAA-EMC#2619)
  Add C384mx025_3DVarAOWCDA yamls (NOAA-EMC#2625)
  Script to keep Jenkins Agent persistent from cron (NOAA-EMC#2634)
@emilyhcliu emilyhcliu mentioned this pull request Jun 10, 2024
Closed
@CatherineThomas-NOAA CatherineThomas-NOAA mentioned this pull request Jun 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees
No one assigned
Labels
CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Hercules-Passed **Bot use only** CI testing on Hercules for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully CI-Wcoss2-Passed **Bot use only** CI testing on WCOSS for this PR has completed successfully
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Disable archiving on half cycles and clean up yamls
6 participants
@DavidHuber-NOAA @AndrewEichmann-NOAA @guillaumevernieres @emcbot @WalterKolczynski-NOAA @TerrenceMcGuinness-NOAA