Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Archiving cleanup #2621

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
169 commits
Select commit Hold shift + click to select a range
de29141
Initial set of archiving yamls.
DavidHuber-NOAA Mar 7, 2024
404a46b
rework gdas.yaml to Jinja template
DavidHuber-NOAA Mar 12, 2024
172dc68
Restart with a blank slate; rename gdas.yaml.
DavidHuber-NOAA Mar 12, 2024
60b9cd4
Create archive task class with initial gdas.yaml.j2
DavidHuber-NOAA Mar 15, 2024
754c196
Create configure method
DavidHuber-NOAA Mar 15, 2024
24dddf5
Updates to archive task.
DavidHuber-NOAA Mar 23, 2024
bdf66df
Add htar features to archive task, update gdas template. (untested)
DavidHuber-NOAA Apr 1, 2024
395bd62
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 1, 2024
4d2e64c
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 4, 2024
e8cb2c1
Update archive protocol function, add templates.
DavidHuber-NOAA Apr 8, 2024
4f14da5
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 8, 2024
7b5a9cc
Archive protocol overhaul.
DavidHuber-NOAA Apr 9, 2024
b51cceb
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 9, 2024
cbc815d
Update snowda archiving to match develop.
DavidHuber-NOAA Apr 9, 2024
a53c56a
Address PEP8 issues.
DavidHuber-NOAA Apr 9, 2024
aa204ed
Add gdasocean_analysis template
DavidHuber-NOAA Apr 9, 2024
fa98a28
Add local (ARCDIR) archiving to archive.py.
DavidHuber-NOAA Apr 11, 2024
406c7a1
Fix PEP8 errors.
DavidHuber-NOAA Apr 11, 2024
a99ecb5
Let archive.py handle tracker pslot renaming.
DavidHuber-NOAA Apr 12, 2024
1fcccf0
Rewrite archive job to use exglobal_archive.py.
DavidHuber-NOAA Apr 15, 2024
bd66212
Enabled GFS archiving and reworked some features
DavidHuber-NOAA Apr 17, 2024
dc9004d
Update wxflow hash.
DavidHuber-NOAA Apr 17, 2024
a8cbe2b
Add enkf archiving.
DavidHuber-NOAA Apr 18, 2024
8611bf8
Remove now-defunct hpssarch_gen.sh.
DavidHuber-NOAA Apr 18, 2024
533cccc
Address PEP8 warnings
DavidHuber-NOAA Apr 18, 2024
05f54dc
Fix additional PEP8 warnings
DavidHuber-NOAA Apr 18, 2024
338bdd2
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 19, 2024
31a1d9a
Fixes for snow DA archiving.
DavidHuber-NOAA Apr 19, 2024
ec747f7
Remove unused fhmin.
DavidHuber-NOAA Apr 19, 2024
58840c4
Better explicitize enkf archiving
DavidHuber-NOAA Apr 19, 2024
7532aeb
Further explicitize archiving templates.
DavidHuber-NOAA Apr 22, 2024
8e5dc8d
Small fix for snow DA archiving.
DavidHuber-NOAA Apr 22, 2024
fa870b3
Fix TC genesis arcdir archiving.
DavidHuber-NOAA Apr 22, 2024
ddd1397
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 22, 2024
5733738
Fix pgrb2b analysis filename
DavidHuber-NOAA Apr 22, 2024
02bb94e
Fix cyclone tracker filenames.
DavidHuber-NOAA Apr 22, 2024
a81ee42
Move forecast hours and restart intervals to config.base.
DavidHuber-NOAA Apr 23, 2024
066ffe2
Archive all sfcf* files.
DavidHuber-NOAA Apr 23, 2024
07a6537
Refactor restart/forecast hour-based archiving
DavidHuber-NOAA Apr 23, 2024
1c24f6b
Add TODOs for GDAS component archiving
DavidHuber-NOAA Apr 23, 2024
c8c2670
Explicitly archive gfs_collective files
DavidHuber-NOAA Apr 23, 2024
a3318e2
More explicitly archive gridded GFS Wave products
DavidHuber-NOAA Apr 23, 2024
6d932ba
Add optional wave station products to archives
DavidHuber-NOAA Apr 23, 2024
5e33693
Fix restart intervals in archiving templates.
DavidHuber-NOAA Apr 23, 2024
2458235
Address shellcheck suggestion
DavidHuber-NOAA Apr 24, 2024
200b0ef
Fix enkf_grp optional file declarations.
DavidHuber-NOAA Apr 24, 2024
7d06b11
Move verfrad warning files to optional list
DavidHuber-NOAA Apr 24, 2024
00c70e1
Fix restart_interval loop control.
DavidHuber-NOAA Apr 24, 2024
a23ee5e
Copy relevant changes to gefs config files.
DavidHuber-NOAA Apr 24, 2024
94011b3
Remove amsua files from archive list
DavidHuber-NOAA Apr 24, 2024
141249c
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 24, 2024
78b063f
Fix wave FHMAX for GEFS.
DavidHuber-NOAA Apr 25, 2024
a30a05f
Added untested enkfgfs warning to archive.py
DavidHuber-NOAA Apr 25, 2024
6662db9
Merge branch 'feature/archive_yaml' of github.com:DavidHuber-NOAA/glo…
DavidHuber-NOAA Apr 25, 2024
1a3ac04
Remove debug print statement
DavidHuber-NOAA Apr 25, 2024
c8ede1b
Remove 'untested' warning
DavidHuber-NOAA Apr 25, 2024
fc6f085
Corrected genesis COM path
DavidHuber-NOAA Apr 25, 2024
fa0adb7
Remove unsaved atm_stoch files
DavidHuber-NOAA Apr 25, 2024
84a1dd8
Move restart_interval* to more logical locations
DavidHuber-NOAA Apr 25, 2024
d76c392
Remove unused gfs_restartb template
DavidHuber-NOAA Apr 25, 2024
546e385
Add master template outline.
DavidHuber-NOAA Apr 30, 2024
b412f7f
Adjust earc groups by resolution
DavidHuber-NOAA Apr 30, 2024
668ca3b
Fix template constants in jinja
DavidHuber-NOAA Apr 30, 2024
5f518db
Rename save_restart variables.
DavidHuber-NOAA Apr 30, 2024
3f16f8b
Fix atardir datasets
DavidHuber-NOAA Apr 30, 2024
912f9e7
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA Apr 30, 2024
235830f
Update wxflow/jinja
DavidHuber-NOAA May 1, 2024
18c1271
Fix iaufhrs typo
DavidHuber-NOAA May 1, 2024
c99c8f4
Fix timedelta typo
DavidHuber-NOAA May 1, 2024
853167a
Apply fixes to master templates
DavidHuber-NOAA May 1, 2024
f4fc98f
Fix dataset parsing
DavidHuber-NOAA May 1, 2024
f788e6f
Add FATAL ERROR to fatal error messages
DavidHuber-NOAA May 1, 2024
13acc22
Fix whitespace
DavidHuber-NOAA May 1, 2024
c869649
Update gdas component head names
DavidHuber-NOAA May 2, 2024
4fa45b3
Flat-layer the master templates, fix wave archiving
DavidHuber-NOAA May 2, 2024
90a00e7
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 2, 2024
eda06ed
Correct CICE namelist.
DavidHuber-NOAA May 2, 2024
51b7838
Add quick fix for wgrib.
DavidHuber-NOAA May 2, 2024
95d88a9
Replace bash templates in Jinja templates.
DavidHuber-NOAA May 3, 2024
32010d0
First attempt at reenabling aerosols
DavidHuber-NOAA May 3, 2024
356be5e
Fix snow diag tarring
DavidHuber-NOAA May 3, 2024
930d013
Fix build_all comment
DavidHuber-NOAA May 3, 2024
d524483
Fix cice restart variable name
DavidHuber-NOAA May 3, 2024
cfb73fe
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 3, 2024
b79e481
Transfer gfs config fixes to gefs configs
DavidHuber-NOAA May 3, 2024
43d1aed
Fix gdas_tasks
DavidHuber-NOAA May 3, 2024
23d6563
Fix aerosol comment
DavidHuber-NOAA May 3, 2024
fa45977
Turn aerosol analyses on for gfs and gdas runs
DavidHuber-NOAA May 3, 2024
ac9128d
Use aero cdump variables to determine archiving
DavidHuber-NOAA May 3, 2024
7fd5b76
Replace waves with aerosols in config.fcst
DavidHuber-NOAA May 3, 2024
b483745
Change mandatory to required
DavidHuber-NOAA May 3, 2024
cc89c16
Make wave point files required
DavidHuber-NOAA May 3, 2024
1ec205c
Add wave IBP data
DavidHuber-NOAA May 6, 2024
d471e9e
Split archive execute method, determine commands (htar/tar, chmod, et…
DavidHuber-NOAA May 6, 2024
196db3e
Remove unused numpy, timedelta modules
DavidHuber-NOAA May 6, 2024
96ac442
Split enkfgdas and enkfgfs master templates; create one tarball at a …
DavidHuber-NOAA May 6, 2024
cfab211
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 8, 2024
c018324
Handle indention at the master level
DavidHuber-NOAA May 8, 2024
4969af8
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 9, 2024
f1b8e88
Consolidated enkf heads to master template
DavidHuber-NOAA May 9, 2024
8f8ed43
Set PYTHONPATH to head of ush/python
DavidHuber-NOAA May 9, 2024
be28de1
Clean up archive.py
DavidHuber-NOAA May 9, 2024
aaa3876
Fix ocean archiving
DavidHuber-NOAA May 9, 2024
48578f7
Removed unused COM, updated gdasocean template
DavidHuber-NOAA May 9, 2024
e450662
Explicitized gdasocean_analysis template
DavidHuber-NOAA May 9, 2024
f4cd9aa
Clean up some of the templates
DavidHuber-NOAA May 9, 2024
91b7820
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 9, 2024
d72f9b4
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 10, 2024
2e3b461
Do not archive ocean analysis on the first cycle
DavidHuber-NOAA May 10, 2024
4e9ecc0
Fix restart copies in forecast_postdet.sh
DavidHuber-NOAA May 10, 2024
c945de4
Add archive support for jedi ATM; fix reset time offsets when DOIAU=NO
DavidHuber-NOAA May 10, 2024
655c197
Reverted restart offset for enkf (does not change with IAU)
DavidHuber-NOAA May 10, 2024
af7c968
Add missing variables
DavidHuber-NOAA May 13, 2024
5543214
Fix VFYARC archiving
DavidHuber-NOAA May 13, 2024
4436a91
Fix offsets for ENKF IAU
DavidHuber-NOAA May 13, 2024
baf6da7
Fix ocean/ice GDAS products
DavidHuber-NOAA May 13, 2024
990d49b
Wrap optional files in appropriate flags
DavidHuber-NOAA May 13, 2024
fe5d68f
Use RUN instead of CDUMP for aerosols.
DavidHuber-NOAA May 13, 2024
9a0704e
Added an 1152 case for earc
DavidHuber-NOAA May 13, 2024
50305dc
Replace CDUMP with RUN for wave, aerosol configs
DavidHuber-NOAA May 13, 2024
dfcf804
Move WGRIB def to module_base
DavidHuber-NOAA May 13, 2024
0106d25
Added missing ;;.
DavidHuber-NOAA May 13, 2024
c67c36f
Fix IAUFHRS
DavidHuber-NOAA May 14, 2024
f27839c
Make gsistat mandatory; remove duplicate flux files
DavidHuber-NOAA May 14, 2024
6e74ce8
Fix ocean time step to archive
DavidHuber-NOAA May 14, 2024
0986bc0
Add missing ice IC file to archive.
DavidHuber-NOAA May 14, 2024
806ec5d
Add missing ocean CSVs.
DavidHuber-NOAA May 14, 2024
7dd2fd4
Archive *gdas hourly outputs at 3-hour intervals
DavidHuber-NOAA May 14, 2024
a9fa4e3
Add missing keys and fix bugs
DavidHuber-NOAA May 14, 2024
4795b67
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 14, 2024
c2b981c
Flesh out enkf restarts.
DavidHuber-NOAA May 16, 2024
714ab9e
Key enkf atmf netcdf files off of IAUFHRS_ENKF
DavidHuber-NOAA May 16, 2024
8f2d4af
Add missing logs and inc file to enkf*.tar
DavidHuber-NOAA May 16, 2024
1220fac
Make atma files optional, correct log name
DavidHuber-NOAA May 16, 2024
f08c2b8
Add sfcanl data files to gdas restart tarballs
DavidHuber-NOAA May 16, 2024
cfd5fbb
Move misplaced endif
DavidHuber-NOAA May 16, 2024
83b969c
Add iaufhrs_enkf definition
DavidHuber-NOAA May 16, 2024
629bf6f
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 16, 2024
7e205e7
Updating nc check script to verify time dimensions
DavidHuber-NOAA May 17, 2024
478a5ba
Ignore shellcheck warning
DavidHuber-NOAA May 17, 2024
3be3bd5
Fixed ncfile variable name
DavidHuber-NOAA May 17, 2024
4e4c0e2
Do not try to collect ecen, esfc logs on first cycle
DavidHuber-NOAA May 17, 2024
ffeaf75
Move staticmethod for compatability with older Python versions
DavidHuber-NOAA May 17, 2024
0c4d92d
Merge remote-tracking branch 'emc/develop' into feature/archive_yaml
DavidHuber-NOAA May 20, 2024
5aeccac
Separate stmp/ptmp for orion and hercules.
DavidHuber-NOAA May 21, 2024
898cc5e
Do not run arch/earc on half cycles
DavidHuber-NOAA May 21, 2024
c49567c
Reorganize enkf templates; add comments; fix increment/analysis archi…
DavidHuber-NOAA May 21, 2024
892d69d
Reorganize gdas archiving; add ensres data; add comments
DavidHuber-NOAA May 21, 2024
4496a96
Reorganize GFS archiving; add comments
DavidHuber-NOAA May 21, 2024
5d5c529
Comment arcdir archiving
DavidHuber-NOAA May 21, 2024
5777645
Add ice archiving comment
DavidHuber-NOAA May 21, 2024
b25c523
Merge remote-tracking branch 'emc/develop' into feature/archive_cleanup
DavidHuber-NOAA May 22, 2024
75c9e61
Fixed bugs
DavidHuber-NOAA May 22, 2024
e58824a
Require all jinja2 archive template variables be defined
DavidHuber-NOAA May 22, 2024
689ec37
Address shellcheck warning
DavidHuber-NOAA May 22, 2024
37d84be
Correct FHMAX_FITS
DavidHuber-NOAA May 23, 2024
0963441
Reformat arcdir.yaml.j2 to remove logic
DavidHuber-NOAA May 23, 2024
4690921
Moved fatal checks to the top of archive.py
DavidHuber-NOAA May 23, 2024
2c4cd4b
Moved file renaming so it can be sent to arcdir.
DavidHuber-NOAA May 23, 2024
0b02dbd
Merge branch 'sandbox/arcdir_reformat' into feature/archive_cleanup
DavidHuber-NOAA May 24, 2024
c48b3f0
Fix typo
DavidHuber-NOAA May 24, 2024
5481da4
Copy to arcdir in one step
DavidHuber-NOAA May 24, 2024
543257e
Add minimal logic into arcdir yaml to avoid undefined math
DavidHuber-NOAA May 24, 2024
e921f45
Grab whatever stats CSVs there are
DavidHuber-NOAA May 24, 2024
ddf111b
Merge remote-tracking branch 'emc/develop' into feature/archive_cleanup
DavidHuber-NOAA May 24, 2024
d87bd93
Merge remote-tracking branch 'emc/develop' into feature/archive_cleanup
DavidHuber-NOAA May 29, 2024
0ff1f54
Condense conditionals
DavidHuber-NOAA May 29, 2024
4e93620
Add documentation to _construct_arcdir_set
DavidHuber-NOAA May 29, 2024
98051eb
Merge branch 'develop' into feature/archive_cleanup
WalterKolczynski-NOAA May 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
196 changes: 115 additions & 81 deletions parm/archive/arcdir.yaml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -2,84 +2,118 @@
{% set cycle_YMDH = current_cycle | to_YMDH %}
{% set cycle_YMD = current_cycle | to_YMD %}
{% set head = RUN + ".t" + cycle_HH + "z." %}
{% if RUN == "gdas" or RUN == "gfs" %}
deterministic:
mkdir:
- "{{ ARCDIR }}"
{% if RUN == "gfs" %}
- "{{ ARCDIR }}/tracker.{{ cycle_YMDH }}/{{ RUN }}"
{% endif %}
{% if FITSARC %}
{% set VFYARC = ROTDIR + "/vrfyarch" %}
- "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}"
{% endif %}
copy:
{% if current_cycle != SDATE and MODE == "cycled" %}
{% if DO_JEDIATMVAR %}
- ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}atmstat", "{{ ARCDIR }}/atmstat.{{ RUN }}.{{ cycle_YMDH }}"]
{% else %}
- ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}gsistat", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}
{% if DO_JEDISNOWDA %}
- ["{{ COM_SNOW_ANALYSIS }}/{{ head }}snowstat.tgz", "{{ ARCDIR }}/snowstat.{{ RUN }}.{{ cycle_YMDH }}.tgz"]
{% endif %}
{% if AERO_ANL_CDUMP == RUN or AERO_ANL_CDUMP == "both" %}
- ["{{ COM_CHEM_ANALYSIS }}/{{ head }}aerostat", "{{ ARCDIR }}/aerostat.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}
- ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.anl", "{{ ARCDIR }}/pgbanl.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
{% endif %} # Full cycle
{% if RUN == "gfs" %}
{% set fhmax, fhout = FHMAX_GFS, FHOUT_GFS %}
{% elif RUN == "gdas" %}
{% set fhmax, fhout = FHMAX, FHOUT %}
{% endif %}
{% for fhr in range(0, fhmax + fhout, fhout) %}
- ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.f{{ '%03d' % fhr }}", "{{ ARCDIR }}/pgbf{{ '%02d' % fhr }}.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
{% endfor %}
{% if RUN == "gfs" %}
{% if FITSARC %}
{% if FHMAX_FITS is defined %}
{% set fhmax = FHMAX_FITS %}
{% else %}
{% set fhmax = FHMAX_GFS %}
{% endif %}
{% for fhr in range(0, fhmax + 6, 6) %}
{% set sfcfile = "/" + head + "sfcf" + '%03d'|format(fhr) + ".nc" %}
{% set sigfile = "/" + head + "atmf" + '%03d'|format(fhr) + ".nc" %}
- ["{{COM_ATMOS_HISTORY}}/{{ sfcfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sfcfile }}"]
- ["{{COM_ATMOS_HISTORY}}/{{ sigfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sigfile }}"]
{% endfor %}
{% endif %} ## FITSARC
{% if path_exists(COM_ATMOS_GENESIS ~ "/storms.gfso.atcf_gen." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}"]
{% endif %}
{% if path_exists(COM_ATMOS_GENESIS ~ "/trak.gfso.atcfunix." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}"]
{% endif %}
## Only created if tracking is on and there were systems to track
{% for basin in ["epac", "natl"] %}
{% if path_exists(COM_ATMOS_TRACK + "/" + basin) %}
- ["{{ COM_ATMOS_TRACK }}/{{ basin }}", "{{ ARCDIR }}/{{ basin }}"]
{% endif %}
{% endfor %}
{% endif %} ## RUN == "gfs"
{% if path_exists(COM_ATMOS_TRACK ~ "/atcfunix." ~ RUN ~ "." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_TRACK }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_TRACK }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}
{% endif %} # gfs or gdas
{% if current_cycle != SDATE and (RUN == "enkfgdas" or RUN == "enkfgfs") %}
ensemble:
mkdir:
- "{{ ARCDIR }}"
copy:
{% if DO_JEDIATMENS %}
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atmensstat", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atminc.ensmean.nc", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}.ensmean.nc"]
{% else %}
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}enkfstat", "{{ ARCDIR }}/enkfstat.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}gsistat.ensmean", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}.ensmean"]
{% endif %}
{% endif %} # enkfgdas or enkfgfs

# Select data to store in the ARCDIR and VFYARC from deterministic runs
Base: &base # GDAS, GFS, ENKFGDAS, or ENKFGFS
common:
mkdir:
- "{{ ARCDIR }}"

# Common files to be added to both the gfs and gdas keys below
Deterministic: &deterministic
cyclone:
copy:
# Cyclone forecasts, produced for both gdas and gfs cycles
## Only created if tracking is on and there were systems to track
{% if path_exists(COM_ATMOS_TRACK ~ "/atcfunix." ~ RUN ~ "." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_TRACK }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunix.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_TRACK }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}", "{{ ARCDIR }}/atcfunixp.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}

# Cyclone tracking data
{% for basin in ["epac", "natl"] %}
{% if path_exists(COM_ATMOS_TRACK + "/" + basin) %}
- ["{{ COM_ATMOS_TRACK }}/{{ basin }}", "{{ ARCDIR }}/{{ basin }}"]
{% endif %}
{% endfor %}

{% if MODE == "cycled" %}
analysis:
copy:
# Analysis data (if we are running in cycled mode)
- ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.anl", "{{ ARCDIR }}/pgbanl.{{ RUN }}.{{ cycle_YMDH }}.grib2"]

{% if DO_JEDIATMVAR %}
- ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}atmstat", "{{ ARCDIR }}/atmstat.{{ RUN }}.{{ cycle_YMDH }}"]
{% else %}
- ["{{ COM_ATMOS_ANALYSIS }}/{{ head }}gsistat", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}

{% if DO_JEDISNOWDA %}
- ["{{ COM_SNOW_ANALYSIS }}/{{ head }}snowstat.tgz", "{{ ARCDIR }}/snowstat.{{ RUN }}.{{ cycle_YMDH }}.tgz"]
{% endif %}

{% if AERO_ANL_CDUMP == RUN or AERO_ANL_CDUMP == "both" %}
- ["{{ COM_CHEM_ANALYSIS }}/{{ head }}aerostat", "{{ ARCDIR }}/aerostat.{{ RUN }}.{{ cycle_YMDH }}"]
{% endif %}

{% endif %}

{% if RUN == "gfs" %}
gfs: # GFS specific
<<: *base
<<: *deterministic

gfs:
copy:
{% for fhr in range(0, FHMAX_GFS + 1, FHOUT_GFS) %}
- ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.f{{ '%03d' % fhr }}", "{{ ARCDIR }}/pgbf{{ '%02d' % fhr }}.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
{% endfor %}

# Cyclone genesis data (only present if there are storms)
{% if path_exists(COM_ATMOS_GENESIS ~ "/storms.gfso.atcf_gen." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_GENESIS }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/storms.gfso.atcf_gen.altg.{{ cycle_YMDH }}"]
{% endif %}

{% if path_exists(COM_ATMOS_GENESIS ~ "/trak.gfso.atcfunix." ~ cycle_YMDH) %}
- ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_GENESIS }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}", "{{ ARCDIR }}/trak.gfso.atcfunix.altg.{{ cycle_YMDH }}"]
{% endif %}

{% if DO_FIT2OBS %}
fit2obs:

mkdir:
{% set VFYARC = ROTDIR + "/vrfyarch" %}
- "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}"

copy:
{% for fhr in range(0, FHMAX_FITS + 1, 6) %}
{% set sfcfile = "/" + head + "sfcf" + '%03d'|format(fhr) + ".nc" %}
{% set sigfile = "/" + head + "atmf" + '%03d'|format(fhr) + ".nc" %}
- ["{{COM_ATMOS_HISTORY}}/{{ sfcfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sfcfile }}"]
- ["{{COM_ATMOS_HISTORY}}/{{ sigfile }}", "{{ VFYARC }}/{{ RUN }}.{{ cycle_YMD }}/{{ cycle_HH }}/{{ sigfile }}"]
{% endfor %}

{% endif %}
{% endif %}

{% if RUN == "gdas" %}
gdas: # GDAS specific
<<: *base
<<: *deterministic
gdas:
copy:
{% for fhr in range(0, FHMAX + 1, FHOUT) %}
- ["{{ COM_ATMOS_GRIB_1p00 }}/{{ head }}pgrb2.1p00.f{{ '%03d' % fhr }}", "{{ ARCDIR }}/pgbf{{ '%02d' % fhr }}.{{ RUN }}.{{ cycle_YMDH }}.grib2"]
{% endfor %}
{% endif %}

Ensemble: &ensemble # ENKFGDAS or ENKFGFS
analysis:
copy:
# Copy ensemble analyses
{% if DO_JEDIATMENS %}
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atmensstat", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}atminc.ensmean.nc", "{{ ARCDIR }}/atmensstat.{{ RUN }}.{{ cycle_YMDH }}.ensmean.nc"]
{% else %}
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}enkfstat", "{{ ARCDIR }}/enkfstat.{{ RUN }}.{{ cycle_YMDH }}"]
- ["{{ COM_ATMOS_ANALYSIS_ENSSTAT }}/{{ head }}gsistat.ensmean", "{{ ARCDIR }}/gsistat.{{ RUN }}.{{ cycle_YMDH }}.ensmean"]
{% endif %}

{% if "enkf" in RUN %}
{{ RUN }}:
<<: *base
<<: *ensemble
{% endif %}
88 changes: 48 additions & 40 deletions parm/archive/enkf.yaml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -2,24 +2,46 @@ enkf:
name: "ENKF"
target: "{{ ATARDIR }}/{{ cycle_YMDH }}/{{ RUN }}.tar"
required:
# Logs
{% for mem in range(1, nmem_ens + 1) %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}fcst_mem{{ '%03d' % mem }}.log"
{% endfor %}
{% for fhr in range(fhmin, fhmax + 1, fhout) %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}epos{{ '%03d' % (fhr - fhmin) }}.log"
{% endfor %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}echgres.log"
- "logs/{{ cycle_YMDH }}/{{ RUN }}esfc.log"
{% for grp in range(iaufhrs | length) %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}ecen{{ '%03d' % grp }}.log"
{% endfor %}

{% if DO_JEDIATMENS %}
{% set steps = ["atmensanlinit", "atmensanlrun", "atmensanlfinal"] %}
{% else %}
{% set steps = ["eobs", "eupd"] %}
{% if lobsdiag_forenkf %}
{% do steps.append("ediag") %}
{% else %}
{% for mem in range(1, nmem_ens + 1) %}
{% do steps.append("eomg_mem{{ '%03d' % mem }}") %}
{% endfor %}
{% endif %}
{% endif %}

{% for step in steps %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}{{ step }}.log"
{% endfor %}

# Ensemble mean and spread
{% for fhr in range(3, fhmax + 1, 3) %}
- "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensmean.nc"
- "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}sfcf{{ '%03d' % fhr }}.ensmean.nc"
{% if ENKF_SPREAD %}
- "{{ COM_ATMOS_HISTORY_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmf{{ '%03d' % fhr }}.ensspread.nc"
{% endif %}
{% endfor %}
{% for mem in range(1, nmem_ens + 1) %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}fcst_mem{{ '%03d' % mem }}.log"
{% endfor %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}epos*.log"
- "logs/{{ cycle_YMDH }}/{{ RUN }}echgres.log"

{% if current_cycle != SDATE %}
# TODO archive ecen logs based on actual groups. Will need to emulate numpy.array_split to do so.
- "logs/{{ cycle_YMDH }}/{{ RUN }}ecen*.log"
- "logs/{{ cycle_YMDH }}/{{ RUN }}esfc.log"

# Ensemble mean state
{% if not DO_JEDIATMENS %}
{% set da_files = ["enkfstat",
"gsistat.ensmean",
Expand All @@ -34,41 +56,27 @@ enkf:
{% for file in da_files %}
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}{{ file }}"
{% endfor %}
{% if DOIAU %}
{% for fhr in iaufhrs %}
{% if fhr == IAU_OFFSET %}

# Ensemble mean analyses/increments
# 6-hr analysis/increment
{% if do_calc_increment %}
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmanl.ensmean.nc"
{% endif %} # calc increment
{% else %} # fhr != IAU_OFFSET
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc"
{% endif %} # fhr == IAU_OFFSET
{% endfor %} # IAUFHRS
{% endif %} # DOIAU

{% if DO_JEDIATMENS %}
{% set steps = ["atmensanlinit", "atmensanlrun", "atmensanlfinal"] %}
{% else %}
{% set steps = ["eobs", "eupd"] %}
{% if lobsdiag_forenkf %}
{% do steps.append("ediag") %}
{% else %}
{% for mem in range(1, nmem_ens + 1) %}
{% do steps.append("eomg_mem{{ '%03d' % mem }}") %}
{% endfor %}
{% endif %}
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atminc.ensmean.nc"
{% endif %}

{% for step in steps %}
- "logs/{{ cycle_YMDH }}/{{ RUN }}{{ step }}.log"
{% endfor %}
{% endif %} # not the first cycle
optional:
{% if current_cycle != SDATE and DOIAU %}
{% for fhr in iaufhrs %}
{% if fhr != IAU_OFFSET %}
{% if DOIAU %}
# IAU increments/analyses

{% for fhr in iaufhrs if fhr != 6 %}
{% if do_calc_increment %}
# Store analyses instead of increments
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atma{{ '%03d' % fhr }}.ensmean.nc"
{% else %}
# Store increments
- "{{ COM_ATMOS_ANALYSIS_ENSSTAT | relpath(ROTDIR) }}/{{ head }}atmi{{ '%03d' % fhr }}.ensmean.nc"
{% endif %} # fhr == IAU_OFFSET
{% endfor %} # IAUFHRS
{% endif %}
{% endfor %}

{% endif %} # End IAU
# End of analysis mean increments/analyses
12 changes: 6 additions & 6 deletions parm/archive/enkf_grp.yaml.j2
Original file line number Diff line number Diff line change
Expand Up @@ -4,19 +4,20 @@ enkf_grp:
required:
{% for mem in range(first_group_mem, last_group_mem + 1) %}
{% set imem = mem - first_group_mem %}
# Construct member COM directories
{% set COM_ATMOS_ANALYSIS_MEM = COM_ATMOS_ANALYSIS_MEM_list[imem] %}
{% set COM_ATMOS_HISTORY_MEM = COM_ATMOS_HISTORY_MEM_list[imem] %}
{% set COM_ATMOS_RESTART_MEM = COM_ATMOS_RESTART_MEM_list[imem] %}

{% for iaufhr in iaufhrs_enkf %}
- "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}atmf{{ "%03d" % iaufhr }}.nc"
# Forecast data
{% for fhr in range(3, 10, 3) %}
- "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}atmf{{ "%03d" % fhr }}.nc"
{% endfor %}

{% if 6 in iaufhrs_enkf %}
# Only store the 6-hour surface forecast
- "{{ COM_ATMOS_HISTORY_MEM | relpath(ROTDIR) }}/{{ head }}sfcf006.nc"
{% endif %}

{% if current_cycle != SDATE %}
# Store the individual member analysis data
{% if not lobsdiag_forenkf %}
- "{{ COM_ATMOS_RESTART_MEM | relpath(ROTDIR) }}/{{ head }}gsistat"
{% endif %}
Expand All @@ -25,5 +26,4 @@ enkf_grp:
{% else %}
- "{{ COM_ATMOS_ANALYSIS_MEM | relpath(ROTDIR) }}/{{ head }}ratminc.nc"
{% endif %}
{% endif %}
{% endfor %} # first_group_mem to last_group_mem
Loading
Loading