Sync develop branch with EMC develop #34
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…1498) * ignore archiving certain sfluxgrbf??? files when they are not present. Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>
Changes the GDASApp hash to a more stable version than the tip of develop. See post-merge converastion in #1506
Reorganizes the entire COM directory into a more hierarchical structure and uses centrally-defined templates to define COM paths.
## Hierarchical Structure
To organize output a lot better and not have 30000+ files in a single directory, all of the component COM directories are divided into a number of subdirectories for each type of output.
Sample directory trees
### Cycled atmosphere only
```
gdas.20211222/00
├── analysis
│ └── atmos
│ └── gsidiags
│ ├── dir.0000
│ ├── dir.0001
│ ├── (Additional dir.* directories omitted for brevity)
│ └── dir.0083
├── model_data
│ └── atmos
│ ├── history
│ ├── master
│ └── restart
├── obs
└── products
└── atmos
├── cyclone
│ └── tracks
└── grib2
├── 0p25
├── 0p50
└── 1p00
101 directories
```
```
enkfgdas.20211222/00
├── earc00
├── ensstat
│ ├── analysis
│ │ └── atmos
│ │ └── gsidiags
│ │ ├── dir.0000
│ │ ├── dir.0001
│ │ ├── (Additional dir.* omitted for brevity)
│ │ └── dir.0039
│ └── model_data
│ └── atmos
│ └── history
├── mem001
│ ├── analysis
│ │ └── atmos
│ └── model_data
│ └── atmos
│ ├── history
│ ├── master
│ └── restart
└── mem002
├── analysis
│ └── atmos
└── model_data
└── atmos
├── history
├── master
└── restart
64 directories
```
```
gfs.20211222/00
├── analysis
│ └── atmos
├── model_data
│ └── atmos
│ ├── history
│ ├── master
│ └── restart
├── obs
└── products
└── atmos
├── bufr
├── cyclone
│ ├── genesis_vital
│ └── tracks
├── gempak
│ ├── 0p25
│ ├── 0p50
│ ├── 1p00
│ ├── 35km_atl
│ ├── 35km_pac
│ └── 40km
├── grib2
│ ├── 0p25
│ ├── 0p50
│ └── 1p00
└── wmo
26 directories
```
```
enkfgfs.20211222/00
├── earc00
├── ensstat
│ ├── analysis
│ │ └── atmos
│ │ └── gsidiags
│ │ ├── dir.0000
│ │ ├── dir.0001
│ │ ├── (Additional dir.* directories removed for brevity)
│ │ └── dir.0039
│ └── model_data
│ └── atmos
│ └── history
├── mem001
│ ├── analysis
│ │ └── atmos
│ └── model_data
│ └── atmos
│ ├── history
│ ├── master
│ └── restart
└── mem002
├── analysis
│ └── atmos
└── model_data
└── atmos
├── history
├── master
└── restart
64 directories
```
### S2SWA coupled
prototype (forecast-only):
```
gfs.20130401/00/
├── model_data
│ ├── atmos
│ │ ├── history
│ │ ├── input
│ │ ├── master
│ │ └── restart
│ ├── chem
│ │ └── history
│ ├── ice
│ │ ├── history
│ │ ├── input
│ │ └── restart
│ ├── med
│ │ └── restart
│ ├── ocean
│ │ ├── history
│ │ ├── input
│ │ └── restart
│ └── wave
│ ├── history
│ ├── prep
│ └── restart
└── products
├── atmos
│ ├── cyclone
│ │ ├── genesis_vital
│ │ └── tracks
│ ├── gempak
│ │ ├── 0p25
│ │ ├── 0p50
│ │ ├── 1p00
│ │ ├── 35km_atl
│ │ ├── 35km_pac
│ │ └── 40km
│ ├── grib2
│ │ ├── 0p25
│ │ ├── 0p50
│ │ └── 1p00
│ └── wmo
├── ocean
│ ├── 2D
│ ├── 3D
│ ├── grib
│ │ ├── 0p25
│ │ └── 0p50
│ └── xsect
└── wave
├── gempak
├── gridded
├── station
└── wmo
51 directories
```
### Trees with files
gdas: https://gist.github.com/WalterKolczynski-NOAA/f1de04901e2703fd24d38146d2669789
gfs: https://gist.github.com/WalterKolczynski-NOAA/5d1b7c0a0f4b8cfff0be1ae54082316a
enkfgdas: https://gist.github.com/WalterKolczynski-NOAA/860aaa804e3e70e191e7cae2ebb1055b
enkfgfs: https://gist.github.com/WalterKolczynski-NOAA/130bfff4650ed8b07cf395079b65d318
S2SWA P8: https://gist.github.com/WalterKolczynski-NOAA/6ae90c6eafb573878f60682ce47179db
## Templating
All of the COM paths have been replaced with new variables that are derived from a set of templates centrally defined in `config.com`. Variables in the templates are then substituted at runtime to generate the COM paths via the use of `envsubst`. To facilitate this, there is a new function, `generate_com` (see below), provided to automatically generate the COM paths.
Where possible, COM paths are defined at the j-job level and made read-only. However, many of the EnKF scripts loop over the ensemble members, forcing the definitions to be made at the exscript level instead (and be mutable).
The arguments to `generate_com()` are the list of COM variables to generate, optionally accompanied by a template to use using a colon to separate them. When no template is specified, the variable will be generated using the ${varname}_TMPL template. Two options are accepted, `-r` and `-x`, which will mark the variable as read-only and for export, respectively (the same as with the `declare` builtin). It is best practice to define any additional variables needed by the template on the same line to avoid adding them to the calling script’s scope. Here are some examples used in the code:
Generate the path to the atmos analysis directory for the current cycle and `$RUN` (implicitly from the `$COM_ATMOS_ANALYSIS_TMPL` template) and mark as read-only and export:
```
YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS
```
Generate the path to the atmos history directory for the previous cycle's gdas from the `$COM_ATMOS_HISTORY_TMPL` template and mark as read-only and export:
```
RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \
COM_ATMOS_HISTORY_PREV:COM_ATMOS_HISTORY_TMPL
```
Generate the path to the first ensemble member's history directory of the current cycle and `$RUN` and mark for export:
```
MEMDIR=’mem001’ YMD=${PDY} HH=${cyc} generate_com -x COM_ATMOS_HISTORY
```
## Additional information
The staging of initial conditions in `setup_expy.py` has been updated to stage in the new locations. The source of the initial conditions can **either** be in the new hierarchical structure or in the old flat structure and the script will stage the files in the new structure. The destination paths are hard-coded here, so if any changes are made to the analysis, input, or restart templates, they will need to be mirrored in `setup_expy.py`.
### Stipulations
All changes in this PR are subject to approval by several stakeholders, including NCO. Sample COM trees above are subject to revision based on feedback (for instance, file X isn't really an obs file).
File name updates are not included in this PR. File names (primarily for coupled components) will be updated to comply with NCO standards in a future PR.
AWIPS jobs are now almost working (they do not in current develop), but one last program is still ending with an error.
Work on fit2obs is deferred, so that portion of the verify job does not work.
WAFS scripts are all external and have not yet been updated. WAFS is expected to be packaged separately going forward, so will need to be updated like any other downstream package.
Some scripts that are not part of our normal development workflow have not yet been updated. I may be able to knock a few more off this list, but some just aren’t available in development mode currently:
- All UFSDA app jobs (to be handled separately)
- With associated dev jobs (may still modify and test)
- JGDAS_ATMOS_GLDAS
- ~~JGLOBAL_WAVE_GEMPAK~~
- ~~JGLOBAL_WAVE_POST_BNDPNT~~
- ~~JGLOBAL_WAVE_POST_BNDPNTBLL~~
- ~~JGLOBAL_WAVE_PRDGEN_BULLS~~
- ~~JGLOBAL_WAVE_PRDGEN_GRIDDED~~
- ~~JGLOBAL_WAVE_PREP~~
- With no associated dev job
- JGDAS_ATMOS_GEMPAK_META_NCDCJGFS_ATMOS_FBWIND
- JGFS_ATMOS_FSU_GENESIS
- JGFS_ATMOS_GEMPAK_META
- JGFS_ATMOS_GEMPAK_NCDC_UPAPGIF
- JGLOBAL_ATMOS_EMCSFC_SFC_PREP
- JGLOBAL_ATMOS_POST_MANAGER
- JGLOBAL_ATMOS_TROPCY_QC_RELOC
+ All downstream scripts for the above
There are also a few scripts that are not available to the development workflow that I have already made a good-faith effort at updating:
- JGDAS_ATMOS_GEMPAK
- JGFS_ATMOS_PGRB2_SPEC_NPOESS
## Related Issues
Closes #761
Fixes #978
Fixes #999
Fixes #1207
Partially addresses #198
Partially addresses #289
Partially addresses #293
Partially addresses #1299
Partially addresses #1326
Turns off trace for the duration of the generate_com function unless DEBUG_WORKFLOW is not set to "NO" (the default). In its place, the function will now echo the assignment. Closes #1524
A typo led to the same variable being defined twice instead of the variable that was supposed to be defined. Fixes #1530
The trailing '_TMPL' was accidentally ommited from the ocean history template used during workflow generation. Refs: #1532
Removes most of content of JGDAS_GLOBAL_OCEAN_ANALYSIS_POST to scripts/exgdas_global_marine_analysis_post.py in GDASApp, now just calls that script. Addresses first bullet of #1480
Found some odd bugs in the aerosol DA changes that need to be fixed to work properly. Not sure how they worked before with testing (by luck?) but these are straightforward fixes to implement. one has a missing / in a path for FileHandler and the other has a missing .nc causing files to not be found
The path for ocean analysis files was not properly updated after analysis was moved out of model_data into its own directory.
This PR updates the j-jobs and python classes for aerosol DA to use the new COM directory structure. This PR also includes removing of the chem history staging for the ICSDIR in setup_expt.py. The aerosol fields are treated as FV3 tracers, so they are either available (warm start) with the fv_tracer files or start at 0 (cold start) and must be spun up from emissions. Fixes #1516
Updates the UFS model hash to the version as of 2023 Apr 17. Some associated changes accompany this update - Restart filenames for MOM6 (ufs-community/ufs-weather-model#1599) - Remove store coriolis setting from MOM6 namelist (ufs-community/ufs-weather-model#1599) - Change in atm 'log' file names (ufs-community/ufs-weather-model#1704) - Additions to diag_table for frozen species (ufs-community/ufs-weather-model#1529) - Restart quilting (ufs-community/ufs-weather-model#1633) - Update to post itag (ufs-community/ufs-weather-model#1690) The switch to restart quilting adds an additional constraint on the size of write groups to be divisible by the number of tiles, so all were increased to the next multiple of 6. In the process of updating the diag tables, unused tables were removed. Closes #1279 as moot Closes #1445 Closes #1499 Partially addresses #1277
`$WAVE_RUN` is no longer used in workflow, but one reference remained in the archive job, which would cause failures. The conditional did not need to be replaced by `$RUN` since `$RUN` has already been checked at that point. Fixes #1548
* Remove bump_ver from versions/fix.ver - No longer need a bump version variable in fix.ver. Refs #1552
When the workflow was updated to use ESMF threading, the CICE namelist was not updated to use the new ntasks_cic6 variable instead of ICEPETS. This caused the incorrect number to be used when using multiple threads. Fixes #1549
* Update init.rst cycled coupled IC info * The cycled ATM w/ coupled (S2S) model ICs were updated and the folder was renamed from "C48mx500" to "C48C48mx500" * Added note to mention that the EnKF member ICs are currently just dummy duplicates of the deterministic.
…1564) This PR: - updates the j-jobs to accommodate post COM refactor upgrades. - minor updates to config.resources for land DA tasks - add task dependencies in the rocoto workflow - when doing LandDA, use sfc_data.tile{{ t }}.nc created by Land DA is used as input of global_cycle for the deterministic and case w/ DOIAU=NO.
Begin using feature-GFSv17_com_reorg_log_update install which updated the atm logf filename checked for by prepobs getges_nc.sh. Refs #1570
The section on commit message standards is updated to be for pull requests instead. Additional messaging about using the provided template is added, and bullets not relevant to PRs (line-wrapping and leaving a space between subject and body) are removed.
This PR updates the UFS_UTILS hash to get `gdas_init` updates for the recent COM reorg. A few additional UFS_UTILS commits are also included but aren't significant changes, as noted by @GeorgeGayno-NOAA: "Updates since that hash have been mostly small things like bug fixes. However, the w3nco library was replaced by the w3emc library at ufs-community/UFS_UTILS@7efbe0f". Updates to improve repo build are also included with this hash (ufs-community/UFS_UTILS@b4900c4). New hash from completion of ufs-community/UFS_UTILS#820 Resolves #1527
This PR removes: - `getic.sh` - job used to get initial conditions from HPSS from a past experiment/ops - `init.sh` - job used to create experiment resolution specific initial conditions from the ones fetched by `getic.sh`. Both these jobs were primarily used by the atmosphere model component developers in the past. They have since been staging pre-generated ICs making these jobs obsolete. Documentation will point to ufs-utils to generate the initial conditions. The utilities from ufs-utils were used to identify the appropriate HPSS tarball as well as performing interpolation for the atmospheric model component. There is no such utility available for the other model components making these scripts only partially useful. In a future PR, an update to the definition of `BASE_CPLIC` and `coupled_ic` jobs will be made to make use of `--icsdir` to point to pre-staged locations of initial conditions. Fixes #1569
Fixes a couple bugs and does a little cleanup of the COM refactor for marine DA. COM variable definitions are also updated to match the style used in other scripts.
On WCOSS2 the grib_util module is named a bit different and needs the libjpeg module loaded beforehand.
This PR adds the `--cpus-per-task` flag to non-CFP `APRUN`/srun commands in the env files for Orion and Hera. This is needed as a result of an upgrade to SLURM on Orion that no longer exported the thread value to the environment. Issue #1996 was a hotfix to add this flag for the analysis job that was suddenly running slowly after the SLURM upgrade. A similar SLURM upgrade was done on Hera but the sysadmins there implemented a workaround so this issue did not appear there. This PR adds the `--cpus-per-task` flag for both machines...needed on Orion and as a safety measure on Hera. Also included in this PR is a fix to indentations in the `scripts/exglobal_diag.sh` script that was noticed during work for this PR. Timing differences were not observed for other jobs after adding this srun flag. Job timings were of a similar variation between control and test runs. Any potential additional speedup was not observed. The test runs also reproduced the control runs with `develop`. Resolves #2044
Adds the ability to turn off some coupled components for the ensemble. To enable this, many settings are moved out of `config.fcst` and into `config.ufs` so they are calculated after any overrides of `DO_*` have been made. Some settings also had to be moved out of `config.base` for the same reason. A switch to turn on gocart is added to `config.ufs`, but unlike the other switches this one takes no arguments. The name of the ufs configure template used is converted from a partial file name to the full file name. Resolves #1692
This PR addresses issue #1225. The following was accomplished: - A new Rocoto job is added; `jobs/rocoto/npoess.sh`; - `workflow/applications/gfs_cycled.py` is updated to include the `npoess` task; - `workflow/applications/gfs_forecast_only.py` is updated to include the `npoess` task; - A new function is added to `workflow/rocoto/gfs_tasks.py`, `npoess` is added as new GFS (only) task; - `workflow/rocoto/tasks.py` is updated to include the `npoess` task. Resolves #1225
This PR addresses issue #1221. The following was accomplished: - A new Rocoto job is added; `jobs/rocoto/fbwinds.sh`; - `workflow/applications/gfs_cycled.py` is updated to include the `fbwinds` task; - `workflow/applications/gfs_forecast_only.py` is updated to include the `fbwinds` task; - A new function is added to `workflow/rocoto/gfs_tasks.py`, `fbwinds` is added as new GFS (only) task; - `workflow/rocoto/tasks.py` is updated to include the `fbwinds` task. Resolves #1221
Creates a proper j-job for the metp job, moving much of the material from the `jobs/rocoto` script there (some of it within `jjob_header`). Also updates the old `COMIN` variable for the COM refactor (although it does not appear used in the verify scripts currently). This is a partial step towards restoring/improving metp functionality and remains untested. Refs #1575
Add execute permissions (755)
Adds the wave init job to GEFS. Since the output would be the same for every member, the job is only run once for the control, then links are created in each of the member directories pointing to that copy. New ww3_multi and ww3_shell templates for GEFS are copied from the GFS versions. The GEFS restart frequency is changed to not be zero, and the comment about not setting it to zero is copied from the gfs config.base. The restart interval for the GEFS perturbed members is changed to equal that of the control. Also fixes a bug where `RUNwave` was not defined for ensemble forecasts, and another bug in the GEFS stage_ic dependency. Resolves #2017
#2084) Update module files to spack-stack/1.5.1 and compilers to Intel 2022+ on all non-production machines.
Improved CI robustness for reverting back to **CI-Ready** from any given state New Features: - Improved `scancel` routine (refactored into bash "subroutine") - Improved messaging (see below) when ever a user changes state - Any and all previous build scripts and running experiments are killed as a result of reset to **Ready** Resolves #2060
This PR combines the GDAS and GFS versions of the VMINMON JJOB and ex-driver scripts into single GLOBAL/global versions. Additionally: * some further cleanup of the GSI-Monitor scripts is done; variable settings in ex-driver scripts are moved up to the JJOB level * the fix symlinks are consolidated into a new `fix/mon` subfolder (formerly `fix/gdas`) and copies of the gfs monitor fix files under `fix/products` are removed Resolves #1925
This PR addresses issue #1228. The following is accomplished: - Separate rocoto jobs have been created beneath `jobs/rocoto/` -- `awips_20sh` and `awips_g2sh`; these jobs replace `awips.sh` which was calling multiple J-jobs within the respective `awips.sh` scripts; - New tasks has been added to `workflow/rocoto/tasks.py` for the new AWIPS scripts; - The `gfs_cycled` and `gfs_forecast_only` modules beneath `workflow/rocoto` have been updated accordingly. Resolves #1228
This PR:
- Replaces the use of `JGLOBAL_ATMOS_POST` with `JGLOBAL_ATMOS_UPP`
and `JGLOBAL_ATMOS_PRODUCTS`.
- Introduces `JGLOBAL_ATMOS_PRODUCTS` that is responsible for creating
grib2 products from the master file produced by UPP (inline from the model,
or offline by running `JGLOBAL_ATMOS_UPP`). Rocoto job `atmos_products.sh`,
ex-script `exglobal_atmos_products.sh` are also added along with a
`config.atmos_products`. Updates are made to the relevant `env` files as well as
`config.resources` to reflect the addition of this job.
- reduces the number of cores needed appropriately for forecast products when
there is no offline UPP necessary.
- renames older `fv3gfs_dwn_nems.sh` to `interp_atmos_master.sh` and
`inter_flux.sh` to `interp_atmos_sflux.sh`
- removes `fv3gfs_downstream_nems.sh` as it is absorbed into
`exglobal_atmos_products.sh`
- removes no longer used scripts for `gfs_post.sh`, `gfs_transfer.sh` (DBN alerts
are absorbed into `exglobal_atmos_products.sh`), `post.sh`,
`JGLOBAL_ATMOS_POST`, `ex${RUN}_atmos_post.sh` scripts.
- Rocoto workflow related scripts are updated to add `atmos_products` as a job.
When `WRITE_DOPOST` is `.false.`, `upp.sh` job to run the UPP offline is also
added. Dependencies have been appropriately applied.
- removes `PGB1F` option that creates 1-degree grib1 products that are no longer
needed in GFSv17.
The hacks from (now deleted) `post.sh` had to be reinstated in `upp.sh` to load the
modules from UPP instead of `load_fv3gfs_modules.sh`.
This add forecast-only support for Hercules to the global workflow. Partially satisfies #1588. Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>
jderrico-noaa
requested review from
christinaholtNOAA and
NaureenBharwaniNOAA
NaureenBharwaniNOAA
approved these changes
Dec 11, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
sync the main develop branch of GSL global-workflow fork with authoritative EMC global-workflow repository