Sync develop branch with EMC develop #34

…1498) * ignore archiving certain sfluxgrbf??? files when they are not present. Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

Changes the GDASApp hash to a more stable version than the tip of develop. See post-merge converastion in #1506

Reorganizes the entire COM directory into a more hierarchical structure and uses centrally-defined templates to define COM paths. ## Hierarchical Structure To organize output a lot better and not have 30000+ files in a single directory, all of the component COM directories are divided into a number of subdirectories for each type of output. Sample directory trees ### Cycled atmosphere only ``` gdas.20211222/00 ├── analysis │ └── atmos │ └── gsidiags │ ├── dir.0000 │ ├── dir.0001 │ ├── (Additional dir.* directories omitted for brevity) │ └── dir.0083 ├── model_data │ └── atmos │ ├── history │ ├── master │ └── restart ├── obs └── products └── atmos ├── cyclone │ └── tracks └── grib2 ├── 0p25 ├── 0p50 └── 1p00 101 directories ``` ``` enkfgdas.20211222/00 ├── earc00 ├── ensstat │ ├── analysis │ │ └── atmos │ │ └── gsidiags │ │ ├── dir.0000 │ │ ├── dir.0001 │ │ ├── (Additional dir.* omitted for brevity) │ │ └── dir.0039 │ └── model_data │ └── atmos │ └── history ├── mem001 │ ├── analysis │ │ └── atmos │ └── model_data │ └── atmos │ ├── history │ ├── master │ └── restart └── mem002 ├── analysis │ └── atmos └── model_data └── atmos ├── history ├── master └── restart 64 directories ``` ``` gfs.20211222/00 ├── analysis │ └── atmos ├── model_data │ └── atmos │ ├── history │ ├── master │ └── restart ├── obs └── products └── atmos ├── bufr ├── cyclone │ ├── genesis_vital │ └── tracks ├── gempak │ ├── 0p25 │ ├── 0p50 │ ├── 1p00 │ ├── 35km_atl │ ├── 35km_pac │ └── 40km ├── grib2 │ ├── 0p25 │ ├── 0p50 │ └── 1p00 └── wmo 26 directories ``` ``` enkfgfs.20211222/00 ├── earc00 ├── ensstat │ ├── analysis │ │ └── atmos │ │ └── gsidiags │ │ ├── dir.0000 │ │ ├── dir.0001 │ │ ├── (Additional dir.* directories removed for brevity) │ │ └── dir.0039 │ └── model_data │ └── atmos │ └── history ├── mem001 │ ├── analysis │ │ └── atmos │ └── model_data │ └── atmos │ ├── history │ ├── master │ └── restart └── mem002 ├── analysis │ └── atmos └── model_data └── atmos ├── history ├── master └── restart 64 directories ``` ### S2SWA coupled prototype (forecast-only): ``` gfs.20130401/00/ ├── model_data │ ├── atmos │ │ ├── history │ │ ├── input │ │ ├── master │ │ └── restart │ ├── chem │ │ └── history │ ├── ice │ │ ├── history │ │ ├── input │ │ └── restart │ ├── med │ │ └── restart │ ├── ocean │ │ ├── history │ │ ├── input │ │ └── restart │ └── wave │ ├── history │ ├── prep │ └── restart └── products ├── atmos │ ├── cyclone │ │ ├── genesis_vital │ │ └── tracks │ ├── gempak │ │ ├── 0p25 │ │ ├── 0p50 │ │ ├── 1p00 │ │ ├── 35km_atl │ │ ├── 35km_pac │ │ └── 40km │ ├── grib2 │ │ ├── 0p25 │ │ ├── 0p50 │ │ └── 1p00 │ └── wmo ├── ocean │ ├── 2D │ ├── 3D │ ├── grib │ │ ├── 0p25 │ │ └── 0p50 │ └── xsect └── wave ├── gempak ├── gridded ├── station └── wmo 51 directories ``` ### Trees with files gdas: https://gist.github.com/WalterKolczynski-NOAA/f1de04901e2703fd24d38146d2669789 gfs: https://gist.github.com/WalterKolczynski-NOAA/5d1b7c0a0f4b8cfff0be1ae54082316a enkfgdas: https://gist.github.com/WalterKolczynski-NOAA/860aaa804e3e70e191e7cae2ebb1055b enkfgfs: https://gist.github.com/WalterKolczynski-NOAA/130bfff4650ed8b07cf395079b65d318 S2SWA P8: https://gist.github.com/WalterKolczynski-NOAA/6ae90c6eafb573878f60682ce47179db ## Templating All of the COM paths have been replaced with new variables that are derived from a set of templates centrally defined in `config.com`. Variables in the templates are then substituted at runtime to generate the COM paths via the use of `envsubst`. To facilitate this, there is a new function, `generate_com` (see below), provided to automatically generate the COM paths. Where possible, COM paths are defined at the j-job level and made read-only. However, many of the EnKF scripts loop over the ensemble members, forcing the definitions to be made at the exscript level instead (and be mutable). The arguments to `generate_com()` are the list of COM variables to generate, optionally accompanied by a template to use using a colon to separate them. When no template is specified, the variable will be generated using the ${varname}_TMPL template. Two options are accepted, `-r` and `-x`, which will mark the variable as read-only and for export, respectively (the same as with the `declare` builtin). It is best practice to define any additional variables needed by the template on the same line to avoid adding them to the calling script’s scope. Here are some examples used in the code: Generate the path to the atmos analysis directory for the current cycle and `$RUN` (implicitly from the `$COM_ATMOS_ANALYSIS_TMPL` template) and mark as read-only and export: ``` YMD=${PDY} HH=${cyc} generate_com -rx COM_ATMOS_ANALYSIS ``` Generate the path to the atmos history directory for the previous cycle's gdas from the `$COM_ATMOS_HISTORY_TMPL` template and mark as read-only and export: ``` RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \ COM_ATMOS_HISTORY_PREV:COM_ATMOS_HISTORY_TMPL ``` Generate the path to the first ensemble member's history directory of the current cycle and `$RUN` and mark for export: ``` MEMDIR=’mem001’ YMD=${PDY} HH=${cyc} generate_com -x COM_ATMOS_HISTORY ``` ## Additional information The staging of initial conditions in `setup_expy.py` has been updated to stage in the new locations. The source of the initial conditions can **either** be in the new hierarchical structure or in the old flat structure and the script will stage the files in the new structure. The destination paths are hard-coded here, so if any changes are made to the analysis, input, or restart templates, they will need to be mirrored in `setup_expy.py`. ### Stipulations All changes in this PR are subject to approval by several stakeholders, including NCO. Sample COM trees above are subject to revision based on feedback (for instance, file X isn't really an obs file). File name updates are not included in this PR. File names (primarily for coupled components) will be updated to comply with NCO standards in a future PR. AWIPS jobs are now almost working (they do not in current develop), but one last program is still ending with an error. Work on fit2obs is deferred, so that portion of the verify job does not work. WAFS scripts are all external and have not yet been updated. WAFS is expected to be packaged separately going forward, so will need to be updated like any other downstream package. Some scripts that are not part of our normal development workflow have not yet been updated. I may be able to knock a few more off this list, but some just aren’t available in development mode currently: - All UFSDA app jobs (to be handled separately) - With associated dev jobs (may still modify and test) - JGDAS_ATMOS_GLDAS - ~~JGLOBAL_WAVE_GEMPAK~~ - ~~JGLOBAL_WAVE_POST_BNDPNT~~ - ~~JGLOBAL_WAVE_POST_BNDPNTBLL~~ - ~~JGLOBAL_WAVE_PRDGEN_BULLS~~ - ~~JGLOBAL_WAVE_PRDGEN_GRIDDED~~ - ~~JGLOBAL_WAVE_PREP~~ - With no associated dev job - JGDAS_ATMOS_GEMPAK_META_NCDCJGFS_ATMOS_FBWIND - JGFS_ATMOS_FSU_GENESIS - JGFS_ATMOS_GEMPAK_META - JGFS_ATMOS_GEMPAK_NCDC_UPAPGIF - JGLOBAL_ATMOS_EMCSFC_SFC_PREP - JGLOBAL_ATMOS_POST_MANAGER - JGLOBAL_ATMOS_TROPCY_QC_RELOC + All downstream scripts for the above There are also a few scripts that are not available to the development workflow that I have already made a good-faith effort at updating: - JGDAS_ATMOS_GEMPAK - JGFS_ATMOS_PGRB2_SPEC_NPOESS ## Related Issues Closes #761 Fixes #978 Fixes #999 Fixes #1207 Partially addresses #198 Partially addresses #289 Partially addresses #293 Partially addresses #1299 Partially addresses #1326

Turns off trace for the duration of the generate_com function unless DEBUG_WORKFLOW is not set to "NO" (the default). In its place, the function will now echo the assignment. Closes #1524

The incorrect path was used for staging coupled components during the COM refactor update (#1421). These are now corrected. Fixes #1528

A typo led to the same variable being defined twice instead of the variable that was supposed to be defined. Fixes #1530

The trailing '_TMPL' was accidentally ommited from the ocean history template used during workflow generation. Refs: #1532

g-w PR #1421 changed the GFS COM directory structure. This PR updates UFSDA ATM variational analysis jobs and python scripts to work with the updated GFS COM directory structure. Fixes #1517

Removes most of content of JGDAS_GLOBAL_OCEAN_ANALYSIS_POST to scripts/exgdas_global_marine_analysis_post.py in GDASApp, now just calls that script. Addresses first bullet of #1480

Found some odd bugs in the aerosol DA changes that need to be fixed to work properly. Not sure how they worked before with testing (by luck?) but these are straightforward fixes to implement. one has a missing / in a path for FileHandler and the other has a missing .nc causing files to not be found

The path for ocean analysis files was not properly updated after analysis was moved out of model_data into its own directory.

g-w PR #1421 changed the GFS COM directory structure. This PR updates UFSDA ATM ensemble analysis jobs and python script to work with the updated GFS COM directory structure. Fixes #1518

This PR updates the j-jobs and python classes for aerosol DA to use the new COM directory structure. This PR also includes removing of the chem history staging for the ICSDIR in setup_expt.py. The aerosol fields are treated as FV3 tracers, so they are either available (warm start) with the fv_tracer files or start at 0 (cold start) and must be spun up from emissions. Fixes #1516

Updates the UFS model hash to the version as of 2023 Apr 17. Some associated changes accompany this update - Restart filenames for MOM6 (ufs-community/ufs-weather-model#1599) - Remove store coriolis setting from MOM6 namelist (ufs-community/ufs-weather-model#1599) - Change in atm 'log' file names (ufs-community/ufs-weather-model#1704) - Additions to diag_table for frozen species (ufs-community/ufs-weather-model#1529) - Restart quilting (ufs-community/ufs-weather-model#1633) - Update to post itag (ufs-community/ufs-weather-model#1690) The switch to restart quilting adds an additional constraint on the size of write groups to be divisible by the number of tiles, so all were increased to the next multiple of 6. In the process of updating the diag tables, unused tables were removed. Closes #1279 as moot Closes #1445 Closes #1499 Partially addresses #1277

`$WAVE_RUN` is no longer used in workflow, but one reference remained in the archive job, which would cause failures. The conditional did not need to be replaced by `$RUN` since `$RUN` has already been checked at that point. Fixes #1548

* Remove bump_ver from versions/fix.ver - No longer need a bump version variable in fix.ver. Refs #1552

When the workflow was updated to use ESMF threading, the CICE namelist was not updated to use the new ntasks_cic6 variable instead of ICEPETS. This caused the incorrect number to be used when using multiple threads. Fixes #1549

* Update init.rst cycled coupled IC info * The cycled ATM w/ coupled (S2S) model ICs were updated and the folder was renamed from "C48mx500" to "C48C48mx500" * Added note to mention that the EnKF member ICs are currently just dummy duplicates of the deterministic.

Replaces `${CDATE}` with `${PDY}${cyc}` in all `config.*` files. Also fixes permissions on some of the config files that were turned to be executable in some of the JEDI PRs. Fixes #1560 Partially fixes #198

…1564) This PR: - updates the j-jobs to accommodate post COM refactor upgrades. - minor updates to config.resources for land DA tasks - add task dependencies in the rocoto workflow - when doing LandDA, use sfc_data.tile{{ t }}.nc created by Land DA is used as input of global_cycle for the deterministic and case w/ DOIAU=NO.

Begin using feature-GFSv17_com_reorg_log_update install which updated the atm logf filename checked for by prepobs getges_nc.sh. Refs #1570

The section on commit message standards is updated to be for pull requests instead. Additional messaging about using the provided template is added, and bullets not relevant to PRs (line-wrapping and leaving a space between subject and body) are removed.

@GeorgeGayno-NOAA

This PR updates the UFS_UTILS hash to get `gdas_init` updates for the recent COM reorg. A few additional UFS_UTILS commits are also included but aren't significant changes, as noted by @GeorgeGayno-NOAA: "Updates since that hash have been mostly small things like bug fixes. However, the w3nco library was replaced by the w3emc library at ufs-community/UFS_UTILS@7efbe0f". Updates to improve repo build are also included with this hash (ufs-community/UFS_UTILS@b4900c4). New hash from completion of ufs-community/UFS_UTILS#820 Resolves #1527

This PR removes: - `getic.sh` - job used to get initial conditions from HPSS from a past experiment/ops - `init.sh` - job used to create experiment resolution specific initial conditions from the ones fetched by `getic.sh`. Both these jobs were primarily used by the atmosphere model component developers in the past. They have since been staging pre-generated ICs making these jobs obsolete. Documentation will point to ufs-utils to generate the initial conditions. The utilities from ufs-utils were used to identify the appropriate HPSS tarball as well as performing interpolation for the atmospheric model component. There is no such utility available for the other model components making these scripts only partially useful. In a future PR, an update to the definition of `BASE_CPLIC` and `coupled_ic` jobs will be made to make use of `--icsdir` to point to pre-staged locations of initial conditions. Fixes #1569

Fixes a couple bugs and does a little cleanup of the COM refactor for marine DA. COM variable definitions are also updated to match the style used in other scripts.

…1535) Updates to scripts/exglobal_atmos_analysis.sh are being put in again after a CRTM issue was resolved. These updates are needed to resolve an issue with the GSI assimilating too low of a number of observations. Also update gsi_ver to 20230112 in fix.ver Refs #1494 #1550

Allows GEFS experiment directories to be set up using `setup_expt.py`. This only creates the experiment directory; workflow generation will be added in a follow-up PR. To facilitate this, a new positional argument is added in the first position to specify which system to set up: `gfs` or `gefs`. If gfs is chosen, the second positional argument is to choose the mode, and allows the same values as before: `cycled` and `forecast-only`. If gefs is chosen, forecast-only is assumed and no second positional argument is required. To allow both gfs and gefs configure files to exist in workflow concurrently, the `parm/config` directory contains two new directories: gfs and gefs. All of the old config files have been moved into the gfs directory. A few select config files have also been copied to the gefs directory. As yet, they are un- modified (except the changes noted below). These config files will be updated in a future PR so the modifications for GEFS are not hidden in the new file creation. The `yaml/defaults` is also copied over, but the contents were removed since GEFS will not be running its own analyses. A couple variables have been renamed to be more appropriate for use in both systems: `NMEM_ENKF`, `NMEM_EFCS`, and `CASE_ENKF` are now `NMEM_ENS`, `NMEM_ENS_GFS`, and `CASE_ENS` respectively. Further, `HOMEfv3gfs` were removed as unused. Finally, resolution validation was extended to the ensemble resolution as well. Fixes: #1576

Removes all instances of GLDAS from the global-workflow `develop` as it will not be used in GFS v17. Also updates the documentation to reflect this change. Fixes #1512

This changes the way the RadMon diagnostic file checker determines if a diagnostic file is 0-sized. Instead of attempting to parse the output of `tar -tv`, the script now extracts the `radstat` file and runs `du -b` on each gzipped diagnostic file. On S4, the primary group is `domain users`, which, containing a space, caused issues with the previous method. Fixes #1515

Adds S2SA to the list of allowed apps for all experiment types. The workflow config files already supported this option, but it was not on the allowed list. Refs: #1589

…ions. (#1567) Adds schema.py from https://github.com/keleshev/schema Adds a test harness and additional functions to setup the Schema object. Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

Adds a read the docs config to get around docker image/library mismatch. Fixes #1595

Read-the-docs config file was in the wrong location. Fixes #1595

Reverts the python version for RTD back to 3.7 to see if that solves the missing bibtex import issue. Fixes #1595

Restored the python version to 3.11 after degrading it to 3.7 did not solve the bibtex issue. Instead, specify the requirements file in the config (the file already existed). Refs: #1595

Refs: #1595

Still trying to resolve issues with RTD build. Refs: #1595

Refs: #1595

…urable (#1603)

New gfs-utils hash includes gfs_bufr code fix to resolve bugzilla #1208 Other commits included in new hash are: - Partial removal of deprecated utilities and code - Updated Jet's module to point to the EPIC hpc-stack Refs #1245

…ore explicit (#1607) * make some JEDI cycling options configurable through YAML

Updates the code to point to correct fix files with name changes. Fixes #1493

@jiaruidong2017

This PR: - adds a job to prepare IMS snow depth observations as a task in the workflow. This task depends on the `prep.sh` job to bring IMS data. To test this type of data `DMPDIR` in `config.base` needed to be pointed to `"/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump"` - `land_analysis.py` introduces a method `prepare_IMS` for this type of data. - Updates are necessary in the `GDASApp` repo. See companion PR NOAA-EMC/GDASApp#472 This job only runs at the `18z` cycle in the workflow. This is controlled in the XML and not in the script. Consequently, updates are in the `workflow/` scripts to make this happen. Other updates included: - Adds a method (and associated test) to provide a datetime in the form of Julian day of the year. The method is `to_julian` in `timetools.py` - `executable.py` was updated to accept a `list` of arguments (in addition to `str`). **Note:** - `calcfIMS.exe` is a serial executable and yet requires to be run with `srun`. I recommend compiling this executable without MPI. @jiaruidong2017 @CoryMartin-NOAA - There are several deprecation warnings from the python IODA converter that will need to be addressed. I recommend creating an issue to be resolved later. @jiaruidong2017 @CoryMartin-NOAA Part of Land DA work for snow depth assimilation, for which there is no issue open.

The variable type expected by gaussian_sfcanl for donst was recently changed from YES/NO to a boolean (.true./.false.) Fixes #1619

Adds the ability to throttle the number of PRs and cases being run concurrently.

This PR: - allows a user to define their own configuration via a yaml during `setup_expt.py`, while retaining the `defaults.yaml` which is a must when templates are being set - updates to `setup_expt.py` to allow the above. - adds a `test_ci.yaml` as a demonstration for use in CI This PR will facilitate deprecating `config.defaults.s2s` as well as allowing users to make their own configurations.

resens was mischaracterized as ensres. CASE_ANL is not used in atmens_analysis.py set CASE_ANL=CASE in config.atmanl

Fixes for enkf archival bugs Refs #1622 and #1623

@NeilBarton-NOAA

This is an expansion of @NeilBarton-NOAA 's work in PR #1488 * Adds new glo_025 grid to workflow which will be used by GEFSv13 (fixes #1434) * Ensures post works * Creates a section in config.wave for various wave grids so that it's easier to switch between grids * Adds documentation for adding a wave grid so that it will hopefully be easier the next time * Initial condition was added to hera for 2013010100 for this new grid so that C384/ P8 like tests can be made. (Needs to be synced to other platforms if desired). This work expands on the work in PR #1605 Fixes #1434 Fixes #1604

…figs (#1630) Some of the keys in the `defaults` were not being updated in `update_configs()` because it was using a regular `**dict` that would only update keys at the top-level. These dictionaries are nested and deep. This fixes the bug by replacing `**dict` with the `update` method from `AttrDict`.

This PR fixes logic in the main driver so it can function in cron along with the check and run scripts. It also runs with the emcbot account so messaging to the user on the status to GitHub will come from emcbot

This PR adds UFSDA ATM var and ens files to `hpssarch_gen.sh`. Logic is also added to check for the existence of GSI-EnKF DA specific output files before adding said files to the archive list. With this change `hpssgen_arch.sh` works for both GSI and UFS based DA. Fixes #1628

This PR removes WAFS references, etc. from the global-workflow following the approval from NCEP director on making WAFS a standalone post-processing package. Fixes #1641

* Update fix_file.md - remove KateFriedman-NOAA * Update production_update.md - Change auto assignee from KateFriedman-NOAA to WalterKolczynski-NOAA * Update NCO_bug_report.md - Remove KateFriedman-NOAA

Updates `DMPDIR` (global dump archive path) on WCOSS2 to the new dedicated path: `/lfs/h2/emc/dump/noscrub/dump` Closes #1655

Introduces the use of version files in the `develop` branch. This initial introduction will use version files for runtime only. The ability to force version files for build is included but commented out for now. A follow-up PR will turn on build version forcing when ready. Changes include: 1. Update `module_base.*.lua` modulefiles to use module version variables and remove hardcoded versions. 2. Introduce `build.<platform>.ver`, `run.<platform>.ver` files in the `/versions` folder. 3. Update `sorc/link_workflow.sh` to create `build.ver` and `run.ver` under `/versions` folder based on `build.<platform>.ver` and `run.<platform>.ver`. 4. Introduce sourcing of `run.ver` in `ush/load_fv3gfs_modules.sh` 5. Introduce sourcing of `build.ver` in `sorc/build_all.sh` but commented out for now. Components will continue setting their module versions for the time being. 6. Update `HOMEens_tracker` in `config.vrfy` to use `tracker_ver` from version files. 7. Replace obsproc version in `HOMEobsproc` setting in `parm/config/gfs/config.base.emc.dyn` to use `obsproc_run_ver` from version files. Resolves #671 Resolves #1625

@sanatcumar

This PR ensures that we can run C96 5deg and C48 5 deg for 2021032312. Two accompanying fix file updates for this PR: #1645 and #1647. Thank you to: @sanatcumar @barlage @HelinWei-NOAA Bing Fu @jiandewang @guillaumevernieres @aerorahul @NeilBarton-NOAA and others who helped provide input, fix files, directions, and other help!

IAUFHRS and IAUFHRS_ENKF are defined in config.base, but they were not reset to the default value if IAU is off. This commit sets IAUFHRS and IAUFHRS_ENKF to "6" in config.base if DOIAU or DOIAU_ENKF are set to NO. Refs: #1557

This adds an option for 2 or 5 deg global wave models, which will provide a light weight version to assist in engineering development. -- This PR requires PR #1648 to be merged as this builds on it as well as issue #1652 which adds the fix files needed for this PR to be addressed first. The resources were chosen to stay under 1 or 2 node and be faster than the atm component. - Fixes #1333

Adds updates to the CI cron BASH scripts that track an ID based on how many times the label gets updated to Ready. This ID count is used in the driver script so that it can distinguish a real fail and one caused by removing the current build that is overwritten by a new Ready request. Fixes #1649

To facilitate this, a few classes were refactored into abstract classes with child classes for each of the three `$NET`/`$MODE` combinations (gfs/cycled, gfs/forecast-only, and gefs). These child classes then implement the details particular to their configuration. GEFS is implemented as only forecast jobs right now: a fcst job intended for the control and a metatask of efcs jobs intended for the perturbed members. A couple of additional config files were needed in order to parse the configs; these were copied from the gfs versions. Beyond that, changes have not yet been made to update any configs for GEFS. For instance, in contrast to GFS, even the control member of GEFS will want `MEMDIR` defined (to 'mem000'). Fixes #1577

In PR #1601, the removal of the old `applications.py` was inadvertently excluded.

Update logic for interpolated wave gridded post Fixes #1658

Switches the runtime pyenv from ufswm to gfs_workflow on Hera and Orion. gfs_workflow is the new one created just for workflow and is the target going forward. CI modules have already been using it.

…1665)

…sistent with updated GSI-fix (#1650) (#1666)

Add JEDI-based LETKFOI update for snow depth analysis in the global-workflow. These jobs are run in a single landanl task. Additional filter to_f90bool is added to jinja template processing.

The global-workflow runs offline UPP to generate master (and optionally flux) grib2 files for: - analysis - GOES simulated satellite imagery - WAFS products WAFS has been removed from the global-workflow in a previous PR. The offline UPP and product generation in the current workflow is done in a single job. This PR aims to partition the offline UPP from the product generation (for atmos grib2 files). The changes in this PR are not yet activated in the workflow. The offline post is now completely yaml driven for all the variants of the upp run conditions. In a future PR, product generation jobs will be added and subsequently current post-processing jobs will be retired.

This feature changes the CI system to use the test cases from the PR in question. This allows the PR to add or modify tests and have them applied to itself to ensure they work. Fixes #1662

Updates the state of GitHub badges based on the results of CI testing of the last PR to easily see the state and stability of the develop branch. GitHub's Workflow Actions are used to generate the badges in the README.md file for the development branch. The GitHub Actions will update the badges only when a PR is merged to the **develop** branch and only update them according to the state of the Labels in the PR itself. This Action must be able to use the final state of the CI labels of the PR. The labels must therefore be dynamically "pulled" from the PR just before it is merged. Here is the map between the Label states and the Badges (for Orion and Hera respectively): ``` Pass -> Pass Fail -> Fail None -> Pending ``` For example, if the Badges are currently _Pass_ and a PR is merged without CI test Pass Label the corresponding Badge will be updated to _Pending_. Fixes #1643

Updates the UFS hash to bf60924. This required a change in the compile signature due to changes in UFS to how the compiler is defined.

A recent fix of another issue meant that wave config settings were set incorrectly for glo_025 resolution. This removes pass through options which caused the issue since a few variables were intentionally set to nothing but then were set incorrectly to another default.

When setup_expt.py or setup_xml.py fails in CI, the error logs are now reported back to the user via the message stream in the PR. Fixes #1672

…)" (#1707) This reverts commit 38cd0bc.

This PR provides support for running UFS weather model forecasts, using the global-workflow, on the AWS Parallel Works (PW) platform. This PR accomplishes the following: - Building the UFS weather model within the global-workflow build system; - Running a UFS weather model atmosphere-only deterministic forecast using the global-workflow infrastructure. Follow-up PRs will include the following: - Adding documentation for building and using the AWS PW UFSWM spack-stack environment; this is heavily borrowed from Sadegh Sadeghi Tabas's presentation 03 May 2023; - Addressing remaining `TODO` issues within this PR; several notes are made throughout as to how to better streamline the connections between the UFSWM and the global-workflow for the respective cloud platforms; - Adding data-transfer capabilities within the global-workflow for cloud deployments; - Testing additional UFSWM capabilities (e.g., coupling, etc.,) and resolutions (this PR has been tested at C48 and has not been configured for other cubed-sphere resolutions). No dependencies other than the loading of the appropriate AWS PW modules are required. These are included in this PR.

By default, the early-cycle EnKF is designed to write analysis files. The switch controlling this was using `$CDUMP` which was changed by PR #1421. This current PR updates this switch to use `$RUN` in accordance with issue #1299, and restores the default analysis file write capability.

Fix Ocean post treating zero-padded forecast hours as octals. Fixes #1710 Refs #1195

) user will provide a single integer restart_interval (hours) via config.base and config.fcst, config.efcs. scripts will convert this as necessary for model_configure and nems.configure. Logic for determining RERUN revisited based on files available rather than restart list. Fixed a bunch of shell norms --------- Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>

* Create gw_setup script and extend to other machines Add a new script to handle setting up the lmod environment needed to run all the setup for checkout and experiment generation. C `checkout.sh` now uses this script as more recent versions of `git` are needed on some machines. Documentation has been updated to recommend this method on all machines rather than the previous instructions. Updates documentation to add the checkout with UFSDA option. Closes #1677

`BASE_CPLIC` depends on the machine. It is now defined in `hosts/machine.yaml` together with other machine specific paths.

During the COM refactor, a location for the ocn/ice daily files was defined within the products directory. However, the daily files are output directly by the model and land in the history directory. The archive job was looking in the otherwise unused daily product directory for the files and not finding them. The 2D and 3D ocean files had the opposite problem: archive was looking in the history directory, but these files have their own directories in products. This update now: - Removes the COM variable for the daily files - Updates the archive job to: - pull the daily ocean files from history - pull 2D ocean files from the 2D product directory - pull 3D ocean files from the 3D product directory Fixes #1709

* Add S2SA nems.configure template Most of the configuration was setup to allow S2SA, but there was no appropriate nems.configure template. A new one was created by starting with the S2SWA template and removing the wave portions. Also added two UFSDA scripts that were missing from the git ignore list. Fixes #1592 * Update documentation for cycled mode setup The experiment setup documentation for cycled mode had not been updated as coupled capability was added. Also added notes to both the cycled and forecast-only section that ATMW is currently not working. * Update setup_expt to recognize more S2S cycled exps setup_expt would only detect two of the possible S2S options when staging cycled ICs. Refs: #1592 * Change path to volcanic aerosol data The volcanic aerosol data was moved to a new location within the GOCART emissions directory. Refs: #1592 * Disable enkf versions of coupled settings The enkf versions of coupled settings (DO_OCN, etc.) do not work properly because some settings have already been determined by that point based on the non-enkf versions of the settings. The enkf versions are now disabled until there is a solution (see #1692). Refs: #1593 Refs: #1692 * Fix ocnpost detection for gfs free-forecast The task list for the free-forecast was checking against a list of apps that did not include S2SA to determine whether ocnpost is included. That check is now changed to check on do_ocean rather than looking at the app directly. Refs: #1593

…1718) This PR: - allows self cycling of the ocean and ice components in a cycled atmosphere with 3DEnVar (without IAU). - allows staging of initial conditions for coupled ensemble when setting up cycled ensemble. This PR also includes: - numerous improvements involving appropriate locations for defining relevant variables for components. This PR does not: - address archiving of ensemble data when cycling - probably many other issues that are necessary e.g. IAU Partially addresses #613

Using numpy to cast type has been deprecated in favor of using the native python functions (i.e. `int()`, `float()`, etc.). Resolves #1725

Partition name and node size for AWS were missing from `config.resources`. This information has now been added. Resolves #1727

`pygw` from the global-workflow has been moved to its own repository at [wxflow](https://github.com/NOAA-EMC/wxflow). This PR: - removes `pygw` and replaces it with a checkout of `wxflow` in `checkout.sh` and `Externals.cfg` Going forward, until `wxflow` becomes a fixture as a module, `global-workflow` will be required to be checked out and is done as part of `checkout.sh` and `Externals.cfg`

* Add GEFS fcst job support - Makes the minor changes necessary to run GEFS fcst job only from pre-staged ICs. Also many unnecessary DA settings are removed from the GEFS version of `config.base`. - Stop using DA diag table from GEFS, remove unneeded settings Also removed more unneeded DA settings from the GEFS side. - Use RUN instead of CDUMP to determine output frequencies in forecast Refs: #826

Fixes #1486 * Turn fit2obs back on by default Fit2obs had been turned off during the COM refactor because it had not yet been updated for the new paths. Now that the job has been updated, it can be turned back on.

@junwang-noaa

This PR: - moves `nems.configure.*.IN` templates from `ush` to `parm/ufs`. - uses `atparse.bash` from the ufs-weather-model to fill those templates. - make `gefs/config.resources` in WW3 same as in `gfs/config.resources`. - makes use of `HOMEgfs` instead of `SCRIPTDIR` and `script_dir` local variables that are unnecessary aliases and make code understanding easier. - initializes PETS to 0 and then sets the right value if that component is enabled. A follow-up PR will reconcile the templates and use them directly from the ufs-weather-model. Several differences were noticed between the ones in `global-workflow` and `ufs-weather-model` and help may be needed from @junwang-noaa to update the ufs-weather-model (if required). Fixes: #1599

…ra fix (#1739) * change YAMLs that are copied * Use new gdasapp hash

* added two files for this PR (now that devlop has pslot tag updates) Co-authored-by: TerrenceMcGuinness-NOAA <terry.mcguinness@noaa.gov> Co-authored-by: Rahul Mahajan <aerorahul@users.noreply.github.com>

This reverts commit e95228c. The problem in cycled may be fixed, but it breaks forecast-only.

Fix bug with HOMEgfs getting populated incorrectly

@RussTreadon-NOAA

On further examination, `SDATE_GFS` in the context of `metp` task is the first date that METP was run and started generating statistics. This is different when doing cycled (`gfs_cyc` determines `SDATE_GFS`). In the case of forecast-only, `SDATE_GFS` is always the date the first free-forecast was launched (`SDATE`) While the fix by @RussTreadon-NOAA worked for cycling, it broke for forecast-only setups and needed to account for the other mode of operation. Fixes #1748

Some of the scripts being linked in from ufs_utils are not needed for global workflow.

* Update the GDASApp stable-nightly commit hash. * Remove FRAC_GRID dependency from land analysis. Fractional grid will be always be assumed here. * Use SNOWDEPTHVAR="snodl"

Adds a single atmosphere forecast-only test at C48.

Makes the necessary updates to allow the workflow to run (non-wave) coupled forecasts. Some of the coupled functions in postdet had to be updated because they only checked for gfs and gdas runs, so necessary code was never executed. Also updates so that the GEFS control uses `mem000` as the memdir so it appears at the same level as the perturbed members. Relatedly, the default member number in coupled files is changed to 000 instead of 01. Finally, an unrelated error in the earc job is corrected where a character from the middle of the line was moved to the beginning. Resolves #1754

Adds ush/soca to pythonpath in ocean vrfy task jjob in anticipation of python code to be added to said subdirectory.

This PR: - adds a job to prepare GTS snow depth observations as a task in the workflow. This task depends on the `prep.sh` job to bring GTS adpsfc bufr data. To test this type of data DMPDIR in `config.base` needed to be pointed to "/scratch1/NCEPDEV/global/Jiarui.Dong/JEDI/GlobalWorkflow/para_gfs/glopara_dump". - land_analysis.py introduces a method `prepare_GTS` for this type of data. - Updates are necessary in the GDASApp repo. See companion PR NOAA-EMC/GDASApp#541 The `preplandobs` job runs at the all four cycles in the workflow. The `prepare_IMS` job runs only at 18z cycle, and this is controlled in the script.

Trying both ATM C384/C192 cycling with GSI hybrid-4DEnVar (half res of ops) and ATMA C384 3DVar on Hera, I ran into several problems. This PR solves them (for now!) Change summary: 1. The aeroanl layouts are now all 8x8, this is just so that we can use the C96 BUMP files that are 8x8 layout for higher resolutions (this is not expected to be a permanent change, and we will have 'optimized' values later) 2. analcalc would fail (Closes #1738 ), this specifies the amount of memory and allows it to complete 3. The first half cycle C384 forecast would fail, changing `nthreads_fv3` to 2 fixes this 4. The epos jobs do not need 14 nodes allocated, this changes it to 2 nodes, and it runs at this resolution in only 3 mins on Hera 5. A bugfix (missing `f` before a f-string) and changed `_res_enkf` to `_res_anl` because `CASE_ANL` is set in `config.aeroanl` but `CASE_ENKF` will not be set by the setup_expt script if you specify `nmem` to be 0

…able (#1697) - A new soil color dataset is introduced to Noah MP to reflect the right soil albedo to reduce the large warm bias found in the Sahel desert. The more-recent and higher resolution BNU soil texture data will be used to replace the old data. - Noah MP parameter table is part of the source code in the current version. The UFS model has to be recompiled whenever there is any parameter change in the table. Therefore we change this structure and the table will become one of the input file to the model. Closes #1614 Closes #1617

This PR: - moves `parm/parm_fv3diag -> parm/ufs/fv3` - moves `parm/mom6 -> parm/ufs/mom6` - moves `parm/chem -> parm/ufs/chem` - updates scripts referencing this updated path Fixes #1510

GEFS was failing because the rCDUMP in the forecast job was hard- coded to enkfgdas. This resulted in always looking in that directory for previous cycle restart files. This was not caught previously because the manual staging method was copying the gdas directories as well, masking the problem. Resolves #1771

This PR: - temporarily adds a hack to allow testing of the GEFS forecast until the staging is automated. - adds a test for GEFS - updates `setup_expt.py` to compartmentalize gfs and gefs setup options. GEFS has a single mode `forecast-only` and is necessary for `create_experiment.py` to work as designed. To eliminate it would need major work.

This PR moves the GDASApp to CRTM 2.4. It also removes the need for the global-workflow team to maintain CRTM coefficients in their fix space, instead we should be linking/copying them from wherever the CRTM coefficients are stored (like the way GSI does it). For now, that is an 'unofficial' location, but will eventually be the 'official' one.

Turns off unbound variable checking when calling `module reset` on Hera and Orion to supress message about PS1 being unbound since correcting it is outside the control of workflow. Resolves #1780

The GEFS configuration is updated to override the default (GFS) post flatfiles when running GEFS. Resolves #1750

Create a message in PR with link to the PR's ReadTheDocs before it's merged. Fixes #1610

Soundings are allowed in forecast-only mode, but the config file would not be sourced during experiment creation. Now the config file is sourced if bufr soundings (`DO_BUFRSND`) are on. Resolves #1431

This PR: - describes the `requirements.txt` file for RTD documentation deprecation notice described in issue #1798 fixes #1798

This PR: - eliminates superfluous `RUN_ENVIR` definitions from jobs and scripts as `config.base` contains that definition and all jobs must source `config.base`. - retains `RUN_ENVIR` conditionals in `configs` where it is necessary to distinguish between datasets as well as in genesis and tracker jobs where external directories to GFS are created to place products. Fixes #1220

Adds scripting to stage ensemble files for use in hybrid variational and ensemble UFS-based atmospheric DA.

We're getting ready to run HR2 and in the process have found a few minor bugs. While these shouldn't effect others running low resolution cases, I wanted to push these bug fixes for anyone trying to run high resolution. These bugs address: * Issue #1793 Adding extra tasks to write component for hera for C768 (otherwise crashes due to memory) * Avoiding requesting two wave restarts at the same time (this is a known bug that the workflow usually has work around for. A fix for the underlying WW3 bug should be coming within the next month, but this will get us through that waiting period) * Adding a setting that was missed when updating the ufs-waether-model that ensure that CMEPS restarts are written in a reasonable time (See: ufs-community/ufs-weather-model#1873 for more details)

This PR creates `COM_CONF_TMPL`, copies the YAML files generated for land DA from initialize to COM and copies the increment files to COM.

Symlinks for the final restart time used the full path name, even though the targets are in the same directory. This means the links would break if the directory were moved or (more importantly) put in a tarball. The links have now been replaced with relative links. Resolves #1446

A path filter was applied to pushes, but none was applied to PRs, causing the GH action to build documentation to fire every time a PR is opened or updated. Resolves #1791

* Convert issue templates into forms (PR's do not allow forms) Resolves #1747

This PR has the final updates for HR2. There was one added variable for diagnostic output: `iopt_diag=3` and required from land team. This also points to HR2 updates for initial conditions which update the land states in `sfc*` files compared to HR1. Lastly, it was determined that the setting to enable running on hera `WRTTASK_PER_GROUP_PER_THREAD_PER_TILE_GFS=20` actually will fail on WCOSS2. While not ideal, a setting that would run out of the box on both hera and wcoss2 for C768 could not be found, therefore a comment was added noting WCOSS2 needs to use a different setting until it can be resolved. Resolves #1500

This PR partially addresses #1821 in that it allows for a C48 simplified aerosol 3DVar test using the workflow to run in GDASApp CI testing. The workflow team will be able to adapt this test case for their CI testing with minimal effort. This limits the C48 aerosol analysis to 6 PEs and updates the hash of GDASApp to a stable build from this morning. Refs #1821

* Update logf filenames for postsnd job scripts * Update gfs-utils hash in checkout.sh Refs #1832

…1826) Add a job to convert bufr format GDA dumps to ioda format for use in atmospheric UFS-DA. Resolves #1820

Co-authored-by: saeideh banihashemi <saeideh.banihashemi@dlogin03.dogwood.wcoss2.ncep.noaa.gov>

NOAA-EMC/GSI#602 was merged in without the subsequent script changes needed in the workflow. This PR fixes that.

* adding cases for weekly tests and tests for each pr * updated path for cases to cases/pr in the main cron driver script to reflect new tree structor in cases dir in support of weekly vs pr based ci tests --------- Co-authored-by: Terry McGuinness <terry.mcguinness@noaa.gov>

…1831)

GFS now v16.3.9 in operations. Refs #1813

…1836) * Update the GDASApp's commit hash.

…1858) GFS v16.3.9 was implemented 20230911 12Z. The following changes were implemented into operations: - turns off ingest of NOAA-21 radiances into the GSI, - PlanetIQ GPSRO and NOAA-20 OMPS (Ozone Mapping and Profiler Suite) Ozone retrievals will be actively assimilated, - OMPS observation errors will be adjusted. g-w PR #1852 brought these changes into g-w branch [`dev/gfs.v16`](https://github.com/NOAA-EMC/global-workflow/tree/dev/gfs.v16). This PR brings the same changes into g-w `develop`. Fixes #1856 Fixes #1857 Related to, but not dependent on, GSI [#621](NOAA-EMC/GSI#621)

This PR came about attempting to understand how the atmos downstream products are generated from the master grib2 files before they are updated to python-based scripts and improved control. This PR: - removes explicit if blocks over the two downsets that largely repeats the same operations and condenses into a loop - introduces a generic `run_mpmd.sh` that is tested on Hera and WCOSS2 and can be used elsewhere in the global-workflow. Its input is a script that needs the instructions to be executed, one-per-line. No change in information content is expected from this update to the products. This PR is necessary if the global-workflow team is to mange the post processing and product generation scripts.

Yet more base bugs were found in post. To eliminate these once and for all, `fhr` is changed to have no leading zeros, and places where the zero-padded string is needed now use `fhr3`. Resolves #1195

…ests (#1869) A BASH script using **GitHub CLI** is added to create a branch from _develop_ named _weekly_tests_ and moves the cases from `$HOMEgfs/ci/cases/weekly` to `$HOMEgfs/ci/cases/pr` and then opens a PR and labels it with _CI-Hera-Ready_ and _CI-Orion-Ready_ thus creating a weekly high resolution test run that can easily be launched by a **cron** job. Resolves #1663

Co-authored-by: Terry McGuinness <terry.mcguinness@noaa.gov>

Removes `${STMP}/RUNDIR/${pslot}_sha` before creating an experiment to avoid name cashing on restarts Resolves #1881

This fixes a bug in IC linking within setup_expt.py by correcting the date string based on experiment start type -- warm starts should look at `rdate` (-6 hours) and cold starts should look at `idate` (current cycle). Fixes #1783

This PR updates the global workflow so that it can handle the latest version of the JEDI aero staticb, and also archives the aerosol diagnostic files.

Update the data dependency path for the wavepostbndpntbll job to use the ATMOS/atmos subfolder instead of WAVE/wave for the atm.logf file it needs to fire off. Refs #1887

This PR fixes a few bugs that were reported by developers: - a namelist section is unclosed in `parsing_namelists_FV3.sh` - resolution is not being properly cut from `CASE` - date should be in UTC to properly go back and forth around DST dates. Resolves #1796

PR #1822 converted some formerly stand-alone scripts and moved them inside one of the standard product scripts. However, `trim_rh.sh` is also used by gempak and AWIPS, so this change caused those jobs to start failing (they are not yet tested by CI). Those functions have now been moved into a separate product functions script that can be sourced by all the scripts that need the functions. A third similar script was also eliminated and converted into a function. Additionally fixes a base bug in AWIPS and updates some scripts to satisfy (or at least mostly satisfy) the linter. **AWIPS still failing when creating GRIB1 products.** That will be addressed in a future PR (and may need a subject-matter expert). Resolves #1890

When the prototype S2S was first added, we placed some settings in a `config.defaults.s2sw` file to avoid interference with existing settings. Now that we are moving towards v17 and run S2S regularly, it is time to merge these settings back into the other config files. The wave output frequency settings are updated to the coarser fre- quency as that is more suitable for development. The prior (ops) settings are noted as comments. There are a couple of settings that control ice fields within FV3. These are placed in `config.ice`, so when the ice model is on those settings are still used. Changes the output frequency from 3 to 6. While we may leave this setting as the default for development, it is required right now because the ocean frequency is hard-coded to 6 but links are created based on `$FHOUT_GFS`. Resolving #1629 should correct this limitation. "Gaussian" archive is updated to the former override values. Resolves #1606 Refs #1629

Simply renaming the platform config files in ${HOMEgfs}/ci/platforms to config.${MACHINE} so wxflow can read them too in support of up and coming functional testing.

@JessicaMeixner-NOAA

This PR: - replaces the PR C48 S2S forecast-only test with a S2SW configuration - adds a weekly C384 S2SWA forecast-only test. The C384 S2SWA is a weekly test and will not be triggered/tested in this PR, but will be tested on Friday. See notes on Orion below in the testing section. @JessicaMeixner-NOAA kindly provided the details in #1794 (comment)

…1882) This PR changes the module loads in g-w jobs `anal.sh`, `eobs.sh`, and `eupd.sh` to use machine specific modulefiles from `sorc/gsi_enkf.fd/modulefiles`. This is necessary because [NOAA-EMC/GSI#571](NOAA-EMC/GSI#571) updated `gsi.x` and `enkf.x` builds to intel/2022 on NOAA RDHPCS machines. Resolves #1863

Update MINMON if-block in jobs/rocoto/vrfy.sh to only invoke this section when MODE is cycled. Refs #1076

This PR determines the ensemble resolution for ocean, ice and waves based on the deterministic configuration as there is no dual-resolution capability requested for these model components. For cycled DA, if the deterministic uses a coupled model, the ensemble will also be a coupled model. This can be turned off manually via adding `DO_AERO_ENKF`, `DO_OCN_ENKF`, `DO_ICE_ENKF` and `DO_WAVE_ENKF` flags set in `config.base` or `config.efcs`. This is currently handled manually and will be automated when a requirement arises for this flexibility. Fixes #1730

MET+ jobs rely on files already having been placed in the local archive directory, but the job only depended on post, resulting in the job prematurely firing. The dependency is now updated to be on the archive job instead. Resolves #1756

) Bugfix and updates to weekly CI test that deletes the new modified branch if it is already on the repo but now also makes sure that the intermediary branch that creates the PR is coming from emcbot.

…1915) This PR updates the check CI bash script in cron to remove the PR that is created by the weekly CI test driver when all the tests has passed on all the platforms. This is done by the script checking if all other labels for all the platforms for the PR that has currently been flagged to have run to a successful completion by the check ci bash script.

Bugfix: left out pipe to word count` | wc -l` after updating: `num_platforms=$(ls ../platforms/config.* | wc -l)` to `num_platforms=$(find ../platforms -type f -name "config.*")` when it was flagged by the BASH linter.

Minor updates the configuration related to running the marine DA and the coupled UFS. What's proposed in this PR: - Memory increased for the marine DA analysis step - The coupled UFS is (was?) running out of memory without the changes in `config.ufs`. - Updates to the IAU section of MOM_input to allow for a current increment.

Co-authored-by: TerrenceMcGuinness-NOAA <terry.mcguinness@noaa.gov>

This PR addresses issue #1501. All instances of `SENDCOM` gates have been removed. The workflow was inconsistent in its use of `SENDCOM` to gate movement of files to `COM` and have been removed based on NCO guidelines. Resolves #1501.

* replace HOMEgfs_PR with HOMEgfs, and HOMEgfs with ROOT_DIR. ROOT_DIR is the location where the ci scripts live * update type to system in case yamls * move create_experiment.py to workflow/ * detect HOMEgfs in create_experiment.py based on location. update to use jinja templates in the cases * remove export of HOMEgfs from driver.sh

Moves the clean-up that was previously done in the archive jobs into their own separate job. The clean-up is also streamlined considerably by using only `COM_TOP` instead of going through every template. There is also additional streamlining/corrections in the function that does the actual removing. Some settings used by both jobs were elevated to `config.base`. Others only needed for cleanup were moved to the new config for that job. Also corrects a small error encountered when attempting to rerun an ensemble forecast. Resolves #583 Resolves #1872

When #1906 was merged, the dependency for enkf cycles was hard-coded to use the enkfgdas archive instead of depending on the `RUN`.

… as part of gefs from gefs/config.resources (#1952)

* Multithread the checkout script #1953

Set SENDCOM to YES in config.vrfy to get outputs copied back to COM. Will reevaluate the need for SENDCOM when moving the tracker/genesis jobs out of the vrfy job with issue #235 work. Refs #1947

Co-authored-by: henrywinterbottom-wxdev <henry.winterbottom.wxdev@gmail.com>

* added -u to global checkout so CI test builds tests for GDASapps * Update check_ci.sh needed more quotes --------- Co-authored-by: TerrenceMcGuinness-NOAA <terry.mcguinness@noaa.gov>

When PR #1823 was merged, the name of the marine product template was not updated in ocnpost, leading the products to be placed in the wrong location and missed by the archive job. Resolves #1902

Update the "State of operations" GFS version number to new v16.3.10 (Annual CO2 fix file update in operations). Refs #1924

This updates the ufs-weather-model to the commit hash from 10-12-23 from the HR2 tag. The diffs can be seen here: ufs-community/ufs-weather-model@GFSv17.HR2...68050e5 Resolves #1811

Certain tarballs may or may not contain `rstprod` data. For instance, the first half cycle gdas and enkfgdas tarballs will not contain `rstprod`, while future cycles likely will. Also, some systems do not have `rstprod` on them. This will test the contents of the tarballs first before attempting to change the group to rstprod. Resolves #1460

Update GDASApp hash to bring recent UFSDA development into g-w. Resolves #1972

Adjusts source paths for forecast-only ICs in support of extension to ensembles. New paths omit the resolution (that is captured by the IC id name) and allow for a member level. Directories with the reorganized ICs have been added to the three tier-1 machines under glopara space. Building on this, also adds capability to stage GEFS ICs. Resolves #911

This PR adds the files under `analysis/ocean` (except those in `bump` and `vrfy`) to the HPSS archiving job, in `gdasocean_analysis.tar`, which were previously not archived. Resolves NOAA-EMC/GDASApp/issues/693

…1974) Small changes to driver CI bash scripts to recapture error log from `create_experment.py` logger Fixes #1939

rCDUMP is needed by the exscript but was never exported by the jjob.

Optimized/parallelized build scripts. A few notes: 1) The default number of build jobs is 20, but one could argue for using 40. When researching this, I looked up what the SRW is compiling with. That system uses 40 cores, which seems a little excessive, but on testing the global workflow, the actual number of cores being used at any given time rarely exceeds 16 when running with 40 cores. This is because the builds tend to use multiple threads in the beginning when compiling low-level modules while the higher-level modules are more or less serial AND because the GDASApp takes several minutes to initialize all of its subrepositories by which time the smaller builds are complete. 2) I also updated checkout.sh so that all checkouts are simultaneous. The CPU load for `git submodule` is quite low, so running 16 instead of 8 jobs at once is not much more expensive. 3) To make this work, I had to add `-j` options to most of the build scripts. The only exception is build_upp, for which the build script within the UPP is hard coded to use 6 cores. 4) I fixed a few small bugs in the build scripts along the way. 5) Lastly, this reduce the total build time from ~2.5 hours for the entire system (including GDAS and GSI in the same build) to ~40 minutes when running with `-j 40`. Resolves #1978

Changes the respective `.txt` files used for the HPSS archive to the `DATA` path. Each defined text file has been updated to explicitly write to `DATA` rather than `COM` such that the files are deleted rather than being archived. Resolves #760

Fixes missing `$` in variable reference in archive job. Resolves #2022

This PR addresses issue #1130. A new task `postanl.sh` has been added such that the `anl` post-processing is not it's own task within the Rocoto workflow. This change prevents the GDAS experiment from breaking due to failures caused by the `anl` not being available for the first cycle. Resolves #1130

This PR will move the GSI monitor tasks from the `vrfy` job into their own separate jobs in the rocoto mesh. This takes care of part of issue #235. Changes in this PR: 1. Create new individual GSI monitor jobs in the rocoto mesh to match how it is done in operations. The new jobs are: `gdasverfozn`, `gdasverfrad`, `gdasvminmon`, and `gfsvminmon`. The names correspond to the names in operations and the ecf script names. The jobs are only added to cycled mode, none of the jobs are run in forecast-only mode. 2. Updates to GSI monitor scripts: * Update `ush/jjob_header.sh` arguments in JJOB scripts to call new individual job names and configs. * Introduce `COM_*` variables and use of the `generate_com` utility. This results in the outputs landing in `COM` subfolders, like occurs in operations, instead of landing in the online archive. * Some variable cleanup in JJOB and ush scripts, includes cleaning up usage of `DATA` variable and its removal at the end of the jobs. * Add missing exit statements to `JGFS_ATMOS_VMINMON` and `JGDAS_ATMOS_VERFRAD`. 3. Create new rocoto job scripts for the new individual GSI monitor jobs. 4. Create new configs for the new individual GSI monitor jobs. 5. Add new individual GSI monitor jobs to resource configuration, using the same resources as in operations. 6. Move and rename GSI monitor job switches from `config.vrfy` to `config.base`. Also add switches to setup system. New switches to control each job being added to the rocoto mesh are: `DO_VERFOZN`, `DO_VERFRAD`, `DO_VMINMON` 7. Add GSI monitor job output to archival job and into resulting HPSS tarballs or local tarballs. This now matches what is done in operations. 8. Remove the GSI monitor tasks from the rocoto `vrfy` job and associated `config.vrfy`. Example output locations and files (not showing contents of oznmon/horiz or oznmon/time for brevity, see archival script for full list of output files): ``` kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gdas.20211221/00/products/atmos/*mon gdas.20211221/00/products/atmos/minmon: total 112 -rw-r--r-- 1 kate.friedman emc 14030 Oct 25 15:00 2021122100.costs.txt -rw-r--r-- 1 kate.friedman emc 84932 Oct 25 15:00 2021122100.cost_terms.txt -rw-r--r-- 1 kate.friedman emc 808 Oct 25 15:00 2021122100.gnorms.ieee_d -rw-r--r-- 1 kate.friedman emc 808 Oct 25 15:00 2021122100.reduction.ieee_d -rw-r--r-- 1 kate.friedman emc 80 Oct 25 15:00 gnorm_data.txt gdas.20211221/00/products/atmos/oznmon: total 8 drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 horiz drwxr-sr-x 2 kate.friedman emc 4096 Oct 25 15:05 time gdas.20211221/00/products/atmos/radmon: total 21036 -rw-r--r-- 1 kate.friedman emc 231 Oct 25 15:12 bad_diag.2021122100 -rw-r--r-- 1 kate.friedman emc 9035 Oct 25 15:12 bad_pen.2021122100 -rw-r--r-- 1 kate.friedman emc 1449 Oct 25 15:12 low_count.2021122100 -rw-r--r-- 1 kate.friedman emc 20523403 Oct 25 15:07 radmon_angle.tar.gz -rw-r--r-- 1 kate.friedman emc 217272 Oct 25 15:08 radmon_bcoef.tar.gz -rw-r--r-- 1 kate.friedman emc 502151 Oct 25 15:10 radmon_bcor.tar.gz -rw-r--r-- 1 kate.friedman emc 264480 Oct 25 15:12 radmon_time.tar.gz -rw-r--r-- 1 kate.friedman emc 684 Oct 25 15:12 warning.2021122100 kate.friedman@dlogin09:/lfs/h2/emc/ptmp/kate.friedman/comrot/testmonitor2> ll gfs.20211221/00/products/atmos/minmon/ total 88 -rw-r--r-- 1 kate.friedman emc 10530 Oct 25 14:55 2021122100.costs.txt -rw-r--r-- 1 kate.friedman emc 63882 Oct 25 14:55 2021122100.cost_terms.txt -rw-r--r-- 1 kate.friedman emc 808 Oct 25 14:55 2021122100.gnorms.ieee_d -rw-r--r-- 1 kate.friedman emc 608 Oct 25 14:55 2021122100.reduction.ieee_d -rw-r--r-- 1 kate.friedman emc 80 Oct 25 14:55 gnorm_data.txt ``` Lingering work and considerations: 1. Issue #1925 2. Consider not hardcoding the list of types in archival and perhaps tie it to something reliable or configurable (maybe into `config.verfozn`?): `subtyplist="gome_metop-b omi_aura ompslp_npp ompsnp_n20 ompsnp_npp ompstc8_n20 ompstc8_npp sbuv2_n19"` Resolves #1908

During the update to stage_ic, the copying of the additional res_N files for 0p25 was omitted. These are now properly copied. Resolves #2027

…S2SW job hanging issue on WCOSS2 (#2021)

@JessicaMeixner-NOAA

Allows cycling in IAU mode with the S2S UFS model, soca and the GSI. Most of the work related to making IAU work was done by @JessicaMeixner-NOAA . Thanks @JessicaMeixner-NOAA ! Resolves #1943 Refs #1776

Cleaned up the logs for CI PRs sent to the GitHub Messages: - Eliminates redundant accumulative reporting - Reports on all created or skipped experiment instantiations in a single message - Gives one line report once for each experiment completion - Single final report completion of CI See some examples in forked [PR 195](TerrenceMcGuinness-NOAA#195) Resolves #2007

Reduce the default number of build jobs for the GDASApp to 8 from 16. This is needed for Orion as the build crashes during a linking step. Though not verified, it appears this may be caused by using too much memory with 16 builds. The issue disappears when using 8 build jobs. Resolves #2029

This PR updates the `ufs-weather-model` to the latest hash that included the dust fix (PR #1922). Along with this I removed the nitrates by default in support of the GEFSv13 EP4 and EP5 experiments. Removed unneeded inputs that should help with speed improvements.

Adding C1152 setting in host.yaml, config.base and config.stage_ic so that can run C1152 S2SW. Resolves #2019

The stack UPP was using on Orion was recently deleted, leaving us unable to build the UFS verison of UPP there. UPP has an updated version using spack-stack that works, so the checkout script is modified to check out that version of UPP after UFS has been checked out. This temporary hack can be removed once we move to a UFS version that advances the UPP version to or beyond 78f369b. Resolves #2041

Fixes a couple things to get workflow to work on Orion again: First, while #2042 allowed post to be built, the `upp.x` executable would fail due to a missing library. Now post jobs (temporarily) uses the UPP modules (plus others needed for runtime) instead of the normal workflow runtime module. This should actually fix #2041. This hack can likely be removed once we move to spack-stack. Second, the launcher command for the analysis jobs on Orion are updated to specify the number of CPUs per task. This was necessary as a recent slurm update on Orion discontinued reading that value from the scheduler. Other jobs will be handled later (see #2044). Additionally, a change is made to `module-setup.sh` that will eliminate the annoyance of `gw_setup.sh` breaking tab completion because undefined variable checking was turned on for the script but not back off. Resolves #1996 Resolves #2041 Refs #2044

The default wave grid for C384 has changed since P8, so a single wave IC on the new grid is available under a different IC name.

This PR addresses issue #1444. The `waveinit` and `waveprep` tasks have been added to the GDAS have cycle. The downstream dependencies for the respective GFS and GDAS forecast tasks have been removed. Resolves #1444

This PR adjusts the `fit2obs` job in the rocoto mesh to better define when it exists and how the archive job dependencies accommodate it not being in the first half cycle. Changes in PR: 1. Remove the `fit2obs` job from the first half cycle `cycledef`. 2. Adjust the archive job dependencies to now include the `fit2obs` job in the new logic check added in PR #1983. The `gdasarch` job in the first half cycle will check if `gdasfit2obs` is complete OR no prior cycle exists. Also cleaned up this new logic check block some to make it a tad simpler. 3. Also, fix the addition of the `fit2obs` config to the config list in setup scripts. Now tied to whether `fit2obs` job is on instead of always adding the config to the list. Resolves #2043

The UFS weather-model (WM) files are place in the COM_CONF path rather than the respective component model COM history paths. Resolves #2015

This PR: - adds an option to warm|cold start the ufs-weather-model when setting up an experiment for GFS/GEFS - adds the capability to stage FV3 restarts when the atmosphere is warm started from FV3 restarts. This would be used in the GEFS free-forecast for part of the reforecast capability. - If a coupled model is warm started for the atmosphere, also check if mediator restarts are present. If so, stage mediator restarts to allow restarting the coupled model. Fixes #2016

This PR creates new individual jobs for the tracker and genesis jobs that are being pulled out of the `vrfy` job (issue #235). The `tracker` and `genesis` jobs will be on by default while the `genesis_fsu` job will be off by default. This mimics the current settings in the `vrfy` job. No enhancements or upgrades are made to the jobs themselves in this PR, only to separate them from the `vrfy` job. However, one change compared to how they currently run in the `vrfy` job is to no longer run the `tracker` in the gdas suite. This mimics how the `ens_tracker` package is run in operations (gfs only). Changes in this PR: 1. Create rocoto job scripts for tracker/genesis jobs. 2. Create configs for tracker/genesis jobs, as well as `config.tropcy` for shared settings. 3. Add tracker/genesis jobs to `config.resources`; use resources from the ecf scripts for `ens_tracker` jobs in operations 4. Add tracker/genesis jobs to rocoto setup for both cycled and forecast-only modes. 5. Update arguments for `jjob_header.sh` script in tracker/genesis JJOB scripts to use specific job names and configs instead of vrfy. 6. Rename `tracker_ver` to `ens_tracker_ver` in version files to mimic variable within tracker package. 7. Rename and move tracker/genesis switches to `config.base`. 8. Remove tracker/genesis jobs from `vrfy` job (`jobs/rocoto/vrfy` and `config.vrfy`). 9. Reduce vrfy job resources to accommodate tasks having been removed. Refs #235 Resolves #1988 Resolves #2048

Fix two bugs in the exgdas_enkf_earc.sh script: 1) Replace ARCH_LIST with DATA to match changes done in PR #2000 and update path to generated archive list files. 2) Add status check and exit after script attempts to make tarball on HPSS. Refs #2064

Use the assim_freq from config.base to configure "offset" in dependencies instead of hard-coding them. Resolves #2038

The hist_avg setting of CICE was changed to be an array.

This PR adds/modifies namelist options and activates new 'ccpp' suite definition files for the updated Unified Gravity Wave Physics (UGWP) in the ufs-weather- model recently updated for HR3. The changes to UGWP are the use of the GSL drag suite for orographic gravity wave drag (GWD) and blocking, as well as the new UGWP Version 1 non-stationary GWD. See ufs-community/ufs-weather-model#1923 for details. Resolves #1986

A few small errors/cleanup in the dependencies, mostly in fcst: A few small errors/cleanup have crept into the dependencies with recent changes, mostly in fcst: - Fixed the offset, as it would yield a string rocoto doesn't recognize when the interval was 24+ hrs - Removed the duplicate wave dependency. When `waveprep` is included, it depends on `waveinit` so there is no need to add a `waveinit` dependency. - Converted an interval in the archive job that was missed in #2039 Resolves #2075

Updates of generating GFS V17 UPP products to support HR3: - Update g2tmpl to 1.0.12 - Add changes to launch cfp on WCOSS2 to generate pgrb2/pgrb2b files - Changes for generating new products (MERRA2, six winter weather variables, 1km HLCY, SDEN) in pgrb2 files Resolves #1965 Resolves #1994

A few of the setup scripts were checking for the existence of functions set by the preamble before calling them so they could also be used stand-alone. However, the `type` command returns `1` if the function name does not exist, which will cause the script to die is `set -e` is on. Now the `type` command fails over to an empty string.

Updates the model to the most recent commit of ufs-weather-model. This should allow for the removal of the UPP hack that was added in PR #2042 This required changing nems.configure -> ufs.configure. Resolves #1942

On WCOSS2 the grib_util module is named a bit different and needs the libjpeg module loaded beforehand.

This PR adds the `--cpus-per-task` flag to non-CFP `APRUN`/srun commands in the env files for Orion and Hera. This is needed as a result of an upgrade to SLURM on Orion that no longer exported the thread value to the environment. Issue #1996 was a hotfix to add this flag for the analysis job that was suddenly running slowly after the SLURM upgrade. A similar SLURM upgrade was done on Hera but the sysadmins there implemented a workaround so this issue did not appear there. This PR adds the `--cpus-per-task` flag for both machines...needed on Orion and as a safety measure on Hera. Also included in this PR is a fix to indentations in the `scripts/exglobal_diag.sh` script that was noticed during work for this PR. Timing differences were not observed for other jobs after adding this srun flag. Job timings were of a similar variation between control and test runs. Any potential additional speedup was not observed. The test runs also reproduced the control runs with `develop`. Resolves #2044

Adds the ability to turn off some coupled components for the ensemble. To enable this, many settings are moved out of `config.fcst` and into `config.ufs` so they are calculated after any overrides of `DO_*` have been made. Some settings also had to be moved out of `config.base` for the same reason. A switch to turn on gocart is added to `config.ufs`, but unlike the other switches this one takes no arguments. The name of the ufs configure template used is converted from a partial file name to the full file name. Resolves #1692

Retires the `vrfy` job from the global-workflow system. All associated scripts, configs, and references to the job are removed. Additionally, the switches listed in RTD were updated to reflect new ones added in other PRs for issue #235. Resolves #2069 Refs #235

This PR addresses issue #1225. The following was accomplished: - A new Rocoto job is added; `jobs/rocoto/npoess.sh`; - `workflow/applications/gfs_cycled.py` is updated to include the `npoess` task; - `workflow/applications/gfs_forecast_only.py` is updated to include the `npoess` task; - A new function is added to `workflow/rocoto/gfs_tasks.py`, `npoess` is added as new GFS (only) task; - `workflow/rocoto/tasks.py` is updated to include the `npoess` task. Resolves #1225

This PR addresses issue #1221. The following was accomplished: - A new Rocoto job is added; `jobs/rocoto/fbwinds.sh`; - `workflow/applications/gfs_cycled.py` is updated to include the `fbwinds` task; - `workflow/applications/gfs_forecast_only.py` is updated to include the `fbwinds` task; - A new function is added to `workflow/rocoto/gfs_tasks.py`, `fbwinds` is added as new GFS (only) task; - `workflow/rocoto/tasks.py` is updated to include the `fbwinds` task. Resolves #1221

Creates a proper j-job for the metp job, moving much of the material from the `jobs/rocoto` script there (some of it within `jjob_header`). Also updates the old `COMIN` variable for the COM refactor (although it does not appear used in the verify scripts currently). This is a partial step towards restoring/improving metp functionality and remains untested. Refs #1575

Add execute permissions (755)

Adds the wave init job to GEFS. Since the output would be the same for every member, the job is only run once for the control, then links are created in each of the member directories pointing to that copy. New ww3_multi and ww3_shell templates for GEFS are copied from the GFS versions. The GEFS restart frequency is changed to not be zero, and the comment about not setting it to zero is copied from the gfs config.base. The restart interval for the GEFS perturbed members is changed to equal that of the control. Also fixes a bug where `RUNwave` was not defined for ensemble forecasts, and another bug in the GEFS stage_ic dependency. Resolves #2017

Refs #2097

#2084) Update module files to spack-stack/1.5.1 and compilers to Intel 2022+ on all non-production machines.

Improved CI robustness for reverting back to **CI-Ready** from any given state New Features: - Improved `scancel` routine (refactored into bash "subroutine") - Improved messaging (see below) when ever a user changes state - Any and all previous build scripts and running experiments are killed as a result of reset to **Ready** Resolves #2060

This PR combines the GDAS and GFS versions of the VMINMON JJOB and ex-driver scripts into single GLOBAL/global versions. Additionally: * some further cleanup of the GSI-Monitor scripts is done; variable settings in ex-driver scripts are moved up to the JJOB level * the fix symlinks are consolidated into a new `fix/mon` subfolder (formerly `fix/gdas`) and copies of the gfs monitor fix files under `fix/products` are removed Resolves #1925

This PR addresses issue #1228. The following is accomplished: - Separate rocoto jobs have been created beneath `jobs/rocoto/` -- `awips_20sh` and `awips_g2sh`; these jobs replace `awips.sh` which was calling multiple J-jobs within the respective `awips.sh` scripts; - New tasks has been added to `workflow/rocoto/tasks.py` for the new AWIPS scripts; - The `gfs_cycled` and `gfs_forecast_only` modules beneath `workflow/rocoto` have been updated accordingly. Resolves #1228

Refs #1356

This PR: - Replaces the use of `JGLOBAL_ATMOS_POST` with `JGLOBAL_ATMOS_UPP` and `JGLOBAL_ATMOS_PRODUCTS`. - Introduces `JGLOBAL_ATMOS_PRODUCTS` that is responsible for creating grib2 products from the master file produced by UPP (inline from the model, or offline by running `JGLOBAL_ATMOS_UPP`). Rocoto job `atmos_products.sh`, ex-script `exglobal_atmos_products.sh` are also added along with a `config.atmos_products`. Updates are made to the relevant `env` files as well as `config.resources` to reflect the addition of this job. - reduces the number of cores needed appropriately for forecast products when there is no offline UPP necessary. - renames older `fv3gfs_dwn_nems.sh` to `interp_atmos_master.sh` and `inter_flux.sh` to `interp_atmos_sflux.sh` - removes `fv3gfs_downstream_nems.sh` as it is absorbed into `exglobal_atmos_products.sh` - removes no longer used scripts for `gfs_post.sh`, `gfs_transfer.sh` (DBN alerts are absorbed into `exglobal_atmos_products.sh`), `post.sh`, `JGLOBAL_ATMOS_POST`, `ex${RUN}_atmos_post.sh` scripts. - Rocoto workflow related scripts are updated to add `atmos_products` as a job. When `WRITE_DOPOST` is `.false.`, `upp.sh` job to run the UPP offline is also added. Dependencies have been appropriately applied. - removes `PGB1F` option that creates 1-degree grib1 products that are no longer needed in GFSv17. The hacks from (now deleted) `post.sh` had to be reinstated in `upp.sh` to load the modules from UPP instead of `load_fv3gfs_modules.sh`.

This add forecast-only support for Hercules to the global workflow. Partially satisfies #1588. Co-authored-by: Walter Kolczynski - NOAA <Walter.Kolczynski@noaa.gov>

Sync develop branch with EMC develop #34

Sync develop branch with EMC develop #34

Commits on Apr 24, 2023

Commits on Apr 25, 2023

Commits on Apr 26, 2023

Commits on Apr 27, 2023

Commits on Apr 28, 2023

Commits on May 1, 2023

Commits on May 2, 2023

Commits on May 3, 2023

Commits on May 4, 2023

Commits on May 5, 2023

Commits on May 8, 2023

Commits on May 9, 2023

Commits on May 10, 2023

Commits on May 11, 2023

Commits on May 12, 2023

Commits on May 15, 2023

Commits on May 16, 2023

Commits on May 17, 2023

Commits on May 18, 2023

Commits on May 19, 2023

Commits on May 22, 2023

Commits on May 23, 2023

Commits on May 26, 2023

Commits on May 30, 2023

Commits on May 31, 2023

Commits on Jun 1, 2023

Commits on Jun 2, 2023

Commits on Jun 6, 2023

Commits on Jun 9, 2023

Commits on Jun 12, 2023

Commits on Jun 13, 2023

Commits on Jun 14, 2023

Commits on Jun 15, 2023

Commits on Jun 18, 2023

Commits on Jun 22, 2023

Commits on Jun 23, 2023

Commits on Jun 26, 2023

Commits on Jun 27, 2023

Commits on Jun 28, 2023

Commits on Jun 30, 2023

Commits on Jul 6, 2023

Commits on Jul 7, 2023

Commits on Jul 10, 2023

Commits on Jul 11, 2023

Commits on Jul 13, 2023

Commits on Jul 14, 2023

Commits on Jul 17, 2023

Commits on Jul 19, 2023

Commits on Jul 20, 2023

Commits on Jul 21, 2023

Commits on Jul 24, 2023

Commits on Jul 26, 2023

Commits on Jul 27, 2023

Commits on Jul 28, 2023

Commits on Aug 4, 2023

Commits on Aug 7, 2023

Commits on Aug 8, 2023

Commits on Aug 9, 2023

Commits on Aug 14, 2023

Commits on Aug 17, 2023

Commits on Aug 18, 2023

Commits on Aug 21, 2023

Commits on Aug 23, 2023

Commits on Aug 24, 2023

Commits on Aug 25, 2023

Commits on Aug 28, 2023

Commits on Aug 30, 2023

Commits on Aug 31, 2023

Commits on Sep 7, 2023

Commits on Sep 8, 2023

Commits on Sep 11, 2023

Commits on Sep 12, 2023

Commits on Sep 13, 2023

Commits on Sep 14, 2023

Commits on Sep 15, 2023

Commits on Sep 18, 2023

Commits on Sep 21, 2023

Commits on Sep 25, 2023