Skip to content

Commit

Permalink
Merge branch 'develop' into feature/version-files
Browse files Browse the repository at this point in the history
* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)
  • Loading branch information
KateFriedman-NOAA committed Jan 30, 2023
2 parents e4e6002 + e092e8b commit e81ed7c
Show file tree
Hide file tree
Showing 529 changed files with 16,986 additions and 58,918 deletions.
31 changes: 31 additions & 0 deletions .github/scripts/build_docs.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
#! /bin/bash

set -eux

# path to docs directory relative to top level of repository
# $GITHUB_WORKSPACE is set if the actions/checkout@v3 action is run first

cwd=$(pwd)
DOCS_DIR="${GITHUB_WORKSPACE}/docs"

# run Make to build the documentation and return to previous directory
cd "${DOCS_DIR}"
make clean html
cd "${cwd}"

# copy HTML output into directory to create an artifact
mkdir -p artifact/documentation
cp -R "${DOCS_DIR}/build/html/." artifact/documentation

# check if the warnings.log file is empty
# Copy it into the artifact and documeentation directories
# so it will be available in the artifacts
warning_file="${DOCS_DIR}/build/warnings.log"
if [[ -s ${warning_file} ]]; then
cp -r "${DOCS_DIR}/build/warnings.log" artifact/doc_warnings.log
cp artifact/doc_warnings.log artifact/documentation
echo "Warnings were encountered while building documentation."
echo "========== Begin warnings =========="
cat artifact/doc_warnings.log
echo "=========== End warnings ==========="
fi
51 changes: 51 additions & 0 deletions .github/workflows/docs.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
name: Build and Deploy Documentation
on:
push:
branches:
- develop
- feature/*
- main/*
- bugfix/*
- release/*
paths:
- docs/**
pull_request:
types: [opened, reopened, synchronize]

jobs:
documentation:
runs-on: ubuntu-latest
name: Build and deploy documentation

steps:
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.9"

- name: Install (upgrade) python dependencies
run: |
pip install --upgrade pip sphinx sphinx-gallery sphinx_rtd_theme sphinxcontrib-bibtex
- name: Checkout
uses: actions/checkout@v3

- name: Build documentation
run: |
./.github/scripts/build_docs.sh
- name: Upload documentation (on success)
uses: actions/upload-artifact@v3
if: always()
with:
name: documentation
path: artifact/documentation

- name: Upload warnings (on failure)
uses: actions/upload-artifact@v3
if: failure()
with:
name: documentation_warnings.log
path: artifact/doc_warnings.log
if-no-files-found: ignore

26 changes: 26 additions & 0 deletions .github/workflows/linters.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
#
name: linters
on:
pull_request:

permissions:
contents: read

defaults:
run:
shell: bash -o pipefail {0}

jobs:
lint:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v3
with:
fetch-depth: 0

- name: Lint shell scripts
uses: redhat-plumbers-in-action/differential-shellcheck@v3
with:
token: ${{ secrets.GITHUB_TOKEN }}
24 changes: 24 additions & 0 deletions .github/workflows/pynorms.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
name: Check Norms
on: [push, pull_request]

jobs:
check_norms:
runs-on: ubuntu-latest
name: Check Python coding norms with pycodestyle

steps:

- name: Install dependencies
run: |
pip install --upgrade pip
pip install pycodestyle
- name: Checkout
uses: actions/checkout@v3
with:
path: global-workflow

- name: Run pycodestyle
run: |
cd $GITHUB_WORKSPACE/global-workflow
pycodestyle -v --config ./.pycodestyle --exclude='.git,.github' ./
36 changes: 36 additions & 0 deletions .github/workflows/pytests.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
name: Run pytests
on: [push, pull_request]

jobs:
run_pytests:
runs-on: ubuntu-latest
name: Install pygw and run tests with pytests
strategy:
max-parallel: 1
matrix:
python: ["3.7", "3.8", "3.9", "3.10"]

steps:
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python }}

- name: Install (upgrade) python dependencies
run: |
pip install --upgrade pip
- name: Checkout
uses: actions/checkout@v3
with:
path: global-workflow

- name: Install pygw
run: |
cd $GITHUB_WORKSPACE/global-workflow/ush/python/pygw
pip install .[dev]
- name: Run pytests
run: |
cd $GITHUB_WORKSPACE/global-workflow/ush/python/pygw
pytest -v src/tests
114 changes: 29 additions & 85 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,14 @@ __pycache__
*.[aox]
*.mod
*.sw[a-p]
._*
.DS_Store
.idea/
.vscode/

# Ignore editor generated backup files
#-------------------------------------
*~
# Ignore folders
#-------------------
exec/
Expand All @@ -17,8 +22,23 @@ install*/
# Ignore fix directory symlinks
#------------------------------
fix/0readme
fix/fix_*
fix/gdas/
fix/aer
fix/am
fix/chem
fix/cice
fix/cpl
fix/datm
fix/gdas
fix/gldas
fix/gsi
fix/lut
fix/mom6
fix/orog
fix/reg2grb2
fix/sfc_climo
fix/ugwd
fix/verif
fix/wave
fix/wafs

# Ignore parm file symlinks
Expand Down Expand Up @@ -75,71 +95,12 @@ parm/wafs
#--------------------------------------------
sorc/*log
sorc/logs
sorc/ufs_model.fd
sorc/gfs_post.fd
sorc/gfs_wafs.fd
sorc/gldas.fd
sorc/gsi_enkf.fd
sorc/gsi.fd
sorc/enkf.fd
sorc/gdas.cd
sorc/gsi_utils.fd
sorc/gsi_monitor.fd
sorc/ufs_utils.fd
sorc/verif-global.fd

# Ignore sorc symlinks
#---------------------
sorc/calc_analysis.fd
sorc/calc_increment_ens.fd
sorc/calc_increment_ens_ncio.fd
sorc/emcsfc_ice_blend.fd
sorc/emcsfc_snow2mdl.fd
sorc/fregrid.fd
sorc/gdas2gldas.fd
sorc/getsfcensmeanp.fd
sorc/getsigensmeanp_smooth.fd
sorc/getsigensstatp.fd
sorc/gfs_ncep_post.fd
sorc/gldas2gdas.fd
sorc/gldas_forcing.fd
sorc/gldas_model.fd
sorc/gldas_post.fd
sorc/gldas_rst.fd
sorc/global_chgres.fd
sorc/global_cycle.fd
sorc/global_enkf.fd
sorc/global_gsi.fd
sorc/interp_inc.fd
sorc/make_hgrid.fd
sorc/make_solo_mosaic.fd
sorc/ncdiag_cat.fd
sorc/nst_tf_chg.fd
sorc/oznmon_horiz.fd
sorc/oznmon_time.fd
sorc/radmon_angle.fd
sorc/radmon_bcoef.fd
sorc/radmon_bcor.fd
sorc/radmon_time.fd
sorc/recentersigp.fd
sorc/upp.fd
sorc/wafs_awc_wafavn.fd
sorc/wafs_blending.fd
sorc/wafs_blending_0p25.fd
sorc/wafs_cnvgrib2.fd
sorc/wafs_gcip.fd
sorc/wafs_grib2_0p25.fd
sorc/wafs_makewafs.fd
sorc/wafs_setmissing.fd
sorc/*.cd
sorc/*.fd

# Ignore scripts from externals
#------------------------------
# jobs symlinks
jobs/JGDAS_ATMOS_GLDAS
jobs/JGDAS_ATMOS_VERFOZN
jobs/JGDAS_ATMOS_VERFRAD
jobs/JGDAS_ATMOS_VMINMON
jobs/JGFS_ATMOS_VMINMON
jobs/JGFS_ATMOS_WAFS
jobs/JGFS_ATMOS_WAFS_BLENDING
jobs/JGFS_ATMOS_WAFS_BLENDING_0P25
Expand All @@ -148,11 +109,6 @@ jobs/JGFS_ATMOS_WAFS_GRIB2
jobs/JGFS_ATMOS_WAFS_GRIB2_0P25
# scripts symlinks
scripts/exemcsfc_global_sfc_prep.sh
scripts/exgdas_atmos_gldas.sh
scripts/exgdas_atmos_verfozn.sh
scripts/exgdas_atmos_verfrad.sh
scripts/exgdas_atmos_vminmon.sh
scripts/exgfs_atmos_vminmon.sh
scripts/exgfs_atmos_wafs_blending.sh
scripts/exgfs_atmos_wafs_blending_0p25.sh
scripts/exgfs_atmos_wafs_gcip.sh
Expand All @@ -168,29 +124,17 @@ ush/fv3gfs_driver_grid.sh
ush/fv3gfs_filter_topo.sh
ush/fv3gfs_make_grid.sh
ush/fv3gfs_make_orog.sh
ush/gldas_archive.sh
ush/gldas_forcing.sh
ush/gldas_get_data.sh
ush/gldas_liscrd.sh
ush/gldas_post.sh
ush/gldas_process_data.sh
ush/global_chgres.sh
ush/global_chgres_driver.sh
ush/global_cycle.sh
ush/global_cycle_driver.sh
ush/minmon_xtrct_costs.pl
ush/minmon_xtrct_gnorms.pl
ush/minmon_xtrct_reduct.pl
ush/mkwfsgbl.sh
ush/ozn_xtrct.sh
ush/radmon_ck_stdout.sh
ush/radmon_err_rpt.sh
ush/radmon_verf_angle.sh
ush/radmon_verf_bcoef.sh
ush/radmon_verf_bcor.sh
ush/radmon_verf_time.sh
ush/ufsda
ush/rstprod.sh
ush/wafs_blending.sh
ush/wafs_grib2.regrid.sh
ush/wafs_intdsk.sh
ush/finddate.sh
ush/make_NTC_file.pl
ush/make_ntc_bull.pl
ush/make_tif.sh
ush/month_name.sh
6 changes: 6 additions & 0 deletions .pycodestyle
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
[pycodestyle]
count = False
ignore = E402,W504
max-line-length = 160
statistics = True
exclude = Experimental
13 changes: 13 additions & 0 deletions .shellcheckrc
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
# Global settings for Shellcheck (https://github.com/koalaman/shellcheck)
enable=all

external-sources=false

# Disable variable referenced but not assigned
disable=SC2154

# Disable following non-constant source
disable=SC1090

# Disable non-existent binary
disable=SC1091
Loading

0 comments on commit e81ed7c

Please sign in to comment.