Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix checking for restart files #1186

Merged
merged 2 commits into from Dec 18, 2022
Merged

Fix checking for restart files #1186

merged 2 commits into from Dec 18, 2022

Conversation

XianwuXue-NOAA
Copy link
Contributor

@XianwuXue-NOAA XianwuXue-NOAA commented Dec 16, 2022 •
edited by WalterKolczynski-NOAA

Description
Undoes the portion of PR #1179 that caused a new bug while attempting to fix #1140, without removing the linter fixes. Instead /dev/null is 'searched' if ${RSTDIR_ATM} is not defined. That situation will always result in zero files found, ensuring a rerun is not triggered.

Fixes #1140
Fixes #1185
Moots #1190

Type of change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

  • Cycled with early EnKF on Orion
  • Full coupled on Orion
  • Forecast-only on Orion

Checklist

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • New and existing tests pass with my changes

@XianwuXue-NOAA
Fix a bug for running fcst jobs for gdas
398a3cf 
 On branch hotfix/fix_forecast_det_1185
	modified:   ush/forecast_det.sh

Refs: NOAA-EMC#1185
@WalterKolczynski-NOAA
Copy link
Contributor

This undoes a fix I just merged this morning: #1179

I suspect this is only a problem with early-cycle enkf, which (I think) uses gfs as the $CDUMP. That line should short-circuit false before reaching filecount for gdas.

What we need is a more direct solution to the original issue this morning's PR was supposed to fix (RSTDIR_ATM being unbound). Use this:

filecount=$(find "${RSTDIR_ATM:-/dev/null}" -type f | wc -l)

@XianwuXue-NOAA
Improve the statement
cc311e3 
 On branch hotfix/fix_forecast_det_1185
	modified:   ush/forecast_det.sh

Refs: NOAA-EMC#1185
@XianwuXue-NOAA
Copy link
Contributor Author

This undoes a fix I just merged this morning: #1179

I suspect this is only a problem with early-cycle enkf, which (I think) uses gfs as the $CDUMP. That line should short-circuit false before reaching filecount for gdas.

What we need is a more direct solution to the original issue this morning's PR was supposed to fix (RSTDIR_ATM being unbound). Use this:

filecount=$(find "${RSTDIR_ATM:-/dev/null}" -type f | wc -l)

Good method. In fact, it will not affect not only the early enkf, but also the late enkf. I just push a new comment based on your suggestions.

WalterKolczynski-NOAA
WalterKolczynski-NOAA approved these changes Dec 16, 2022
View reviewed changes
Copy link
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good once all the testing is complete

@XianwuXue-NOAA
Copy link
Contributor Author

@WalterKolczynski-NOAA Tested and confirmed the changes

@CoryMartin-NOAA
Copy link
Contributor

This is not just a problem with early cycle EnKF, I just found a similar issue with the half cycle gdasfcst run.

@XianwuXue-NOAA
Copy link
Contributor Author

This is not just a problem with early cycle EnKF, I just found a similar issue with the half cycle gdasfcst run.

@CoryMartin-NOAA Thanks a lot for your checking. Yes, same in my test. So this PR should fix the issue.

@aerorahul
Copy link
Contributor

The offending line here was added as part of the PR #1179
In the PR description, it says that a forecast-only test was performed.
This section of the script is used also by the cycled mode.
Please revert PR #1179 and re-open issue #1140
The PR #1179 may be resolving the particular issue in #1140 but is causing other issues.

@lgannoaa
Copy link
Contributor

The test with #1140 was done with forecast-only. However, the filecount variable in question was not initialized. That caused problem when running gdas and enkf. I have a fix and is in testing now.
Will update issue 1140 and submit PR as needed.

@WalterKolczynski-NOAA
Copy link
Contributor

I'm able to run cycled (with early cycle), fo, and coupled all successfully using this PR, with no mention of any unbound variable in the forecast logs.

@XianwuXue-NOAA
Copy link
Contributor Author

I'm able to run cycled (with early cycle), fo, and coupled all successfully using this PR, with no mention of any unbound variable in the forecast logs.

Thanks. I ran 1.5 cycles with early enkf C768/C384 successfully

@WalterKolczynski-NOAA WalterKolczynski-NOAA changed the title Fix a bug for running fcst jobs for gdas Fix checking for restart files Dec 18, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA merged commit 8581eac into NOAA-EMC:develop Dec 18, 2022
@WalterKolczynski-NOAA WalterKolczynski-NOAA mentioned this pull request Dec 18, 2022
Closed
7 tasks
@XianwuXue-NOAA XianwuXue-NOAA deleted the hotfix/fix_forecast_det_1185 branch January 26, 2023 00:19
KateFriedman-NOAA added a commit to KateFriedman-NOAA/global-workflow that referenced this pull request Jan 30, 2023
@KateFriedman-NOAA
Merge branch 'develop' into feature/version-files
e81ed7c 
* develop:
  Correct issue in linking final restart files (NOAA-EMC#1285)
  Remove execute permissions from config files (NOAA-EMC#1281)
  Make needed updates to run forecast from GEFS (NOAA-EMC#1203)
  Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259)
  Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237)
  Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280)
  More marine DA j-jobs (NOAA-EMC#1270)
  Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265)
  Add new jjob for ocean analysis bmat (NOAA-EMC#1239)
  Retire ecf/versions in develop (NOAA-EMC#1267)
  Deploy documentation to RTD (NOAA-EMC#1264)
  Temporarily disable failing pytest (NOAA-EMC#1263)
  Remove incorrect/misleading comments in config.base (NOAA-EMC#1261)
  Add initial Sphinx documentation (NOAA-EMC#1258)
  Remove nemsio support (NOAA-EMC#1255)
  Increase wallclock for diag jobs (NOAA-EMC#1216)
  Use correct resources for GFS gempak (NOAA-EMC#1214)
  Abstract common j-job tasks (NOAA-EMC#1230)
  Add missing mkgfsawps.x link (NOAA-EMC#1218)
  Fix post sounding job (NOAA-EMC#1212)
  Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240)
  Use fracoro data for all new UFS applications (NOAA-EMC#1182)
  Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238)
  Add more user defined parameters for the marine DA (NOAA-EMC#1235)
  Update pytests action version and run sequentially (NOAA-EMC#1236)
  Add utility to compare Fortran namelists (NOAA-EMC#1234)
  Updates for pygw (NOAA-EMC#1231)
  Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)
  Move member up in directory hierarchy (NOAA-EMC#1201)
  Enable staging ics for cycled experiments. (NOAA-EMC#1199)
  Add tests for configuration.py (NOAA-EMC#1192)
  Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191)
  define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193)
  Fix checking for restart files (NOAA-EMC#1186)
  Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188)
  Update archive job memory request value for R&Ds (NOAA-EMC#1183)
  Reorder post so all flux files are generated when running offline (NOAA-EMC#1181)
  Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179)
  Add missing jobids in some pre-job scripts (NOAA-EMC#1176)
  Remove existing directory if it exists when getic runs (NOAA-EMC#1165)
  Add logging decorator, test and test for yaml_file (NOAA-EMC#1178)
  fix coding norm check in `hosts.py` (NOAA-EMC#1174)
  Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172)
  Support for the GDASApp testing in containers (NOAA-EMC#1151)
  ATM 3DVAR with and without IAU (NOAA-EMC#1113)
  Enable checking for python norms and fix violating code (NOAA-EMC#1168)
  Enforce decimal math in atmos post (NOAA-EMC#1171)
  Update marine DA j-jobs to new format (NOAA-EMC#1149)
  Add utility to manipulate files en masse  (NOAA-EMC#1166)
  add action to run pytests (NOAA-EMC#1167)
  Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162)
  Add a task base class and basic logger (NOAA-EMC#1160)
  Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154)
  move configuration.py to pygw. Use it from there.  return AttrDict after sourcing configs (NOAA-EMC#1153)
  JEDI based Marine DA tasks (NOAA-EMC#1134)
  Allow customizations based on user/configuration (NOAA-EMC#1146)
  First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120)
  enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142)
  Fix typo in .shellcheckrc
  Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138)
  Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132)
  Address shellcheck warnings in env files (NOAA-EMC#1136)
  Adds group size and nmem for GEFS (NOAA-EMC#1127)
  Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133)
  Convert archive jobs to proper j-jobs (NOAA-EMC#1115)
  Update C48 forecast to run with one thread (NOAA-EMC#1131)
  Improved error messages from atmos analysis (NOAA-EMC#1125)
  Update MODULEPATH for Orion (NOAA-EMC#1126)
  MPMD variable updates and fix (NOAA-EMC#1124)
  Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122)
  Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112)
  Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111)
  Correct syntax in remaining sorc scripts (NOAA-EMC#1105)
  Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104)
  Add Early Cycle EnKF workflow (NOAA-EMC#1022)
  Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101)
  Fixed gfs-utils links (NOAA-EMC#1099)
  Fix build scripts and bring into compliance (NOAA-EMC#1096)
  Feature/updates for gdas app (NOAA-EMC#1091)
  Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094)
  Resource updates to support WCOSS2 (NOAA-EMC#1070)
  Set COMPILER in link for detect machine (NOAA-EMC#1092)
  gfs utils update (NOAA-EMC#1088)
  GFS-UTILS update for build and ush scripts (NOAA-EMC#1082)
  Update UFS version to 2022 Oct 19 (NOAA-EMC#1083)
  Use more cycledefs for task control (NOAA-EMC#1078)
  removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079)
  Update UFS to Sept 9 version (NOAA-EMC#1073)
  Modify default file location for monitor data when using rocoto (NOAA-EMC#1065)
  Fix companion ocean resolution for C48 (NOAA-EMC#1066)
  Add trailing slash for gldas topo path (NOAA-EMC#1064)
  Limit number of CPU for post (NOAA-EMC#1061)
  Fix eupd trace (NOAA-EMC#1057)
  Port to S4 (NOAA-EMC#1023)
  Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049)
  Add GDAS to the partial build list (NOAA-EMC#1050)
  Fix group number being treated as octal in gdas arch (NOAA-EMC#1053)
  Remove trace from link script (NOAA-EMC#1046)
  Update gfs-utils hash to 3a609ea (NOAA-EMC#1048)
  Fix link script usage statement (NOAA-EMC#1045)
  Replace preamble variable commands with functions (NOAA-EMC#1012)
  Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009)
  Rename post scripts (NOAA-EMC#1038)
  Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039)
  WCOSS2 run support and script/config updates (NOAA-EMC#1030)
  Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
  initial commit for incoming yaml work (NOAA-EMC#1029)
  Fix radiance verification failing to find diag files (NOAA-EMC#1031)
  Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
  Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018)
  Update GSI Monitor for radmon fix
  Correct shell linter config (NOAA-EMC#1013)
  Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016)
  Add shell linter Github action for pull requests (NOAA-EMC#1007)
  Build updates for WCOSS2 (NOAA-EMC#1002)
  Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001)
  Fix preamble id (NOAA-EMC#996)
  Add missing "atmos" into job dependencies (NOAA-EMC#998)
  Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992)
  Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990)
  Add GSI monitor scripts (NOAA-EMC#969)
  Fix product generation at some fcst hrs (NOAA-EMC#988)
  Add initial config files for global aerosol DA (NOAA-EMC#986)
  Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979)
  use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970)
  Externals.cfg was stale and had drifted off (NOAA-EMC#965)
  Fix post comparison with zero-padded numbers (NOAA-EMC#964)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@WalterKolczynski-NOAA WalterKolczynski-NOAA WalterKolczynski-NOAA approved these changes

@aerorahul aerorahul Awaiting requested review from aerorahul

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Fix a bug in ush/forecast_det.sh Unbound variable (RSTDIR_ATM) in forecast_det.sh (minor bug)
5 participants
@XianwuXue-NOAA @WalterKolczynski-NOAA @CoryMartin-NOAA @aerorahul @lgannoaa