Remove unnecessary variables which reference to nemsio #1259

lgannoaa · 2023-01-20T18:24:39Z

Description
Remove unnecessary variables which reference to nemsio (part 2 of #601)

Wire OUTPUT_FILE as netcdf
Replace SUFFIX with wired .nc name
Remove OUTPUT_FILE
Remove use of affix and replace with wired name "nc"
Replace format with wired name "netcdf"

Fixes #601

How Has This Been Tested?

Low Res Cycled test on Orion
Forecast-only test on Hera

Hera fcst only
HOMEgfs /scratch1/NCEPDEV/global/Lin.Gan/git/PR/issue-601-p2
pslot issue-601-p2
expdir /scratch1/NCEPDEV/global/Lin.Gan/expdir/issue-601-p2
COM: /scratch1/NCEPDEV/stmp2/Lin.Gan/ptmp/com/issue-601-p2
log: /scratch1/NCEPDEV/stmp2/Lin.Gan/ptmp/com/issue-601-p2/logs

Orion cycled low res:
HOMEgfs: /work/noaa/global/lgan/git/PR/issue-601-p2
pslot: issue-601-p2
expdir: /work/noaa/global/lgan/expdir/issue-601-p2
COM: /work/noaa/stmp/lgan/comrot/issue-601-p2/
log: /work/noaa/stmp/lgan/comrot/issue-601-p2/logs

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
My changes generate no new warnings

Wire OUTPUT_FILE as netcdf reference to nemsio Replace SUFFIX with wired .nc name Remove OUTPUT_FILE Remove use of affix and replace with wired name "nc" Replace format with wired name "netcdf" Replace fformat to wire in "nc" Remove local unreferenced use of SUFFIX Part 2 of the GitHub: 601

lgannoaa · 2023-01-20T18:44:53Z

@aerorahul This PR is currently marked as draft because it has not been tested. Because there are 40 files changed, it will be efficient to get your review on file level change before start running test. Thanks

… as needed. Update workflow_tasks.py for this issue

aerorahul

Looks right.
Tests were run for forecast-only and cycled and all jobs ran successfully (as evident from logs)
This is a major clean up of removing obsolete and long deprecated functionality.

aerorahul

@lgannoaa
Why is ush/drive_makeprepbufr.sh being removed?
And if you removed it, how did your tests cycled tests run/pass?

WalterKolczynski-NOAA

Looks to be in pretty good shape. A couple settings yet that you might be able to remove, and a script was removed I wasn't expecting.

WalterKolczynski-NOAA · 2023-01-25T20:56:50Z

ush/hpssarch_gen.sh

+      echo  "${dirname}${head}atmf${fhr}.ensmean.nc       " >>enkf${CDUMP}.txt
+      echo  "${dirname}${head}sfcf${fhr}.ensmean.nc       " >>enkf${CDUMP}.txt
+      if [ -s $ROTDIR/${dirpath}${head}atmf${fhr}.ensspread.nc ]; then
+	echo  "${dirname}${head}atmf${fhr}.ensspread.nc     " >>enkf${CDUMP}.txt


Please fix indentation

WalterKolczynski-NOAA · 2023-01-25T21:20:43Z

ush/gfs_bufr_netcdf.sh

@@ -82,7 +80,7 @@ do
   ic=0
   while [ $ic -lt 1000 ]
   do
-      if [ ! -f $COMIN/${RUN}.${cycle}.logf${hh2}.${fformat} ]
+      if [ ! -f $COMIN/${RUN}.${cycle}.logf${hh2}.txt ]


This is a change (txt instead of .nc). Was the wrong filename being tested before?

If you take a the COM. It should be the logf.txt. There is no such file called logf.nc

WalterKolczynski-NOAA · 2023-01-25T21:23:24Z

ush/drive_makeprepbufr.sh

@@ -1,142 +0,0 @@
-#! /usr/bin/env bash


Why is this script being removed? Is it related to nemsio somehow?

This is a hardwired driver only configured for nemsio. Therefore, it is removed.

jobs/rocoto/vrfy.sh

scripts/exgfs_atmos_postsnd.sh

lgannoaa · 2023-01-25T22:14:41Z

Why is ush/drive_makeprepbufr.sh being removed?
And if you removed it, how did your tests cycled tests run/pass?

@aerorahul this is a hardwired driver only process nemsio work. Therefore, it should be removed. This script is not used in test.

aerorahul · 2023-01-25T22:26:34Z

Why is ush/drive_makeprepbufr.sh being removed?
And if you removed it, how did your tests cycled tests run/pass?

@aerorahul this is a hardwired driver only process nemsio work. Therefore, it should be removed. This script is not used in test.

I don't think so. This script creates the prepbufr file. It is very much used and using netcdf model output.

aerorahul · 2023-01-25T22:29:50Z

I take that back. Let me take a closer look. This file might have a different use.

…1-p2

aerorahul · 2023-01-25T22:33:29Z

Why is ush/drive_makeprepbufr.sh being removed?
And if you removed it, how did your tests cycled tests run/pass?

@aerorahul this is a hardwired driver only process nemsio work. Therefore, it should be removed. This script is not used in test.

It is no longer used, and can be removed. If I remember, this was introduced at a time when it was difficult or not possible to call JGLOBAL_PREP directly.

…obal-workflow into feature/issue-601-p2

* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)

lgannoaa self-assigned this Jan 20, 2023

lgannoaa requested a review from aerorahul January 20, 2023 18:45

lgannoaa and others added 8 commits January 23, 2023 19:14

Merge branch 'develop' into feature/issue-601-p2

80859cd

Remove reference to GSUFFIX and ASUFFI. Remove nemsio only scripts

8f9157f

merge develop

c8f8311

Merge conflict

4c6da74

Clean up testing script

f6fc5b6

Clean up space

2b01ab8

Clean up exgdas_enkf_sfc.sh

7927df4

Using GSUFFIX in JGDAS_ENKF_SELECT_OBS and exglobal_atmos_analysis.sh…

9e634dd

… as needed. Update workflow_tasks.py for this issue

lgannoaa marked this pull request as ready for review January 24, 2023 20:27

aerorahul requested a review from WalterKolczynski-NOAA January 25, 2023 03:07

aerorahul approved these changes Jan 25, 2023

View reviewed changes

aerorahul requested changes Jan 25, 2023

View reviewed changes

WalterKolczynski-NOAA requested changes Jan 25, 2023

View reviewed changes

lgannoaa added 2 commits January 25, 2023 22:30

Remve indentation in hpssarch_gen.sh

a8cecef

Merge remote-tracking branch 'upstream/develop' into feature/issue-60…

057251d

…1-p2

aerorahul self-requested a review January 25, 2023 22:33

aerorahul approved these changes Jan 25, 2023

View reviewed changes

Merge branch 'feature/issue-601-p2' of https://github.com/lgannoaa/gl…

310d0c4

…obal-workflow into feature/issue-601-p2

WalterKolczynski-NOAA approved these changes Jan 25, 2023

View reviewed changes

WalterKolczynski-NOAA merged commit b576389 into NOAA-EMC:develop Jan 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove unnecessary variables which reference to nemsio #1259

Remove unnecessary variables which reference to nemsio #1259

lgannoaa commented Jan 20, 2023 •

edited by WalterKolczynski-NOAA

Loading

lgannoaa commented Jan 20, 2023

aerorahul left a comment

aerorahul left a comment •

edited

Loading

WalterKolczynski-NOAA left a comment

WalterKolczynski-NOAA Jan 25, 2023

WalterKolczynski-NOAA Jan 25, 2023

lgannoaa Jan 25, 2023

WalterKolczynski-NOAA Jan 25, 2023

lgannoaa Jan 25, 2023

lgannoaa commented Jan 25, 2023

aerorahul commented Jan 25, 2023

aerorahul commented Jan 25, 2023

aerorahul commented Jan 25, 2023

Remove unnecessary variables which reference to nemsio #1259

Remove unnecessary variables which reference to nemsio #1259

Conversation

lgannoaa commented Jan 20, 2023 • edited by WalterKolczynski-NOAA Loading

lgannoaa commented Jan 20, 2023

aerorahul left a comment

Choose a reason for hiding this comment

aerorahul left a comment • edited Loading

Choose a reason for hiding this comment

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 25, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 25, 2023

Choose a reason for hiding this comment

lgannoaa Jan 25, 2023

Choose a reason for hiding this comment

WalterKolczynski-NOAA Jan 25, 2023

Choose a reason for hiding this comment

lgannoaa Jan 25, 2023

Choose a reason for hiding this comment

lgannoaa commented Jan 25, 2023

aerorahul commented Jan 25, 2023

aerorahul commented Jan 25, 2023

aerorahul commented Jan 25, 2023

lgannoaa commented Jan 20, 2023 •

edited by WalterKolczynski-NOAA

Loading

aerorahul left a comment •

edited

Loading