More marine DA j-jobs #1270

guillaumevernieres · 2023-01-23T16:51:00Z

Description
Addition of 2 j-jobs:

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY
This job currently dumps a few files of the impulse response of B. Not meant to ever go in ops.
jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
This job will eventually generate relevant figures from the marine DA cycle. It currently points to a script that does not yet exist.

I also took this PR opportunity to:

update how we load the modules, following what was done for the atmosphere.
modify ush/load_ufsda_modules.sh so we could optionally specify what to load (GDAS or EVA), default is GDAS when no argument is present
Use POST to copy what's needed in COM
fixes Use the marine DA POST j-job to copy what we keep from the cycle in COMROT #1273

Type of change

New feature (non-breaking change which adds functionality)

How Has This Been Tested?
GDASApp CI/Container
GDASApp CI/Orion for the JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY jjob

Checklist

My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

github-advanced-security

shellcheck found more than 10 potential problems in the proposed changes. Check the Files changed tab for more details.

jobs/rocoto/ocnanalvrfy.sh

guillaumevernieres · 2023-01-23T17:58:24Z

preemptively assuming the last commit fixes the linter check ...

aerorahul

Most changes look ok.
I have a few clarifying questions that probably @CoryMartin-NOAA could answer better as he knows how the env files are used to set variables for jobs.

aerorahul · 2023-01-23T18:01:13Z

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY

+source "${HOMEgfs}/ush/preamble.sh"
+
+
+export DATA="${DATAROOT}/${RUN}ocnanal_${cyc}"


Is there a reason why every DATA is ${RUN}ocnanl_${cyc}?
We need this solely if it will need to be preserved after this job is completed.
If it is not needed, then we should stick with the default ${job}.${jobid} that are defined upstream.

This comment applies to the JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY as well.

yes, at least for now it needs to be preserved.

This directory is being gutted here:

global-workflow/jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_POST

Line 21 in f98433f

[[ "${KEEPDATA}" = "NO" ]] && rm -rf "${DATA}"

I suggest the post job to copy what it needs to COM and create a working directory for this job in $DATAROOT

ok, will look into it @aerorahul

@aerorahul , I created an issue here to cleanup how we keep what we need from the cycle. Happy to add the necessary changes to that PR, or to do it in a different PR.

To elaborate a bit further: $DATA should only contain temporary data during runtime and is wiped at the end of the job. Typically that means a random directory name is okay. With the prep/run/post paradigm, that doesn't work because RUN needs to know where PREP staged stuff, but prep/run/post should still be considered a single job for purposes of $DATA and deleted at the end of POST. Anything that is needed outside of that "one" job needs to get placed into $COM.

Understood @WalterKolczynski-NOAA , I'll push the modifications that address that issue (#1273 ) later today ... maybe.
I'll use the default $DATA (default is ${job}.${jobid}?) for the JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY, that job will use data saved in $COM. The rest of the marine DA j-jobs all depend on PREP, so I'll have to stick to the specified $DATA=${RUN}ocnanal_${cyc}.

aerorahul · 2023-01-23T18:02:00Z

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY

+
+
+export DATA="${DATAROOT}/${RUN}ocnanal_${cyc}"
+source "${HOMEgfs}/ush/jjob_header.sh" -e "ocnanalrun" -c "base ocnanal ocnanalprep"


Is this job going to use the same resources of ocnanalrun? I highly doubt that.

oops, nope.

guillaumevernieres · 2023-01-23T18:40:28Z

humm ... looks like I pushed the shellcheck fix to the aether!

AndrewEichmann-NOAA

Looks plausible

RussTreadon-NOAA · 2023-01-24T14:22:45Z

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY

+###############################################################
+# Run relevant script
+
+EXSCRIPT=${GDASPREPPY:-${HOMEgfs}/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_bmat_vrfy.sh}


Will this script eventually move to g-w scripts?

Yes, at some point in the future @RussTreadon-NOAA .

OK. I understand the utility in keeping it in GDASApp for the time being.

RussTreadon-NOAA · 2023-01-24T14:23:24Z

jobs/JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY

+###############################################################
+# Run relevant script
+
+EXSCRIPT=${GDASPREPPY:-${HOMEgfs}/sorc/gdas.cd/scripts/exgdas_global_marine_analysis_vrfy.sh}


Same question as before. Will this script eventually move to g-w scripts/?

RussTreadon-NOAA

Changes look straightforward to me. Approve.

guillaumevernieres · 2023-01-24T18:59:28Z

@aerorahul , @WalterKolczynski-NOAA , I think I addressed your comments.
It "works" with a companion branch that I have yet to push in the GDASApp,
POST is now actually doing something, I'll move the file manipulation in a dedicated script later, as @aerorahul suggested.

gvernier:gw$ ctest 
Test project /home/gvernier/wrk/gw-nameupdate/sorc/gdas.cd/build/test/soca/gw
    Start 1: test_gdasapp_soca_run_clean
1/5 Test #1: test_gdasapp_soca_run_clean ..........................   Passed    0.01 sec
    Start 2: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP
2/5 Test #2: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP ...   Passed    4.16 sec
    Start 3: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN
3/5 Test #3: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN ....   Passed   23.65 sec
    Start 4: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST
4/5 Test #4: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST ...   Passed    1.52 sec
    Start 5: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY
5/5 Test #5: test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY ...   Passed    3.91 sec

100% tests passed, 0 tests failed out of 5

Total Test time (real) =  33.26 sec

guillaumevernieres · 2023-01-25T18:28:57Z

@WalterKolczynski-NOAA , @aerorahul , I've addressed your comments. Could I get a quick review/merge?

* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)

guillaumevernieres added 3 commits January 23, 2023 11:13

bmat_vrfy and ana_vrfy jobs

5f1fdc0

forgotten files ...

f9f82e4

tidy run job

c785bdd

github-advanced-security bot found potential problems Jan 23, 2023

View reviewed changes

code norm

92f794c

github-advanced-security bot found potential problems Jan 23, 2023

View reviewed changes

jobs/rocoto/ocnanalvrfy.sh Fixed Show fixed Hide fixed

jobs/rocoto/ocnanalvrfy.sh Fixed Show fixed Hide fixed

guillaumevernieres requested review from WalterKolczynski-NOAA, aerorahul, RussTreadon-NOAA, CoryMartin-NOAA and AndrewEichmann-NOAA January 23, 2023 17:57

aerorahul reviewed Jan 23, 2023

View reviewed changes

wrong resources

b7ea944

more shellcheck

71490f1

guillaumevernieres mentioned this pull request Jan 23, 2023

Use the marine DA POST j-job to copy what we keep from the cycle in COMROT #1273

Closed

AndrewEichmann-NOAA reviewed Jan 23, 2023

View reviewed changes

RussTreadon-NOAA reviewed Jan 24, 2023

View reviewed changes

RussTreadon-NOAA approved these changes Jan 24, 2023

View reviewed changes

guillaumevernieres added 2 commits January 24, 2023 13:37

cp stuff to com

5afef87

fixed shell norms

ede4396

guillaumevernieres mentioned this pull request Jan 24, 2023

Plot marine DA stuff NOAA-EMC/GDASApp#286

Merged

3 tasks

WalterKolczynski-NOAA approved these changes Jan 25, 2023

View reviewed changes

WalterKolczynski-NOAA merged commit f78afeb into NOAA-EMC:develop Jan 25, 2023

guillaumevernieres deleted the feature/more-soca-jjobs branch March 27, 2023 13:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More marine DA j-jobs #1270

More marine DA j-jobs #1270

guillaumevernieres commented Jan 23, 2023 •

edited

Loading

github-advanced-security bot left a comment

guillaumevernieres commented Jan 23, 2023

aerorahul left a comment

aerorahul Jan 23, 2023

guillaumevernieres Jan 23, 2023

aerorahul Jan 23, 2023

guillaumevernieres Jan 23, 2023

guillaumevernieres Jan 23, 2023

WalterKolczynski-NOAA Jan 24, 2023

guillaumevernieres Jan 24, 2023

aerorahul Jan 23, 2023

guillaumevernieres Jan 23, 2023

guillaumevernieres commented Jan 23, 2023

AndrewEichmann-NOAA left a comment

RussTreadon-NOAA Jan 24, 2023

guillaumevernieres Jan 24, 2023

RussTreadon-NOAA Jan 24, 2023

RussTreadon-NOAA Jan 24, 2023

RussTreadon-NOAA Jan 24, 2023

RussTreadon-NOAA left a comment

guillaumevernieres commented Jan 24, 2023

guillaumevernieres commented Jan 25, 2023

		source "${HOMEgfs}/ush/preamble.sh"


		export DATA="${DATAROOT}/${RUN}ocnanal_${cyc}"



		export DATA="${DATAROOT}/${RUN}ocnanal_${cyc}"
		source "${HOMEgfs}/ush/jjob_header.sh" -e "ocnanalrun" -c "base ocnanal ocnanalprep"

More marine DA j-jobs #1270

More marine DA j-jobs #1270

Conversation

guillaumevernieres commented Jan 23, 2023 • edited Loading

github-advanced-security bot left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Jan 23, 2023

aerorahul left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guillaumevernieres commented Jan 23, 2023

AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

guillaumevernieres commented Jan 24, 2023

guillaumevernieres commented Jan 25, 2023

guillaumevernieres commented Jan 23, 2023 •

edited

Loading