add ensemble based bump localization to gsibec hybvar yaml

[RussTreadon-NOAA]

This PR is opened to add bump generated localization to hybvar_gsibec.yaml. A user provided ensemble is processed by bump to generate localization values.

Please note the localization added to the ensemble section of hybvar_gsibec.yaml is for demonstration purposes only. Subsequent testing and tuning is required for scientific validation.

Fixes #460

[RussTreadon-NOAA]

Install RussTreadon-NOAA:feature/hybvar_gsibec on Hera and Orion within g-w develop. Manually execute ctest -R test_gdasapp with the following results:

Hera:
40 out of 50 tests Passed. The Failed tests are

        1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1451 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1452 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1453 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1456 - test_gdasapp_land_imsproc (Failed)
        1462 - test_gdasapp_run_jedi_exe_3dhofx (Failed)
        1463 - test_gdasapp_run_jedi_exe_3dvar (Failed)
        1464 - test_gdasapp_run_jedi_exe_3denvar (Failed)

The test_gdasapp_run_jedi_exe_3d* tests fail because directory /scratch1/NCEPDEV/stmp4/Cory.R.Martin/R2D2_SHARED/gfs/fc/oper_gdas/c96/2021-12-21/ no longer exists. @CoryMartin-NOAA , has this directory moved to a new location?

test_gdasapp_land_imsproc fails with

1456: Test command: /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/test/land/test_imsproc.sh "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/build" "/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd"
1456: Environment variables:
1456:  GDASAPP_TESTDATA=/scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/data
1456:  Python3_VERSION_MAJOR=3
1456:  Python3_VERSION_MINOR=7
1456: Test timeout computed to be: 1500
1456: ln: failed to create symbolic link './20210323.180000.sfc_data.tile1.nc': File exists

I previously ran the ctests in my working copy. test_imsproc.sh does not remove or empty WORKDIR prior to populating the directory. I manually removed WORKDIR and reran test_gdasapp_land_imsproc. The rerun passed

(gdasapp) Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/build$ ctest -R test_gdasapp_land_imsproc
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/build
    Start 1456: test_gdasapp_land_imsproc
1/1 Test #1456: test_gdasapp_land_imsproc ........   Passed    8.09 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   8.43 sec

@jiaruidong2017 , I recommend that you modify test_imsproc.sh so that it creates a clean WORKDIR each time the script runs.

Orion:
43 out of 50 tests Passed. The Failed tests are

        1445 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
        1446 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1447 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1448 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1452 - test_gdasapp_land_imsproc (Failed)

PR #457 addresses the soca failures on both Hera and Orion. test_gdasapp_land_imsproc failed on Orion for the same reason it failed on Hera. Manually removing WORKDIR and rerunning yielded a Passed on Orion

[CoryMartin-NOAA]

@RussTreadon-NOAA whoops, no I'm just dumb and deleted that directory in R2D2 in an attempt to save space forgetting we still used R2D2 files in these tests... let me restore it from Orion today.

[RussTreadon-NOAA]

@RussTreadon-NOAA whoops, no I'm just dumb and deleted that directory in R2D2 in an attempt to save space forgetting we still used R2D2 files in these tests... let me restore it from Orion today.

Got it. Thanks, @CoryMartin-NOAA . I'll rerun when the Hera files are restored.

[jiaruidong2017]

@jiaruidong2017 , I recommend that you modify test_imsproc.sh so that it creates a clean WORKDIR each time the script runs.

Thanks @RussTreadon-NOAA This has been addressed on the PR #463

[RussTreadon-NOAA]

ctests rerun on Hera with the change from PR #463. Orion appears to be having fileset issues this morning. This complicates efforts to refresh Hera c96 with files from Orion. I'll check back later today to see if I can rerun `test_gdasapp_run_jedi_exe_3d* on Hera.

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon May  8 13:03:04 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  8 14:18:14 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon May  8 14:35:35 UTC 2023
Tests: 82% tests passed, 9 tests failed out of 50
	1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1455 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1456 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1457 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1458 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1459 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1467 - test_gdasapp_run_jedi_exe_3dhofx (Failed)
	1468 - test_gdasapp_run_jedi_exe_3dvar (Failed)
	1469 - test_gdasapp_run_jedi_exe_3denvar (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/461/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Mon May  8 08:02:13 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  8 09:55:56 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon May  8 10:09:16 CDT 2023
Tests: 88% tests passed, 6 tests failed out of 50
	1454 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1455 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1456 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1457 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1458 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1459 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/461/global-workflow/sorc/gdas.cd/build/log.ctest

[CoryMartin-NOAA]

@RussTreadon-NOAA files should be copied over to Hera now (at least Globus said it has succeeded)

[RussTreadon-NOAA]

@RussTreadon-NOAA files should be copied over to Hera now (at least Globus said it has succeeded)

The copy worked! The test_gdasapp_run_jedi_exe_3d* tests pass on Hera.

[guillaumevernieres]

Looks good to me @RussTreadon-NOAA . Soca is still on ignore mode, so do feel free to merge.

[RussTreadon-NOAA]

Given approval from @guillaumevernieres and @CoryMartin-NOAA , merge this PR into develop.