Adapt the marine DA to the new COM structure #457

guillaumevernieres · 2023-05-01T19:10:35Z

Does what the title says mainly, but also:

Fixes Adapt the marine DA to the new COM structure #453
Fixes Observation staging location #204
Fixes Add icepack to .gitignore #449
Fixes test_gdasapp_soca_ana_prep is broken #448

@RussTreadon-NOAA , @CoryMartin-NOAA , I took the liberty of removing your R2D2 dependencies in stage.py assuming it wasn't used anymore. This was done mainly because of a name change in pygw (parse_j2yaml to parse_yaml) ... I deleted the functions instead, happy to revert if you still need these around.

.gitignore

parm/soca/obs/config/adt_3a_egm2008.yaml

parm/soca/obs/config/oc_aqua.yaml

test/soca/gw/prep_com.sh

ush/soca/run_jjobs.py

emcbot · 2023-05-01T20:25:03Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon May  1 19:19:38 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  1 20:18:13 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon May  1 20:25:02 UTC 2023
Tests: 66% tests passed, 17 tests failed out of 50
	1442 - test_gdasapp_soca_prepcom (Failed)
	1445 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1446 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1447 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1448 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1458 - test_gdasapp_run_jedi_exe_3dhofx (Failed)
	1459 - test_gdasapp_run_jedi_exe_3dvar (Failed)
	1460 - test_gdasapp_run_jedi_exe_3denvar (Failed)
	1461 - test_gdasapp_run_jedi_exe_letkf (Failed)
	1463 - test_gdasapp_atm_jjob_var_init (Failed)
	1464 - test_gdasapp_atm_jjob_var_run (Failed)
	1465 - test_gdasapp_atm_jjob_var_final (Failed)
	1466 - test_gdasapp_atm_jjob_ens_init (Failed)
	1467 - test_gdasapp_atm_jjob_ens_run (Failed)
	1468 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/457/global-workflow/sorc/gdas.cd/build/log.ctest

emcbot · 2023-05-01T20:29:27Z

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Mon May  1 14:17:16 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  1 15:21:19 CDT 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon May  1 15:29:26 CDT 2023
Tests: 66% tests passed, 17 tests failed out of 50
	1442 - test_gdasapp_soca_prepcom (Failed)
	1445 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP (Failed)
	1446 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
	1447 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
	1448 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
	1449 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
	1450 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
	1458 - test_gdasapp_run_jedi_exe_3dhofx (Failed)
	1459 - test_gdasapp_run_jedi_exe_3dvar (Failed)
	1460 - test_gdasapp_run_jedi_exe_3denvar (Failed)
	1461 - test_gdasapp_run_jedi_exe_letkf (Failed)
	1463 - test_gdasapp_atm_jjob_var_init (Failed)
	1464 - test_gdasapp_atm_jjob_var_run (Failed)
	1465 - test_gdasapp_atm_jjob_var_final (Failed)
	1466 - test_gdasapp_atm_jjob_ens_init (Failed)
	1467 - test_gdasapp_atm_jjob_ens_run (Failed)
	1468 - test_gdasapp_atm_jjob_ens_final (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/457/global-workflow/sorc/gdas.cd/build/log.ctest

RussTreadon-NOAA · 2023-05-03T15:49:17Z

@guillaumevernieres , I executed test_gdasapp_soca from an installation of feature/newcom at bce9561 with the following results

33% tests passed, 8 tests failed out of 12

Total Test time (real) = 111.95 sec

The following tests FAILED:
        1458 - test_gdasapp_soca_prepcom (Failed)
        1461 - ` (Failed)
        1462 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT (Failed)
        1463 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_BMAT_VRFY (Failed)
        1464 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_RUN (Failed)
        1465 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_CHKPT (Failed)
        1466 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_POST (Failed)
        1467 - test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_VRFY (Failed)

Rerunning test_gdasapp_soca_prepcom with -VV shows the following error

1458: + cp /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/sorc/gdas.cd/build/test/soca/gw/COM/gdas.20180415/06/model_data/ocean/history/gdas.t06z.ocnf003.nc /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/sorc/gdas.cd/build/soca_static/bkgerr/stddev/ocn.ensstddev.fc.2019-04-01T00:00:00Z.PT0S.nc
1458: cp: cannot create regular file '/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/sorc/gdas.cd/build/soca_static/bkgerr/stddev/ocn.ensstddev.fc.2019-04-01T00:00:00Z.PT0S.nc': No such file or directory
1/1 Test #1458: test_gdasapp_soca_prepcom ........***Failed    0.27 sec

A -VV run of test_gdasapp_soca_JGDAS_GLOBAL_OCEAN_ANALYSIS_PREP contains the warning

/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/work/sorc/gdas.cd/../../workflow/setup_expt.py:67: UserWarning: User did not provide '\
--icsdir' to stage initial conditions
  warnings.warn("User did not provide '--icsdir' to stage initial conditions")

Should I stand down on this and wait for subsequent updates or development or keep digging?

guillaumevernieres · 2023-05-03T17:32:27Z

@RussTreadon-NOAA , sorry, I should have posted I am still fixing stuff. It's working on containers, building now on hpc to test.

RussTreadon-NOAA · 2023-05-03T17:45:08Z

@RussTreadon-NOAA , sorry, I should have posted I am still fixing stuff. It's working on containers, building now on hpc to test.

Got it. I'll hit pause.

CoryMartin-NOAA · 2023-05-08T17:23:19Z

@guillaumevernieres I do think we use R2D2 now still in the run_jedi_exe tests since #461 showed errors after some clown (me) deleted R2D2 files on Hera, so that may solve some of these issues?

guillaumevernieres · 2023-05-08T17:51:31Z

@guillaumevernieres I do think we use R2D2 now still in the run_jedi_exe tests since #461 showed errors after some clown (me) deleted R2D2 files on Hera, so that may solve some of these issues?

Ha! Yes, maybe for the atm tests @CoryMartin-NOAA . My grepping didn't return anything so I removed a few of the atmos stuff that were pointing to a deprecated function. I'll put them back after the hera ci finishes, if need be. I've been a bit trigger happy with the delete key and ended up destroying the soca static dir ... Oops.

emcbot · 2023-05-08T17:55:27Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon May  8 17:15:50 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  8 17:42:50 UTC 2023
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Mon May  8 17:55:27 UTC 2023
Tests: 92% tests passed, 4 tests failed out of 50
	1479 - test_gdasapp_run_jedi_exe_3dhofx (Failed)
	1480 - test_gdasapp_run_jedi_exe_3dvar (Failed)
	1481 - test_gdasapp_run_jedi_exe_3denvar (Failed)
	1482 - test_gdasapp_run_jedi_exe_letkf (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/457/global-workflow/sorc/gdas.cd/build/log.ctest

guillaumevernieres · 2023-05-08T18:05:23Z

What a nightmare!!!
@CoryMartin-NOAA , @RussTreadon-NOAA : It looks like some of the atmos test are having the same conda issue now:

39/50 Test #1479: test_gdasapp_run_jedi_exe_3dhofx ..........................***Failed    5.04 sec
+ bindir=/scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/457/global-workflow/sorc/gdas.cd/build
+ srcdir=/scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/457/global-workflow/sorc/gdas.cd
+ [[ -d /scratch1 ]]
+ machine=hera
+ set +x

CommandNotFoundError: Your shell has not been properly configured to use 'conda deactivate'.
To initialize your shell, run

    $ conda init <SHELL_NAME>

Currently supported shells are:
  - bash
  - fish
  - tcsh
  - xonsh
  - zsh
  - powershell

See 'conda init --help' for more information and options.

IMPORTANT: You may need to close and restart your shell after running 'conda init'.

RussTreadon-NOAA · 2023-05-08T18:21:04Z

Manual execution of test_gdasapp_run_jedi on Hera works for me

(gdasapp) Hera(hfe05):/scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/build$ ctest -R test_gdasapp_run_jedi
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/global-workflow/test3/sorc/gdas.cd/build
    Start 1462: test_gdasapp_run_jedi_exe_3dhofx
1/4 Test #1462: test_gdasapp_run_jedi_exe_3dhofx ....   Passed   33.74 sec
    Start 1463: test_gdasapp_run_jedi_exe_3dvar
2/4 Test #1463: test_gdasapp_run_jedi_exe_3dvar .....   Passed   13.93 sec
    Start 1464: test_gdasapp_run_jedi_exe_3denvar
3/4 Test #1464: test_gdasapp_run_jedi_exe_3denvar ...   Passed   13.67 sec
    Start 1465: test_gdasapp_run_jedi_exe_letkf
4/4 Test #1465: test_gdasapp_run_jedi_exe_letkf .....   Passed   11.44 sec

100% tests passed, 0 tests failed out of 4

Total Test time (real) =  73.42 sec

emcbot · 2023-05-08T20:57:15Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon May  8 20:43:03 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  8 20:44:09 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon May  8 20:57:14 UTC 2023
Tests: 100% tests passed, 0 tests failed out of 50

guillaumevernieres · 2023-05-08T21:07:22Z

Are you OK with the changes @RussTreadon-NOAA ?
The orion ci will fail (if it ever runs) since it was submitted before the fix. Not sure I want to wait 1/2 day + for that thing to run.

emcbot · 2023-05-08T21:42:52Z

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Mon May  8 20:43:03 UTC 2023 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon May  8 20:44:09 UTC 2023
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Mon May  8 20:57:14 UTC 2023
Tests: 100% tests passed, 0 tests failed out of 50

Build: SUCCESS
Build: Completed at Mon May 8 21:30:27 UTC 2023

Tests: SUCCESS
Tests: Completed at Mon May 8 21:42:51 UTC 2023
Tests: 100% tests passed, 0 tests failed out of 50

RussTreadon-NOAA · 2023-05-08T23:33:25Z

Green for hera-GW-RT works for me.

RussTreadon-NOAA

All 50 Hera GW-RT tests pass. Approve.

guillaumevernieres and others added 4 commits April 28, 2023 20:06

works but needs tidy

cd1b839

adopted -new com for ctests

562b68a

Merge branch 'develop' into feature/newcom

5a07b91

code tidy

ec1da2b

guillaumevernieres requested review from RussTreadon-NOAA, CoryMartin-NOAA and AndrewEichmann-NOAA May 1, 2023 19:11

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels May 1, 2023

emcbot added orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed orion-GW-RT Queue for automated testing with global-workflow on Orion hera-GW-RT Queue for automated testing with global-workflow on Hera labels May 1, 2023

CoryMartin-NOAA reviewed May 1, 2023

View reviewed changes

emcbot added the hera-GW-RT-Failed Automated testing with global-workflow failed on Hera label May 1, 2023

emcbot added orion-GW-RT-Failed Automated testing with global-workflow failed on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels May 1, 2023

guillaumevernieres mentioned this pull request May 2, 2023

update atm var and ens ctests to new GFS COM directory structure (#455) #456

Merged

working state, reverted yamls, started switching to FileHandler

bce9561

guillaumevernieres force-pushed the feature/newcom branch from dc4fc88 to bce9561 Compare May 3, 2023 00:02

use filehandler for post

22a84c0

emcbot removed the hera-GW-RT Queue for automated testing with global-workflow on Hera label May 3, 2023

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels May 8, 2023

first try # 127

115506e

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels May 8, 2023

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels May 8, 2023

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels May 8, 2023

guillaumevernieres added 3 commits May 8, 2023 14:37

Merge branch 'develop' into feature/newcom

a18e26b

reverted stage.atmos stuff

33baf1c

no module load

4c7745f

guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels May 8, 2023

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels May 8, 2023

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels May 8, 2023

RussTreadon-NOAA approved these changes May 8, 2023

View reviewed changes

guillaumevernieres merged commit 8c2706e into develop May 9, 2023

guillaumevernieres deleted the feature/newcom branch May 22, 2023 12:59

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adapt the marine DA to the new COM structure #457

Adapt the marine DA to the new COM structure #457

guillaumevernieres commented May 1, 2023

emcbot commented May 1, 2023

emcbot commented May 1, 2023

RussTreadon-NOAA commented May 3, 2023

guillaumevernieres commented May 3, 2023

RussTreadon-NOAA commented May 3, 2023

CoryMartin-NOAA commented May 8, 2023

guillaumevernieres commented May 8, 2023

emcbot commented May 8, 2023

guillaumevernieres commented May 8, 2023

RussTreadon-NOAA commented May 8, 2023

emcbot commented May 8, 2023

guillaumevernieres commented May 8, 2023

emcbot commented May 8, 2023

RussTreadon-NOAA commented May 8, 2023

RussTreadon-NOAA left a comment

Adapt the marine DA to the new COM structure #457

Adapt the marine DA to the new COM structure #457

Conversation

guillaumevernieres commented May 1, 2023

emcbot commented May 1, 2023

emcbot commented May 1, 2023

RussTreadon-NOAA commented May 3, 2023

guillaumevernieres commented May 3, 2023

RussTreadon-NOAA commented May 3, 2023

CoryMartin-NOAA commented May 8, 2023

guillaumevernieres commented May 8, 2023

emcbot commented May 8, 2023

guillaumevernieres commented May 8, 2023

RussTreadon-NOAA commented May 8, 2023

emcbot commented May 8, 2023

guillaumevernieres commented May 8, 2023

emcbot commented May 8, 2023

Build: SUCCESS Build: Completed at Mon May 8 21:30:27 UTC 2023

RussTreadon-NOAA commented May 8, 2023

RussTreadon-NOAA left a comment

Choose a reason for hiding this comment

Build: SUCCESS
Build: Completed at Mon May 8 21:30:27 UTC 2023