Add task to calculate ensemble mean and variance from background state

[metdyn]

Description

A step towards LETKF is to calculate ensemble mean and variance:

• Adding ensemble mean #365

Alternatively, this functionality may be used as a simple, independent tool.

Tested on SCU17:

swell create localensembleda  -s slurmfile.yaml -p nccs_discover_sles15

slurm_directives_tasks:
  RunJediLocalEnsembleDaExecutable:
    geos_atmosphere:
      nodes: 1
      ntasks-per-node: 36
  GetEnsembleMeanVariance:
    all:
      nodes: 1
      ntasks-per-node: 36

Dependencies

None

Impact

It is part of the localensembleda suite.

[rtodling]

This seems to work - it runs fine, but I am confused as to what's preventing the localensembleda to go beyond simply the ensemean background ... after all, this is behaving as if it were the ensemblemean suite but it's not!

[Dooruk]

@rtodling and @metdyn this is missing the last commit from the develop branch.

Please see following changes. We are abandoning the static StageJedi task to be able to use relative paths and instead running StageJediCycle at each cycle, which puts static files in the {cycle_dir}:

feature/submit/ens_mean...develop

@metdyn, please notice the changes in flow.cylc for 3dvar_atmos and 3dfgat_atmos, they need to be replicated for the localensembleda suite.

[metdyn]

@rtodling and @metdyn this is missing the last commit from the develop branch.

Please see following changes. We are abandoning the static StageJedi task to be able to use relative paths and instead running StageJediCycle at each cycle, which puts static files in the {cycle_dir}:

feature/submit/ens_mean...develop

@metdyn, please notice the changes in flow.cylc for 3dvar_atmos and 3dfgat_atmos, they need to be replicated for the localensembleda suite.

Thanks, @Dooruk, I will take a look and make appropriate changes to the PR to be update with other suites and relative path.

[metdyn]

Hi @Dooruk @asewnath,

I know the ctest swell test code_tests fails at
https://github.com/GEOS-ESM/swell/blob/develop/src/swell/utilities/scripts/task_question_dicts.py#L134-L135

Error message is

FAIL: test_dictionary_comparison (swell.test.code_tests.question_dictionary_comparison_test.QuestionDictionaryTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/gpfsm/dnb07/projects/p234/yyu11/br_jedi_scu17/install_swell/src/swell/swell_ens_mean/src/swell/test/code_tests/question_dictionary_comparison_test.py", line 26, in test_dictionary_comparison
    assert tq_dicts_rc == 0
AssertionError

So I printed out the string of the dict into files:

yaml.dump(question_dict_str_in)
yaml.dump(dict_to_write_str)

diff diff in.txt out.txt

Then I print out the first two lines of differences of the two files:

7,25c7,24
<   \ are the analysis variables?\n  tasks:\n  - GenerateBClimatology\n  - PrepareAnalysis\n\
<   \  - RunJediConvertStateSoca2ciceExecutable\n  - RunJediVariationalExecutable\n\
  \  - GetEnsembleMeanVariance\n  type: string-check-list\n\nbackground_error_model:\n\

---
>   \ are the analysis variables?\n  tasks:\n  - GenerateBClimatology\n  - GetEnsembleMeanVariance\n\
>   \  - PrepareAnalysis\n  - RunJediConvertStateSoca2ciceExecutable\n  - RunJediVariationalExecutable\n\
  \  type: string-check-list\n\nbackground_error_model:\n  ask_question: false\n \

The order between GetEnsembleMeanVariance and PrepareAnalysis is swap.
So the dictionary comparison using the print out string becomes a problem when new tasks is added into task dir,
which changed glob.glob file list.
There should be a better way for the task questions yaml comparison.

Meanwhile I tried to print out the difference between the two dictionaries:

set1 = set(dict1.items())
set2 = set(dict2.items())
set1 ^ set2
(https://stackoverflow.com/questions/32815640/how-to-get-the-difference-between-two-dictionaries-in-python)
There comes an error that the dictionary is not hashable. I did not go further.

[metdyn]

I think this error means I need to regenerate tasks/task_questions.yaml, because new tasks are added in.
In this logic, then the test_dictionary_comparison function would be a good place as a generator. Or I would manually check tasks/task_questions.yaml. How was this done before?

[Dooruk]

@metdyn, let me look into this branch and see what is going on. I will suggest an update to this branch afterwards as a merge. It's a simple unit test issue but I also notice some potential Stage task problems.

If you change this following line in the source code (setting $LOG_INFO to 1 in the environment didn't work for me) and build it again, tests will provide more details to show you a temporary YAML file with the proper task_questions.yaml.

swell/src/swell/test/code_tests/code_tests.py

Lines 28 to 29 in 465990f

	# Turn off the regular info testing
	os.environ["LOG_INFO"] = "0" # Set this to 1 when errors are being debugged

Sorry, this is all very obscure without documentation but this is also how I learned these things..

[metdyn]

@Dooruk thanks and that is the key! I was about to ask you why the logger.info did not work in this test code file, but logger.test() worked in another file.

[Dooruk]

@rtodling, can you confirm RunJediEnsembleMeanVariance works on your end so we can merge this?

[Dooruk]

I added a few comments. There is some duplicate functionality because RunJediEnsembleMean already handles the options where you put inside RunJediLocalDaExecutable. Simply taking them out should suffice.

[metdyn]

I created another PR to help with enHX with packets.
#405

[Dooruk]

This is good to go on my end, let's wait on @rtodling .

[rtodling]

This works for me - it fails in the HofX, but I understand this is another issue to be tackled elsewhere.