YAN-858 Frequency Scattering of Visibilities

[calgray]

As part of sdp work there is a need to split measurement set data on by frequency for daliuge parallel processing. From previous MS extraction I've used .npy serialized format and created a scatter app that can split multiple input ndarrays on the frequency axes specified by list param "Scatter Axes".

In implementing the scatter app I've noticed input app parameters where not being sent to the engine so I've explicitly populated input app fields for the logical graph such that the scatter app can be processed to the physical graph like a regular in python app. Additionally I send fields from the parent scatter node to the input app so the scatter number is always correct.

[coveralls]

Coverage increased (+0.2%) to 79.429% when pulling edcbbca on YAN-858 into cc15b7d on master.

[calgray]

2 seperate generic scatter apps is fine temporarily, however with the update behavior and parameter changes it should be possible to combine all the features of generic scatter app by adding a 'use_pickle' flag to the params list. Adding these to GenericScatterApp will break existing scatter graphs so an upgrade plan would be needed. I'd suggest either:

• manually update all repo graphs
• a .graph update tool
• support deprecated variables and raise warning

I'd lean towards deprecated warnings that print how to upgrade (namely scatter_axis -> scatter_axes, add use_pickle) and some effort towards manually updating EAGLE-graph-repo.

[awicenec]

The main part of the changes are fine, but we have to address the changes in dm_utils.py in a different way. The reason for the splitting of the fields into the additional appFields was to enable paasing those parameters on to the actual application. The other fields are passed to the drop. This is required in particular for dockerized applications which require command line parameters. Thus I think we need to make sure that the translator is passing on those fields, but still enables the engine to distinguish them. I guess we could add an additional flag to the fields, or we could keep them entierly separate like they are coming from EAGLE. Since this is not really related to this current pull request I will still accept the changes.

[calgray]

This line I found is needed since app parameters are meant to exist at under "fields" in the logical graph. These fields are from the app view in the inputApplication dropdown. The other fields I think some careful consideration needs to be made about. A logical graph schema may help a lot.