Add more versatile options for masking datasets

[pnuu]

Feature Request

Currently we have MaskingCompositor that can be used to mask data with categorical data from NWC SAF PPS and GEO Cloud Type data. There is also CloudCompositor that does masking by creating the mask from the data it self (and adds some transitional smoothing for the transparency).

We should have a more flexible and generic way of masking data, e.g. using quality flags, thresholds and even combination of several things.

Describe the solution you'd like
A generic way for describing different masking operations in the composite YAML files. Either as a modifier, or as a compositor.

Describe any changes to existing user workflow
This feature would simplify the workflow for masking data in multiple use cases.

Additional context
There are two approaches from which to choose:

• a new compositor
• a new modifier

The compositor approach would mean "wrapping" all the desired composites within another compositor. The modifier should be much easier to apply.

But the thing I need input for (I'll add something below on a new comment):

1. use cases
2. ideas on how the configuration should look like for different cases

[pnuu]

Variable transparency using cloud type data

1. Apply different transparencies based on categorical data, e.g. a cloud type product.
2. List of category value/name + transparency values (https://github.com/pytroll/satpy/blob/master/satpy/etc/composites/seviri.yaml#L16-L44)

[peters77]

Some ideas....
Mask a dataset with another dataset (nitrogendioxide_tropospheric_column with [qa_value] , e.g. S5P Tropomi NO2 data)
masking is possible with:

scn[nitrogendioxide_tropospheric_column] = scn[nitrogendioxide_tropospheric_column].where(scn['qa_value‘]>0.75)

First pic colorized NO2 with quality flag applied, second pic the qa_dataset itself.

.yaml could look like:

  no2_tropospheric_clean:
    compositor: !!python/name:satpy.composites.ValueMaskCompositor
    prerequisites:
    - nitrogendioxide_tropospheric_column
    - qa_value
      modifiers: >0.75
    standard_name: no2_tropospheric_clean

Similar approach for cutting unwanted data, e.g. from h03B data.
This means to change a dateset by cutting data at the end or top.
For h03B and to get rid of not processed data I need a

new_scene[h03B] = new_scene[h03B].where(new_scene[h03B] > 0.0001388)
Some examples: raw, with dataset cut and colorized the cutted dataset:

this modified dataset I want to colorize and then use the BackgroundCompositor to add it on top of e.g. IR108.

.yaml could look like:

  colorized_IRR:
    compositor: !!python/name:satpy.composites.ValueMaskCompositor
    prerequisites:
    - h03B
    modifiers: >0.0001388
    standard_name: colorized_IRR

Maybe it could be the same compositor. If I provide a second dataset with a modifier, the second dataset is used to mask the first with optional conditions (<,> ==<...).
If there was only one dataset given with a modifier, the datastet itself is masked by the value given.
But maybe there is no universal approach possible.

As you pointed, maybe a modifier could be added at the Background compositor to give set an alpha value for the dataset which will place on top. Next step could be a variable alpha set by the data of the on top placed dataset.

Another idea: a common modifier which can do some easy math (...*3600) for the channel, like this:

attrs = scn['h03B'].attrs
scn['h03B''] = 3600*scn['h03B']
scn['h03B''].attrs = attrs
scn['h03B'].attrs['units'] = r'mm/hr' 

But maybe this is better applied inside the enhancements .yaml as there it would be more useful to get handy values for colorize...

[pnuu]

Thank you @peters77! The .where() case is a good one, but I think the YAML would look more like this:

modifiers:
  n02_mask:
    compositor: !!python/name:satpy.composites.MaskModifier
    conditions:
      # The conditions to apply, with the associated transparency
      - "> 0.75": 100.0
    prerequisites:
    - qa_value

composites:
  no2_tropospheric_clean:
    compositor: !!python/name:satpy.composites.ValueMaskCompositor
    prerequisites:
    - name: nitrogendioxide_tropospheric_column
      modifiers: [no2_mask]
    standard_name: no2_tropospheric_clean

The last (multiplication and unit assignment) isn't masking and would need another feature issue. It should be pretty simple to implement as an enhancement or modifier. Not sure which would be more .

[peters77]

Yes, that looks better. What will this "100" mean in

modifiers:
  n02_mask:
    compositor: !!python/name:satpy.composites.MaskModifier
    conditions:
    # The conditions to apply, with the associated transparency
     - "> 0.75": 100.0
    - prerequisites:
      qa_value

Do you intend to make a mask of the "qa_value" and the set the alpha to 100% to mask the bad data out!? So you somone is able to do even a blend of bad data!? Interesting!

Would your proposed idea even cover this:
new_scene[h03B] = new_scene[h03B].where(new_scene[h03B] > 0.0001388)

colorized_IRR:
    compositor: !!python/name:satpy.composites.ValueMaskCompositor
    prerequisites:
    - h03B
    modifiers: >0.0001388
    standard_name: colorized_IRR

as here I didn't mask with another dataset but just with a "cutted" version of the dataset itself?
Yes..the last on was just something what raised now 2 two times while colorize....maybe another modifier but does not fit into this masking addon..

[pnuu]

What will this "100" mean

It's the transparency used for the condition. This way we could use the same modifier structure in a more flexible way.

Would your proposed idea even cover this: new_scene[h03B] = new_scene[h03B].where(new_scene[h03B] > 0.0001388)

Yes, just define a modifier using the h03B dataset:

  h03b_mask:
    compositor: !!python/name:satpy.composites.MaskModifier
    conditions:
      # The conditions to apply, with the associated transparency
      - "> 0.0001388": 100.0
    prerequisites:
      - name: h03B

[peters77]

That sounds great! 👍 Hope you will have time to work on or implement it at PCW fully.