Add new where reduction

[ianthomas23]

This partially implements issue #1126, adding a new where reduction that accepts either a max or min reduction. Best illustrated via an example:

import datashader as ds
import numpy as np
import pandas as pd

x = np.arange(2)
df = pd.DataFrame(dict(
    y_from = [0.0, 1.0, 0.0, 1.0, 0.0],
    y_to   = [0.0, 1.0, 1.0, 0.0, 0.5],
    value  = [1.1, 3.3, 5.5, 2.2, 4.4],
    other  = [-55, -77, -99, -66, -88],
))

canvas = ds.Canvas(plot_height=3, plot_width=5)
agg = canvas.line(
    source=df, x=x, y=["y_from", "y_to"], axis=1,
    agg=ds.where(ds.max("value"), "other"),
)

print(agg)

which outputs

<xarray.DataArray (y: 3, x: 5)>
array([[-99., -88., -55., -66., -66.],
       [ nan, -99., -99., -88., -88.],
       [-77., -77., -77., -99., -99.]])
Coordinates:
  * x        (x) float64 0.1 0.3 0.5 0.7 0.9
  * y        (y) float64 0.1667 0.5 0.8333

You can think of this using the max('value') reduction as normal, but then returning the corresponding values from the 'other' column rather that the value column.

What it currently supports:

• where takes either a min or max selector reduction.
• Works on CPU (not GPU), with or without dask.
• Works with antialiased lines.
• Cannot be used within a summary or categorical by reduction.

Note that there is no support for use of first and last within a where because there is no advantage in doing this, you can just use the first or last directly on their own.

Future improvements:

• Support within categorical reductions.
• Support for GPU.
• If lookup_column is not specified, use the index of the row in the supplied DataFrame.
• New max_n, min_n, first_n, last_n reductions.

All of these are possible but fiddly to implement, so I would rather have partial functionality available for users to experiment with and I can add these improvements over time.

Currently some combinations of lines and dask give different results depending on the number of dask partitions, but this has always been the situation and is no worse here.

[ianthomas23]

Needs a docstring with a good example.

[ianthomas23]

I have added a docstring. It is somewhat tortuous English when written in a generic form so I have added a concrete example that should help.

I am using the argument name selector because long-term I would like to divide the Reduction class hierarchy into two:

• Selection reductions that use values from a column without modifying them, e.g. first and max.
• Combination reductions that do some form of mathematical combination of the values from a column, e.g. mean, std, count.

Given that division, a where reduction will accept a Selection but not a Combination to select values from the lookup_column.

[jbednar]

Ok, sounds good. Agreed that it's tortuous, but the example does help a lot.

[jbednar]

Thanks! Can you clarify the current status of types? I.e. can you return an integer aggregate when testing on a float condition?

[ianthomas23]

where always returns a float64 with nans to represent no data, just as min, max, first, last etc reductions.

[jbednar]

Ok, I guess we'll need to deal with datatype issues when we support using the Pandas index as the "column" (actually just imputed values that act like a column, hence needing special support).

[ianthomas23]

Rebased on top of main to pick up the CI fixes.

[ianthomas23]

The reduction in coverage is mostly due to changes to the CUDA append functions and such CUDA code is not run in github actions.

[codecov]

Codecov Report

Merging #1155 (b34ffd6) into main (645ae07) will increase coverage by 0.03%.
The diff coverage is 83.68%.

@@            Coverage Diff             @@
##             main    #1155      +/-   ##
==========================================
+ Coverage   85.39%   85.43%   +0.03%     
==========================================
  Files          35       35              
  Lines        7819     7941     +122     
==========================================
+ Hits         6677     6784     +107     
- Misses       1142     1157      +15     

Impacted Files	Coverage Δ
datashader/core.py	88.05% <ø> (ø)
datashader/reductions.py	86.94% <80.83%> (-0.29%)	⬇️
datashader/compiler.py	95.62% <100.00%> (+0.53%)	⬆️
datashader/glyphs/line.py	92.95% <0.00%> (+0.09%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[ianthomas23]

Pinging @jbednar. I'd like to merge this and add the extra functionality (such as use of a virtual integer row index) as separate PRs.

[jbednar]

Those look like some painful changes, and it's great to have them behind us! Thanks, and it looks good to me!

[jbednar]

Can you make "use" more specific? Cannot accept, does not support, etc.? It's confusing to read about reductions using reductions.

[ianthomas23]

Done.

[jbednar]

Ok, sounds good. Agreed that it's tortuous, but the example does help a lot.