SimpleShuffleLayer should compare parts_out with set(self.parts_out)

[GenevieveBuckley]

I noticed that the SimpleShuffleLayer cull method should probably be comparing:

        if parts_out != set(self.parts_out):

instead of the current

        if parts_out != self.parts_out:

The reason for this is that we often find self.parts_out is something like range(0, 30), and python won't recognize if a set and range are equal or not, unless we turn the range into a set too.

The BroadcastJoinLayer does do this comparison with a set, I think this is the right way:

dask/dask/layers.py

Line 1017 in 6599905

if parts_out != set(self.parts_out):

The consequences aren't super severe, I think it's just taking up extra time running an optimization that doesn't have any real effect (so we may as well save time and skip it).

Here's an example you can use where self.parts_out is a range and not a set. I put a breakpoint in before the line I've edited and looked at the values from there.

from dask.datasets import timeseries

ddf = timeseries().shuffle("id", shuffle="tasks")
ddf.compute()
 

• Tests added / passed
• Passes black dask / flake8 dask / isort dask

[GenevieveBuckley]

@rjzamora I think you're the best person to tell me whether I have the right idea or not

[rjzamora]

Good catch @GenevieveBuckley ! It looks like you are correct.

I'm curious what is causing all the CI failures here :/

[jrbourbeau]

Just merged main to resolve some unrelated CI failures

[GenevieveBuckley]

@dask/maintenance I'd say this is ready. It's been approved by Rick, and James merging main has fixed the CI.