Ensure shuffle split operations are blacklisted from work stealing

[fjetter]

If shuffle split tasks are not blacklisted from work stealing, this can have
catastrophic effects on performance.
See also #4962

[mrocklin]

Hrm, I wonder if we should just blacklist any task that runs very quickly 🤔

😄

[jrbourbeau]

cc @madsbk

[fjetter]

Hrm, I wonder if we should just blacklist any task that runs very quickly 🤔

How quickly should it be? I've seen splits take longer than 5ms. I feel it is incredibly difficult to put a sane number on this but maybe you are right and something like LATENCY * 2 would do the trick? idk I can check. but this PR at least restores the behaviour we originally intended

[mrocklin]

How quickly should it be? I've seen splits take longer than 5ms. I feel it is incredibly difficult to put a sane number on this but maybe you are right and something like LATENCY * 2 would do the trick? idk I can check. but this PR at least restores the behaviour we originally intended

I was joking. You removed this behavior in the work stealing PR, right?

[mrocklin]

Does this fix #4962 ?

[mrocklin]

Should this be merged?

[mrocklin]

Does this fix #4962 ?

Ah, no. We also need to handle decide_worker

[fjetter]

I was joking. You removed this behavior in the work stealing PR, right?

yes but that's only a draft and I want to reiterate what's going on after the sizes are properly measured.

Does this fix #4962 ?

Not yet,

We'll need

• a version of Don't sample dict result of a shuffle group when calculating its size dask#7834
• maybe Improve work stealing for scaling situations #4920
• maybe/maybe not Consider candidates that don't hold any dependencies in decide_worker #4925

Should this be merged?

I believe it will improve performance and stability. I would like to (eventually) get the system into a state where we don't need this but I believe we're not there, yet.

[fjetter]

See also dask/dask#7844 for summary and cause of the regression

[fjetter]

All tests green? That's a sign! (I swear, I didn't cheat)