Intermediate tasks computed before needed

[jakirkham]

Borrowed from #3588 (comment)

Basically Dask appears to load all of the intermediate chunks in memory well before they are used. In some workloads this can exhaust the available memory and crash the workers. Would be good to adjust the task priority to take this into account.

[mrocklin]

Is this specific example still failing?

It would be good for issues like this to include more specific examples that might be directly solvable. This general topic might be too broad. There are many reasons for behavior like this to occur. It's unlikely that any particular solution will "solve" this issue generally.

[mrocklin]

Ah, I see from your other comment that something is still happening. The first step to solving this issue is to provide a minimal example. This is usually painful to create, but I don't expect much progress to be made without it.