Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Change to
dask.order
: be more eager at times (#7929)
`dask.order` will now more eagerly compute dependents if the parent is expected to be released soon. The simplified explanation is that if running a single dependent will allow the parent to be released from memory, then we should do it. Although this is a wash--because we computed one thing and released another, so net zero--this can have big implications. Computing more eagerly earlier can give more opportunities for releasing memory (such as computing one thing and releasing two). The more complicated (and correct) explanation is that we compute a dependent if doing so will let the parent be released *soon*. Computing the depenent *may* allow the parent to be released immediately, but not necessarily so. But, there is the expectation that the parent will be able to be released *soon*, because all of its other dependents are already in one of the inner stacks. Hence, this can introduce local sub-optimal behavior (more work and more memory) with the goal of being more globally optimal. I think this is pretty cool!
- Loading branch information