Skip to content

Commit

Permalink
Change to dask.order: be more eager at times (#7929)
Browse files Browse the repository at this point in the history
`dask.order` will now more eagerly compute dependents if the parent is expected
to be released soon.

The simplified explanation is that if running a single dependent will allow the
parent to be released from memory, then we should do it.  Although this is a
wash--because we computed one thing and released another, so net zero--this can
have big implications.  Computing more eagerly earlier can give more opportunities
for releasing memory (such as computing one thing and releasing two).

The more complicated (and correct) explanation is that we compute a dependent if
doing so will let the parent be released *soon*.  Computing the depenent *may*
allow the parent to be released immediately, but not necessarily so.  But, there
is the expectation that the parent will be able to be released *soon*, because
all of its other dependents are already in one of the inner stacks.

Hence, this can introduce local sub-optimal behavior (more work and more memory)
with the goal of being more globally optimal.  I think this is pretty cool!
  • Loading branch information
eriknw committed Sep 1, 2021
1 parent efac55a commit 9983c19
Show file tree
Hide file tree
Showing 2 changed files with 553 additions and 157 deletions.

0 comments on commit 9983c19

Please sign in to comment.