Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change to dask.order: be more eager at times #7929

Merged
merged 10 commits into from
Sep 1, 2021
Merged

Commits on Jul 23, 2021

  1. Small change to dask.order. Be more eager at times.

    `dask.order` will now more eagerly compute dependents if the parent is expected
    to be released soon.
    
    The simplified explanation is that if running a single dependent will allow the
    parent to be released from memory, then we should do it.  Although this is a
    wash--because we computed one thing and released another, so net zero--this can
    have big implications.  Computing more eagerly earlier can give more opportunities
    for releasing memory (such as computing one thing and releasing two).
    
    The more complicated (and correct) explanation is that we compute a dependent if
    doing so will let the parent be released *soon*.  Computing the depenend *may*
    allow the parent to be released immediately, but not necessarily so.  But, there
    is the expectation that the parent will be able to be released *soon*, because
    all of its other dependents are already in one of the inner stacks.
    
    Hence, this can introduce local sub-optimal behavior (more work and more memory)
    with the goal of being more globally optimal.  I think this is pretty cool!
    eriknw committed Jul 23, 2021
    Configuration menu
    Copy the full SHA
    c4e5eac View commit details
    Browse the repository at this point in the history

Commits on Jul 24, 2021

  1. Add test and descriptions. Clean up current approach.

    Give up trying to add this same trick to the "slow path".  Doing so is tricky.
    eriknw committed Jul 24, 2021
    Configuration menu
    Copy the full SHA
    c3aef63 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2021

  1. Configuration menu
    Copy the full SHA
    09a83cf View commit details
    Browse the repository at this point in the history

Commits on Aug 2, 2021

  1. Update dask.order to be smarter about computing single chains when …

    …available.
    
    This is much like the previous version, but better (and more complicated).
    Unlike before, this is now not over-eager.  It will only eagerly compute a node
    when it is known that doing so will free a node.
    eriknw committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    1f104ea View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    f0e7368 View commit details
    Browse the repository at this point in the history
  3. oops

    eriknw committed Aug 2, 2021
    Configuration menu
    Copy the full SHA
    1362687 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2021

  1. Configuration menu
    Copy the full SHA
    2ebde8d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    893f43f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bda2d2b View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2021

  1. Configuration menu
    Copy the full SHA
    1d8ef1a View commit details
    Browse the repository at this point in the history