Skip to content

Commit

Permalink
Switch to a per-entry state machine in Graph (#6013)
Browse files Browse the repository at this point in the history
### Problem

#4558 introduces a new state for a `Node`: it may be `dirty`, which requires that we remember the previous value of a `Node` while simultaneously computing its next value. I began implementing this state machine with our existing `future::Shared` usage, but ran into the need to either hide values in the closure of the running future, or store values atomically outside of the future's value. This complexity represents a classic case for a state machine.

Additionally, we have seen that `future::Shared` has a relatively high cost in terms of complexity and memory (see #5990).

### Solution

In order to make #4558 easier to implement correctly (and to reduce memory usage), implement the storage of a `Node`'s value in the `Graph` via a state machine.

One of the largest complexities of `Shared` is that it needs to elect one of the waiters of the `Shared` future as the one who will actually do the work of `poll`ing the wrapped future. To avoid that complexity, we introduce usage of the `tokio` `Runtime` for _all_ `Node` executions. This allows waiters to stay very simple, and receive the result value via a `oneshot::channel`.

### Result

#4558 should be much easier to implement. I'll likely wait to land this until #4558 is further along. 

This change reduces memory usage about 200MB further than the fix from #5990 (down to about 2.5GB for `./pants list ::` in Twitter's repo).  Unfortunately, in its current form it also represents a performance regression of about 10% for that same usecase. Although I believe I have a handle on why, I'd like to defer fixing that regression until after #4558 is implemented.

This also fixes #3695, as since we now fail slowly, we are able to render multiple distinct failures at once.
  • Loading branch information
Stu Hood committed Jul 3, 2018
1 parent cafd732 commit ff4162d
Show file tree
Hide file tree
Showing 6 changed files with 403 additions and 233 deletions.

0 comments on commit ff4162d

Please sign in to comment.