Skip to content

Commit

Permalink
Update docs.
Browse files Browse the repository at this point in the history
# Building wheels and fs_util will be skipped. Delete if not intended.
[ci skip-build-wheels]
  • Loading branch information
stuhood committed Sep 18, 2020
1 parent 3834ab3 commit 8bc29b5
Showing 1 changed file with 5 additions and 5 deletions.
10 changes: 5 additions & 5 deletions src/rust/engine/graph/README.md
Expand Up @@ -34,16 +34,16 @@ Confusingly, "root" in the context of `invalidate_from_roots` means a root in th

### Dirtying and Cleaning

When the `Entry` for a `Node` is marked dirty by "Invalidation" (see above) we only _suspect_ that it might need to re-run, because one of its dependencies has been cleared. To determine whether it _actually_ needs to re-run, the next time the dirty `Node` is requested, `Graph::get` (via [`Entry::get_node_result`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L361-L374)) will compare the [`Generation` values](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L37-L57) of the dependencies used to compute the previous value of the `Node` with their current `Generation` values. If none of the (`Generation` values) of the `Node` have changed, then it does not need to re-run: this is called "cleaning".
When the `Entry` for a `Node` is marked dirty by "Invalidation" (see above) we only _suspect_ that it might need to re-run, because one of its dependencies has been cleared. To determine whether it _actually_ needs to re-run, the next time the dirty `Node` is requested, `Graph::get` (via [`Entry::get_node_result`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L361-L374)) will compare the [`Generation` values](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L37-L57) of the dependencies used to compute the previous value of the `Node` with their current `Generation` values. If none of the (`Generation` values) of the `Node` have changed, then it does not need to re-run: this is called "cleaning" or "early cutoff" (in [Build Systems à la Carte](https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems.pdf)).

When a `Node` _does_ need to re-run for some reason (either due to having been cleared by invalidation, or being marked dirty and then failing to be cleaned), its previous and new result are compared to determine whether its `Generation` value should increment. The `Generation` value avoids the need to keep multiple copies of a `Node`'s result value in memory over time, and makes cleaning cheaper by converting potentially expensive equality checks into integer comparisons.

### Uncacheable Nodes

`Node`s can be marked "uncacheable" ([via](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/node.rs#L32-L35)), which causes them to re-run once per "session" (controlled by a [SessionId](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/node.rs#L97-L101) on the Context), regardless of whether any of their inputs have changed. In production usage, a session always corresponds to a single run of Pants (or each iteration of a `--loop`), regardless of whether `pantsd` is in use.
`Node`s can be marked "uncacheable" ([via](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/node.rs#L32-L35)), which causes them to re-run once per "session" (controlled by a [RunId](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/node.rs#L97-L101) on the Context), regardless of whether any of their inputs have changed. In production usage, a session always corresponds to a single run of Pants (or each iteration of a `--loop`), regardless of whether `pantsd` is in use.

A `Node` that is uncacheable is given an [`Uncacheable` `EntryState`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L70-L72), which records which `SessionId` it is valid for. When the value of an uncacheable `Node` is requested, the current `SessionId` is compared to the `SessionId` on the `Node`'s `EntryState`, and the `Node` re-runs if it mismatches.
A `Node` that is uncacheable is given an [`Uncacheable` `EntryState`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L70-L72), which records which `RunId` it is valid for. When the value of an uncacheable `Node` is requested, the current `RunId` is compared to the `RunId` on the `Node`'s `EntryState`, and the `Node` re-runs if it mismatches.

Uncacheable `Node`s may have dependees, but in order to ensure that the uncacheable `Node` is always detected below them in the `Graph`, any cacheable `Node`s that transitively depend on uncacheable `Node`s complete in a different EntryState: `[UncacheableDependencies`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L73-L77). `UncacheableDependencies` acts just like the `Dirty` state: the value it holds cannot be directly consumed until it is cleaned (or until it re-runs because its dependencies have changed).
Uncacheable `Node`s may have dependees, but in order to ensure that the uncacheable `Node` is always detected below them in the `Graph`, any cacheable `Node`s that transitively depend on uncacheable `Node`s complete in a different EntryState: `[UncacheableDependencies`](https://github.com/pantsbuild/pants/blob/01372719f7e6f2a3aa0b9f3ce6909991388071ca/src/rust/engine/graph/src/entry.rs#L73-L77). An `UncacheableDependencies` node for a mismatched `RunId` acts just like the `Dirty` state: the value it holds cannot be directly consumed until it is cleaned (or until it re-runs because its dependencies have changed). An `UncacheableDependencies` node is cleaned by updating its `RunId`.

As an example: if there is a cacheable `Node` `A` that depends on an uncacheable `Node` `B`, and both of them ran in some session, then they will be in the states `UncacheableDependencies` and `Uncacheable`, respectively. If a caller requests `A` in a new session, the `Graph` will observe the `UncacheableDependencies` state and attempt to clean `A` by checking that its dependencies have not changed: that check will recursively inspect `B` to see whether it has changed, and will unconditionally re-run `B` because it is in the `Uncacheable` state. Just like in "Dirtying and Cleaning", if `B` returns the same value as it did in the first session, then its `Generation` will not change: `A` will observe that the `Generation` values of its dependencies have not changed, and will be cleaned with its previous value without re-running. But unlike a `Node` in the `Dirty` state, `A` will stay in the `UncacheableDependencies` state. Note that there is one potential performance issue with this strategy: see https://github.com/pantsbuild/pants/issues/10798 for more information.
As an example: if there is a cacheable `Node` `A` that depends on an uncacheable `Node` `B`, and both of them ran in some session, then they will be in the states `UncacheableDependencies` and `Uncacheable`, respectively. If a caller requests `A` in a new session, the `Graph` will observe the `UncacheableDependencies` state has a mismatched `SessonId`, and will attempt to clean `A` by checking that its dependencies have not changed. That check will recursively inspect `B` to see whether it has changed, and will unconditionally re-run `B` because it is in the `Uncacheable` state for a mismatched `RunId`. Just like in "Dirtying and Cleaning", if `B` returns the same value as it did in the first session, then its `Generation` will not change: `A` will observe that the `Generation` values of its dependencies have not changed, and will be cleaned with its previous value without re-running. But unlike a `Node` in the `Dirty` state, `A` will stay in the `UncacheableDependencies` state: only its `RunId` changes.

0 comments on commit 8bc29b5

Please sign in to comment.