Skip to content

Commit

Permalink
Move file owners computation into the engine and make lighter (#6790)
Browse files Browse the repository at this point in the history
### Problem

#5579 added an optimization to avoid hydrating targets, of which a significant portion was later rolled back by #5636. #5636 reasoned that it was necessary to fully construct the graph in order to detect when targets have been deleted.

The solution used in #5636 though, was heavy-handed. By requesting `TransitiveHydratedTargets`, we fully constructed the graph of dependencies and then later _also_ constructed the dependent graph (in `DependentGraph`). Moreover, fully hydrating the targets meant unnecessarily fully expanding their source globs.

### Solution

Create a `find_owners` rule that consumes an `OwnersRequest` to compute the owning targets of some sources (and, optionally, their transitive dependents). Moving this logic into an `@rule` allows us to directly consume the symbol_table_contraint, without a wrapping collection (as in #5579).

Additionally, switch back to requesting target adaptor subclasses while building the `DependentGraph`. The graph-validation that use to occur via requesting `TransitiveHydratedTargets` now occurs as an explicit step during construction of a `DependentGraph`. A TODO there explains an existing weakness of change-detection in the engine.

### Result

```
./pants --changed-diffspec='49cc12e5cccc6164a2c4aa83c23300ad7254836f^..49cc12e' --changed-include-dependees=direct --glob-expansion-failure=ignore list
```
is 30 percent faster in the pantsbuild pants repo, primarily due to not expanding source globs.
  • Loading branch information
Stu Hood authored and wisechengyi committed Nov 20, 2018
1 parent a840973 commit dfb1e8d
Show file tree
Hide file tree
Showing 11 changed files with 217 additions and 494 deletions.
1 change: 0 additions & 1 deletion src/python/pants/bin/BUILD
Expand Up @@ -22,7 +22,6 @@ python_library(
'src/python/pants/engine/legacy:address_mapper',
'src/python/pants/engine/legacy:graph',
'src/python/pants/engine/legacy:parser',
'src/python/pants/engine/legacy:source_mapper',
'src/python/pants/engine:build_files',
'src/python/pants/engine:fs',
'src/python/pants/engine:legacy_engine',
Expand Down
158 changes: 0 additions & 158 deletions src/python/pants/build_graph/source_mapper.py

This file was deleted.

2 changes: 1 addition & 1 deletion src/python/pants/engine/build_files.py
Expand Up @@ -264,7 +264,7 @@ def create_graph_rules(address_mapper, symbol_table):
symbol_table_constraint = symbol_table.constraint()

partial_hydrate_struct = functools.partial(hydrate_struct, symbol_table_constraint)
partial_hydrate_struct.__name__ = 'hydrate_struct'
functools.update_wrapper(partial_hydrate_struct, hydrate_struct)

return [
# A singleton to provide the AddressMapper.
Expand Down
13 changes: 0 additions & 13 deletions src/python/pants/engine/legacy/BUILD
Expand Up @@ -82,16 +82,3 @@ python_library(
'src/python/pants/util:objects',
],
)

python_library(
name='source_mapper',
sources=['source_mapper.py'],
dependencies=[
':address_mapper',
':graph',
'src/python/pants/base:specs',
'src/python/pants/build_graph',
'src/python/pants/source',
'3rdparty/python:six'
]
)

0 comments on commit dfb1e8d

Please sign in to comment.