[engine] Split SelectDependencies into SelectDependencies and SelectTransitive #4334
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ve is created on the rust side
stuhood
approved these changes
Mar 15, 2017
| @@ -121,13 +120,26 @@ def __repr__(self): | |||
| field_name_portion = ', {}'.format(repr(self.field)) | |||
| else: | |||
| field_name_portion = '' | |||
| return '{}({}, {}{}{}{})'.format(type(self).__name__, | |||
There was a problem hiding this comment.
I don't think this overridden repr is still used anywhere... may be able to remove it.
There was a problem hiding this comment.
Oh, correction... it's used only in unit tests.
src/python/pants/engine/selectors.py
Outdated
| field_types_portion) | ||
|
|
||
|
|
||
| class SelectTransitive(datatype('Dependencies', ['product', 'dep_product', 'field', 'field_types']), |
There was a problem hiding this comment.
s/Dependencies/Transitive/
src/rust/engine/src/nodes.rs
Outdated
| } | ||
|
|
||
| fn store(&self, dep_product: &Value, dep_values: Vec<&Value>) -> Value { | ||
| // Not an inner node, or not a traversal. |
There was a problem hiding this comment.
Comment is probably stale. But also, you can probably just inline this into the caller.
peiyuwang
added a commit
that referenced
this pull request
Mar 22, 2017
…esses within SelectTransitive (#4349) ### Problem As described in #4283, the issue with recursively executing `SelectTransitive` is lots of calls to `store_list` creating massive intermediate lists that increases memory footprint as well as impacts perf. ### Solution Same idea as in #4265 using `loop_fn` rewrite `SelectTransitive` to execute iteratively. One small difference is in order to use `OrderMap` and `HashSet` for state tracking and outputs are in `Value`s, those `Values` are turned into `Key`s. ps: previous PR #4334 paved way for this optimization by splitting `SelectTransitive` from `SelectDependencies(...transitive=true)`. ### Result All existing tests pass. Have verified there is only one `ST` node through viz. Two follow up items * #4358 reenable cycle detection * #3925 Remove BFA now not only is a clean up task, but has performance benefits because the second round hydration is on `Address` which is redundant.
lenucksi
pushed a commit
to lenucksi/pants
that referenced
this pull request
Apr 25, 2017
…ransitive (pantsbuild#4334) ### Problem This is a refactoring change with no functionalities added. it will simplify pantsbuild#4283 so we could optimize just for `SelectTransitive`. ### Solution Introduce `SelectTransitive` with identical attributes with `SelectDependencies` (for now), and split the `if` `else` switch in node handling into each node. With that, `transitive` flag is not needed. ### Result Test all passes.
lenucksi
pushed a commit
to lenucksi/pants
that referenced
this pull request
Apr 25, 2017
…esses within SelectTransitive (pantsbuild#4349) ### Problem As described in pantsbuild#4283, the issue with recursively executing `SelectTransitive` is lots of calls to `store_list` creating massive intermediate lists that increases memory footprint as well as impacts perf. ### Solution Same idea as in pantsbuild#4265 using `loop_fn` rewrite `SelectTransitive` to execute iteratively. One small difference is in order to use `OrderMap` and `HashSet` for state tracking and outputs are in `Value`s, those `Values` are turned into `Key`s. ps: previous PR pantsbuild#4334 paved way for this optimization by splitting `SelectTransitive` from `SelectDependencies(...transitive=true)`. ### Result All existing tests pass. Have verified there is only one `ST` node through viz. Two follow up items * pantsbuild#4358 reenable cycle detection * pantsbuild#3925 Remove BFA now not only is a clean up task, but has performance benefits because the second round hydration is on `Address` which is redundant.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
This is a refactoring change with no functionalities added. it will simplify #4283 so we could optimize just for
SelectTransitive.Solution
Introduce
SelectTransitivewith identical attributes withSelectDependencies(for now), and split theifelseswitch in node handling into each node. With that,transitiveflag is not needed.Result
Test all passes.