Nhoward/rust graph validation pt i #4227

baroquebobcat · 2017-02-01T07:28:41Z

This is the first step towards adapting the static noop culling I worked on last year to the rust code.

This first patch is working towards replacing the graph validator written in python with one in rust. At this point it covers quite a lot of behavior, but it isn't working quite correctly yet.

I'm putting it up in the current state to get some other eyes on it while I iterate.

I'm hoping to have this initial patch wrapped up tomorrow, modulo bugs.

Things to look at:

I replaced the root selector fn dict with a set of subjects and a method. Since all of the subjects now use the same selector, it didn't make sense.
rule_graph.rs is where the meat of the port is. It still is a bit messy with commented python in places. I'm planning on removing the rest of them and leaving TODOs where I want to pick up the thread in the next patch.
I've also added a few comments in the extension points for the next phase, but I can remove or clear them up if they are a problem.

The current graph validation is written in Python. This begins to port it to rust by introducing the basic structure needed. It's still pretty rough. I've included a number of elements that are not yet used which could be removed for this iteration.

stuhood

Thanks Nick!

stuhood · 2017-02-01T20:34:04Z

src/python/pants/engine/subsystem/native.py

@@ -149,6 +149,8 @@
    ExecutionStat execution_execute(RawScheduler*);
    RawNodes* execution_roots(RawScheduler*);

+    void run_validator(RawScheduler*, TypeId*, uint64_t);


Naming nit... the methods are mostly namespaced. So maybe validator_run.

stuhood · 2017-02-01T20:36:28Z

src/rust/engine/src/graph.rs

@@ -213,6 +216,7 @@ impl Graph {
      let src = self.entry_for_id(src_id);
      dsts.into_iter()
        .filter(|dst_id| !(src.dependencies.contains(dst_id) || src.cyclic_dependencies.contains(dst_id)))
+        // RGTODO add filter that removes deps if they will noop based on their rule edges


I think that this will obviously be part of phase 2/3, but it's not clear that this is where it would happen, so the todo is probably misplaced.

stuhood · 2017-02-01T20:38:31Z

src/rust/engine/src/rule_graph.rs

+
+// I think I ought to be able to replace the below with a set of structs keyed by EntryType.
+// My first couple attempts failed.
+#[derive(Eq, Hash, PartialEq, Clone, Debug)]


Since this struct is probably 1 byte wide, just marking it Copy rather than Clone would be perfectly reasonable, and allow you to avoid explicitly calling clone in a bunch of places.

(...and ignore this if you end up going the "one big enum" route.)

stuhood · 2017-02-01T20:44:14Z

src/rust/engine/src/rule_graph.rs

+      entry_type: EntryType::Unreachable,
+      subject_type: None,
+      value: None,
+      rule: Some(rule.clone()), // TODO decide on clone vs lifetimes


I'd stick with cloning for now. Can revisit later if it's an issue, but it seems unlikely to be.

stuhood · 2017-02-01T20:44:39Z

src/rust/engine/src/rule_graph.rs

+      selector: None,
+      reason: None,
+    }
+  }


missing whitespace

stuhood · 2017-02-01T21:04:48Z

src/rust/engine/src/tasks.rs

@@ -8,9 +8,13 @@ use selectors::{Selector, Select, SelectDependencies, SelectLiteral, SelectProje
 * Registry of tasks able to produce each type, along with a few fundamental python
 * types that the engine must be aware of.
 */
+#[derive(Clone)]


Can you open a github issue about removing Clone here? It's a bit error prone, because the tasks get passed around everywhere currently, and cloning them accidentally (probably in a parent struct) would get costly fast.

Done. #4236

stuhood · 2017-02-01T21:07:30Z

src/rust/engine/src/tasks.rs

+    self.all_rules().iter().map(|t| t.product).collect()
+  }
+
+  pub fn is_singleton_task(&self, sought_task: &Task) -> bool {


Worth adding a Singleton/Intrinisic/Normal enum on Task?

hm. that would certainly simplify these. I think I'd want to split that out as a separate change though. Sound reasonable?

stuhood · 2017-02-01T21:08:58Z

src/rust/engine/src/tasks.rs

+  }
+
+  pub fn all_rules(&self) -> Vec<Task> {
+    let mut result: Vec<Task> = Vec::new();


Another way to accomplish method would be:

let rules: Vec<_> = self.tasks.values().chain(self.singletons.values()).chain(self.singletons.values()).collect();

neat. I had to add a flat_map, but that's much more concise without losing readability.

stuhood · 2017-02-01T21:12:18Z

src/rust/engine/src/rule_graph.rs

+    let rules_in_graph: HashSet<Task> = full_dependency_edges.keys().map(|f| f.rule().clone()).collect();
+    let unfulfillable_discovered_during_construction: HashSet<Task> = full_unfulfillable_rules.keys().map(|f| f.rule().clone()).collect();
+    let declared_rules = self.tasks.all_rules();
+    let unreachable_rules: HashSet<&Task> = declared_rules.iter()


Note that when the type annotation is use to guide inference for collect, it isn't necessary to actually fill in its parameters:

let unreachable_rules: HashSet<_> = declared_rules.iter()...

ah, so it's like the diamond operator in Java. Also neat. Is doing it that way preferred over the turbo fish? ie let foo = ...collect::<Vec<_>>()

stuhood · 2017-02-01T21:19:59Z

src/rust/engine/src/rule_graph.rs

+          .filter(|g| !root_rule_dependency_edges.contains_key(g))
+          .map(|g| g.clone());
+        for unseen_rule in unseen_dep_rules {
+          rules_to_traverse.push_back(unseen_rule);


Can use extend here, which consumes an iterator.

rules_to_traverse.extend(unseen_dep_rules);

stuhood

I think my only blocker for landing this would be cleaning up the commented code a bit.

Tweaking the datamodel a bit would likely lay important groundwork too, but I won't block on it.

…o edge collections are properly typed

stuhood

Thanks Nick!

stuhood · 2017-02-03T22:26:18Z

src/rust/engine/src/graph.rs

@@ -1,3 +1,5 @@
+// Copyright 2016 Pants project contributors (see CONTRIBUTORS.md).


If you're going to add these, might as well go with the current year.

stuhood · 2017-02-03T22:27:20Z

src/rust/engine/src/selectors.rs

@@ -59,6 +59,9 @@ impl Selector {
    )
  }

+  /*
+  * The product type this selector will ultimately produce.


I know we're not really following rust style at all, but the indentation is funny here.

yeah. I'll reformat to be at least Javaish

Hm. After thinking a bit, I looked at the Rust book again. It seems like line comments are preferred. And the doc comments are form of line comments. I'm changing it to line comments
https://doc.rust-lang.org/beta/book/comments.html

stuhood · 2017-02-03T22:29:20Z

src/rust/engine/src/rule_graph.rs

+// I think I ought to be able to replace the below with a set of structs keyed by EntryType.
+// My first couple attempts failed.
+#[derive(Eq, Hash, PartialEq, Clone, Debug)]
+enum EntryType {


Much better! It looks like is now just Entry?

stuhood · 2017-02-03T22:32:00Z

src/rust/engine/src/rule_graph.rs

+
+              },
+              &Selector::Task(ref select) =>{
+                // TODO, not sure what task is in this context exactly


It's for symmetry: every Node type has a Selector currently. But probably possible to remove.

-- also removed the Clone on Tasks since an upstream change caused it to not be viable

baroquebobcat · 2017-02-06T18:28:38Z

src/rust/engine/src/tasks.rs

@@ -8,9 +8,14 @@ use selectors::{Selector, Select, SelectDependencies, SelectLiteral, SelectProje
 * Registry of tasks able to produce each type, along with a few fundamental python
 * types that the engine must be aware of.
 */
+// TODO remove Clone https://github.com/pantsbuild/pants/issues/4236
+#[derive(Clone)]


With the merge from master, I was forced to handle this. So it's gone. Closing the issue.

- fix year - s/EntryType/Entry/ - change comments - clarify panics in a few places

…ld#4227) This is the first step towards adapting the static noop culling I worked on last year to the Rust code base. This first patch is working towards replacing the graph validator written in python with one in rust. At this point it covers quite a lot of behavior, but it isn't working quite correctly yet. This patch includes porting the majority of the rule graph construction. Including - replacing the root selector fn dict with a set of subjects and a method. Since all of the subjects now use the same selector, it didn't make sense to keep. - running the rust validator, but ignoring the results. - adding copyright headers. rule_graph.rs is where the meat of the port is. There are still a number of unported sections. I'll be removing the commented python and replacing it with rust in subsequent pull requests The remaining portions are - adding a enum on tasks for their task type - removing unfulfillable entries from the graph - handling SelectDependencies / SelectProjection selectors - handling snapshots - propagating the results back to the python side

baroquebobcat added 13 commits January 31, 2017 11:58

replace property lambda with method

75820ed

rm stale comment from native.py

a68c608

clear some things up and do more translation

33ae760

impl filter based comment

a7e9cb5

do the unreachable diagnostic bit

06382d6

another bit of conversion

f9991a1

comment shuffling, handle the no selector case

46f731e

convert literal clause

10b5646

remove a bunch of unnecessary stuff

2f7a15b

fix some warnings

687b308

more cleanup, impl RuleEdges methods

5bd1e38

runs, but produces an incorrect result

df50d16

stuhood reviewed Feb 1, 2017

View reviewed changes

baroquebobcat added 5 commits February 2, 2017 23:25

pull the remaining threads on the dict -> set for root subject types

251dbc6

clean up unnecessary comments, rm unneeded lifetimes

e431463

bump engine version, rm extra ws

962932a

flip the Entry into the EntryType enum, and specialize Root / Inner s…

4fa58c5

…o edge collections are properly typed

use a match instead of defining input_selectors on EntryType

4f62f5a

stuhood approved these changes Feb 3, 2017

View reviewed changes

baroquebobcat added 2 commits February 6, 2017 10:15

Merge branch 'master' into nhoward/rust_graph_validation_pt_i

2e67f39

-- also removed the Clone on Tasks since an upstream change caused it to not be viable

bump native version

a942bce

baroquebobcat commented Feb 6, 2017

View reviewed changes

baroquebobcat added 2 commits February 6, 2017 10:32

respond to last round of comments

7b7bfae

- fix year - s/EntryType/Entry/ - change comments - clarify panics in a few places

missed the year in lib.rs

a9917ec

baroquebobcat merged commit 62f2b10 into pantsbuild:master Feb 6, 2017

stuhood mentioned this pull request Feb 8, 2017

Guarantee the ability to hit the engine's cache without fetching contents #3365

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nhoward/rust graph validation pt i #4227

Nhoward/rust graph validation pt i #4227

baroquebobcat commented Feb 1, 2017

stuhood left a comment

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

stuhood Feb 1, 2017

stuhood Feb 1, 2017

stuhood Feb 1, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood Feb 1, 2017

baroquebobcat Feb 3, 2017

stuhood left a comment

stuhood left a comment

stuhood Feb 3, 2017

baroquebobcat Feb 6, 2017

stuhood Feb 3, 2017

baroquebobcat Feb 6, 2017

baroquebobcat Feb 6, 2017

stuhood Feb 3, 2017

baroquebobcat Feb 6, 2017

stuhood Feb 3, 2017

baroquebobcat Feb 6, 2017

		@@ -1,3 +1,5 @@
		// Copyright 2016 Pants project contributors (see CONTRIBUTORS.md).

Nhoward/rust graph validation pt i #4227

Nhoward/rust graph validation pt i #4227

Conversation

baroquebobcat commented Feb 1, 2017

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment