incr.comp.: Hash more pieces of crate metadata to detect changes there. #41709

michaelwoerister · 2017-05-02T15:55:55Z

This PR adds incr. comp. hashes for non-Entry pieces of data in crate metadata.

The first part of it I like: EntryBuilder is refactored into the more generally applicable IsolatedEncoder which provides means of encoding something into metadata while also feeding the encoded data into an incr. comp. hash. We already did this for Entry, now we are doing it for various other pieces of data too, like the set of exported symbols and so on. The hashes generated there are persisted together with the per-Entry hashes and are also used for dep-graph dirtying the same way.

The second part of the PR I'm not entirely happy with: In order to make sure that we don't forget registering a read to the new DepNodes introduced here, I added the Tracked<T> struct. This struct wraps a value and requires a DepNode when accessing the wrapped value. This makes it harder to overlook adding read edges in the right places and works just fine.
However, crate metadata is already used in places where there is no tcx yet or even in places where no cnum has been assigned -- this makes it harder to apply this feature consistently or implement it ergonomically. The result is not too bad but there's a bit more code churn and a bit more opportunity to get something wrong than I would have liked. On the other hand, wrapping things in Tracked<T> already has revealed some bugs, so there's definitely some value in it.

This is still a work in progress:

I need to write some test cases.
Accessing the CodeMap should really be dependency tracked too, especially with the new path-remapping feature.

cc @nikomatsakis

rust-highfive · 2017-05-02T15:55:59Z

r? @arielb1

(rust_highfive has picked a reviewer for you, use r? to override)

bors · 2017-05-03T05:57:59Z

☔ The latest upstream changes (presumably #41717) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister · 2017-05-04T14:16:11Z

This should be ready for review now.

One interesting thing in the changes here is that some DepNode variants have a D as identifier, although a CrateNum would be sufficient. This is done because D fields are properly handled during DefPath retracing while CrateNums are not. By convention, I always used a DefId with the DefIndex set to CRATE_DEF_INDEX when only the CrateNum was interesting.

r? @nikomatsakis

michaelwoerister · 2017-05-04T19:31:01Z

I'm not sure if the errors on Travis are legit. I'll need to test locally.

bors · 2017-05-06T01:50:59Z

☔ The latest upstream changes (presumably #41773) made this pull request unmergeable. Please resolve the merge conflicts.

michaelwoerister · 2017-05-08T11:52:02Z

All checks passing now.

nikomatsakis

Gotta' few questions

nikomatsakis · 2017-05-08T18:23:14Z

src/librustc/session/mod.rs

+    // Hook up the codemap with a callback that allows it to register FileMap
+    // accesses with the dependency graph.
+    let cm_depgraph = dep_graph.clone();
+    let codemap_dep_tracking_callback = Box::new(move |filemap: &FileMap| {


I see why you were unhappy.

This makes me wonder: I think that eventually we probably want some way to "hide" data in the tcx (e.g., maybe using privacy), so that it is only exposed via query. This would be a way to have stuff like random bits in the session be treated universally by the red-green system.

(For stuff that's hashed initially, like command-line-options, we might want to make some public accessor methods I guess.)

Yes, some refactorings like that would be nice.

nikomatsakis · 2017-05-08T18:24:22Z

src/librustc_incremental/calculate_svh/mod.rs

+                                    }
+                                    DepNode::FileMap(..) => {
+                                        // These don't make a semantic
+                                        // difference, filter them out.


I'm not sure I follow this -- we use the SVH as a shorthand for "anything changed at all in this crate", you're saying that changes in the filemap shouldn't (e.g.) trigger downstream rebuilds when we are being conservative?

I added this filter in order to appease the svh-related tests in run-pass. There are a few of them asserting that adding a comment (for example) does not change the SVH. These tests are still around from when the SVH tried to encode binary compatibility of crates, I think. The SVH has gotten a bit stricter since, but it's probably good that it's not too sensitive.

Yeah, those tests seem pretty .. rotten. But ok.

nikomatsakis · 2017-05-08T18:27:57Z

src/libsyntax_pos/lib.rs

@@ -491,6 +493,8 @@ impl Decodable for FileMap {
            Ok(FileMap {
                name: name,
                name_was_remapped: name_was_remapped,
+                // `crate_of_origin` has to be set by the importer.
+                crate_of_origin: 0xEFFF_FFFF,


Wait, what's this? :)

Sorry :)
It's a number that is unlikely to ever be a valid CrateNum. 0xFFFF_FFFF is already taken for some other special purpose. I should introduce an INVALID_CRATE const.

nikomatsakis · 2017-05-08T21:33:38Z

ok well r=me

michaelwoerister · 2017-05-09T10:48:29Z

@bors r=nikomatsakis

I added the INVALID_CRATE constant only to remember that it can't be used in libsyntax_pos :P.

bors · 2017-05-09T10:48:30Z

📌 Commit 115602b has been approved by nikomatsakis

bors · 2017-05-09T11:55:42Z

⌛ Testing commit 115602b with merge f3fc547...

@nikomatsakis

…ikomatsakis incr.comp.: Hash more pieces of crate metadata to detect changes there. This PR adds incr. comp. hashes for non-`Entry` pieces of data in crate metadata. The first part of it I like: `EntryBuilder` is refactored into the more generally applicable `IsolatedEncoder` which provides means of encoding something into metadata while also feeding the encoded data into an incr. comp. hash. We already did this for `Entry`, now we are doing it for various other pieces of data too, like the set of exported symbols and so on. The hashes generated there are persisted together with the per-`Entry` hashes and are also used for dep-graph dirtying the same way. The second part of the PR I'm not entirely happy with: In order to make sure that we don't forget registering a read to the new `DepNodes` introduced here, I added the `Tracked<T>` struct. This struct wraps a value and requires a `DepNode` when accessing the wrapped value. This makes it harder to overlook adding read edges in the right places and works just fine. However, crate metadata is already used in places where there is no `tcx` yet or even in places where no `cnum` has been assigned -- this makes it harder to apply this feature consistently or implement it ergonomically. The result is not too bad but there's a bit more code churn and a bit more opportunity to get something wrong than I would have liked. On the other hand, wrapping things in `Tracked<T>` already has revealed some bugs, so there's definitely some value in it. This is still a work in progress: - [x] I need to write some test cases. - [x] Accessing the CodeMap should really be dependency tracked too, especially with the new path-remapping feature. cc @nikomatsakis

bors · 2017-05-09T14:30:32Z

☀️ Test successful - status-appveyor, status-travis
Approved by: nikomatsakis
Pushing f3fc547 to master...

@nikomatsakis

…ookup, r=nikomatsakis ICH: Handle case of removed FileMaps. This PR fixes a bug introduced in rust-lang#41709 where removing a source file between compilation sessions would cause an ICE: https://travis-ci.org/rust-icci/crossbeam/jobs/230582234#L633 r? @nikomatsakis

@nikomatsakis

…ookup, r=nikomatsakis ICH: Handle case of removed FileMaps. This PR fixes a bug introduced in rust-lang#41709 where removing a source file between compilation sessions would cause an ICE: https://travis-ci.org/rust-icci/crossbeam/jobs/230582234#L633 r? @nikomatsakis

@nikomatsakis

…ookup, r=nikomatsakis ICH: Handle case of removed FileMaps. This PR fixes a bug introduced in rust-lang#41709 where removing a source file between compilation sessions would cause an ICE: https://travis-ci.org/rust-icci/crossbeam/jobs/230582234#L633 r? @nikomatsakis

king6cong · 2017-05-21T08:58:34Z

#42101
this issue should be related to this commit

rust-highfive assigned arielb1 May 2, 2017

aidanhs added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label May 3, 2017

michaelwoerister force-pushed the close-metadata-ich-holes branch 2 times, most recently from 1df872d to e3e4fde Compare May 4, 2017 12:39

michaelwoerister changed the title ~~(WIP) incr.comp.: Hash more pieces of crate metadata to detect changes there.~~ incr.comp.: Hash more pieces of crate metadata to detect changes there. May 4, 2017

michaelwoerister force-pushed the close-metadata-ich-holes branch from e3e4fde to b973646 Compare May 4, 2017 14:09

rust-highfive assigned nikomatsakis and unassigned arielb1 May 4, 2017

Mark-Simulacrum added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels May 7, 2017

michaelwoerister force-pushed the close-metadata-ich-holes branch from b973646 to b4b14eb Compare May 8, 2017 10:28

incr.comp.: Hash more pieces of crate metadata to detect changes there.

6a5e2a5

michaelwoerister force-pushed the close-metadata-ich-holes branch from b4b14eb to 6a5e2a5 Compare May 8, 2017 10:31

michaelwoerister added S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. and removed S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. labels May 8, 2017

nikomatsakis reviewed May 8, 2017

View reviewed changes

Add INVALID_CRATE CrateNum constant.

115602b

bors merged commit 115602b into rust-lang:master May 9, 2017

michaelwoerister mentioned this pull request May 10, 2017

ICH: Handle case of removed FileMaps. #41873

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

incr.comp.: Hash more pieces of crate metadata to detect changes there. #41709

incr.comp.: Hash more pieces of crate metadata to detect changes there. #41709

michaelwoerister commented May 2, 2017 •

edited

Loading

rust-highfive commented May 2, 2017

bors commented May 3, 2017

michaelwoerister commented May 4, 2017

michaelwoerister commented May 4, 2017

bors commented May 6, 2017

michaelwoerister commented May 8, 2017

nikomatsakis left a comment

nikomatsakis May 8, 2017

michaelwoerister May 8, 2017

nikomatsakis May 8, 2017

michaelwoerister May 8, 2017

nikomatsakis May 8, 2017

nikomatsakis May 8, 2017

michaelwoerister May 8, 2017

nikomatsakis commented May 8, 2017

michaelwoerister commented May 9, 2017

bors commented May 9, 2017

bors commented May 9, 2017

bors commented May 9, 2017

king6cong commented May 21, 2017

incr.comp.: Hash more pieces of crate metadata to detect changes there. #41709

incr.comp.: Hash more pieces of crate metadata to detect changes there. #41709

Conversation

michaelwoerister commented May 2, 2017 • edited Loading

rust-highfive commented May 2, 2017

bors commented May 3, 2017

michaelwoerister commented May 4, 2017

michaelwoerister commented May 4, 2017

bors commented May 6, 2017

michaelwoerister commented May 8, 2017

nikomatsakis left a comment

Choose a reason for hiding this comment

nikomatsakis May 8, 2017

Choose a reason for hiding this comment

michaelwoerister May 8, 2017

Choose a reason for hiding this comment

nikomatsakis May 8, 2017

Choose a reason for hiding this comment

michaelwoerister May 8, 2017

Choose a reason for hiding this comment

nikomatsakis May 8, 2017

Choose a reason for hiding this comment

nikomatsakis May 8, 2017

Choose a reason for hiding this comment

michaelwoerister May 8, 2017

Choose a reason for hiding this comment

nikomatsakis commented May 8, 2017

michaelwoerister commented May 9, 2017

bors commented May 9, 2017

bors commented May 9, 2017

bors commented May 9, 2017

king6cong commented May 21, 2017

michaelwoerister commented May 2, 2017 •

edited

Loading