Separate impl items from the parent impl

[nikomatsakis]

This change separates impl item bodies out of the impl itself. This gives incremental more resolution. In so doing, it refactors how the visitors work, and cleans up a bit of the collect/check logic (mostly by moving things out of collect that didn't really belong there, because they were just checking conditions).

However, this is not as effective as I expected, for a kind of frustrating reason. In particular, when invoking foo.bar() you still wind up with dependencies on private items. The problem is that the method resolution code scans that list for methods with the name bar -- and this winds up touching all the methods, even private ones.

I can imagine two obvious ways to fix this:

• separating fn bodies from fn sigs (#35078, currently being pursued by @flodiebold)
• a more aggressive model of incremental that @michaelwoerister has been advocating, in which we hash the intermediate results (e.g., the outputs of collect) so that we can see that the intermediate result hasn't changed, even if a particular impl item has changed.

So all in all I'm not quite sure whether to land this or not. =) It still seems like it has to be a win in some cases, but not with the test cases we have just now. I can try to gin up some test cases, but I'm not sure if they will be totally realistic. On the other hand, some of the early refactorings to the visitor trait seem worthwhile to me regardless.

cc #36349 -- well, this is basically a fix for that issue, I guess

r? @michaelwoerister

NB: Based atop of @eddyb's PR #37402; don't land until that lands.

[nikomatsakis]

Hmm, had a thought last night. Another option that might make sense is to pull the Name for each impl item into the parent impl (it can appear both places). This would allow us to do the method search to find methods of name foo while just reading from the parent impl item. Moreover, it solves the problem better than separate the signature from the body: just doing that would mean that if I change the signature of a private method, we have to recompile callers of other methods, just because we haven't separated those two things.

Obviously, if we pursued the more aggressive strategy of rehashing partial results it sort of subsumes both cases. However, I remain somewhat nervous about that in the shorter term (though longer term it seems good). But also it seems advantageous to be able to determine as much as we can without having to re-run parts of the compiler.

[eddyb]

@nikomatsakis Maybe #37402 plus some dep node changes can solve that?
It moves out type information from ty::AssociatedItem so it's less than the signature.

[nikomatsakis]

@eddyb well, it seems to me that the current dep information is correct, unless we move the Name out into the impl. That is, if I were to change the name of various inherent methods on Foo, I ought to re-type-check things that called methods on Foo. And, the way things are setup now, that would be reflected in a change to the impl-item -- indistinguishable from other changes.

So fundamentally there are two ways to fix this:

• refine the HIR so that we separate out the things of interest:
  • this covers both separating sig from body and separating name from impl
  • the more precise we get it, the better things work, so doing both is prob good
• change our approach to hash intermediate results:
  • this compensates for imprecise HIR information

It seems to me that for best results we want both. Having more precise information lets us quickly rule out work without even need to reconfirm. But at a certain point you get diminishing returns. Though perhaps the initial work to recompute signatures and so forth is cheap enough that it doesn't make sense to separate HIR more. I can easily believe that.

I've talked about this with @michaelwoerister a fair amount. I tend to view this as "firewalls". That is, you have points where we hash the intermediate results because we can't (or don't care to) track the edges closely enough. The first example is from the files to the HIR. The second (existing) example is after trans collection. It makes sense to do it in more places, I suspect. Then in between these firewalls you can be conservative and just trace out the impact of a chance and invalidate the work in between said change and the next "firewall" (where you will want to re-hash). This will all fit very nicely with the work you are doing to make the compiler more demand-driven, of course, so I had hoped we could start moving more in that direction as that proceeds, but in short term get the HIR precise enough. (I still believe that.)

[michaelwoerister]

Another option that might make sense is to pull the Name for each impl item into the parent impl (it can appear both places).

That seems like a good idea to me.

[michaelwoerister]

Obviously, if we pursued the more aggressive strategy of rehashing partial results it sort of subsumes both cases.

I also think that is true and as a consequence we should avoid doing any contortions in HIR representation just to improve dependency tracking. But the examples so far seem to make sense on their on right.

[michaelwoerister]

Why wrap this in a closure?

[michaelwoerister]

So, I did one pass looking through all the commits and I think it should indeed be merged. There are many good refactorings in it that make things cleaner, especially with respect to what's going on in ty::collect. Also making the different levels for HIR visitors more explicit is a good thing.

[nikomatsakis]

@michaelwoerister so I just pushed a new commit intended to make the "nested" pattern a little more future-proof. This was mildly more tedious than I thought because it requires having better lifetime annotations to make it work (most visitors are written to accept a much wider range of lifetimes than is actually needed, since it winds up needing less explicit use of 'tcx). But it seems like an improvement regardless. Take a look.

[bors]

☔️ The latest upstream changes (presumably #37670) made this pull request unmergeable. Please resolve the merge conflicts.

[michaelwoerister]

I just pushed a new commit intended to make the "nested" pattern a little more future-proof.

I like it. It's a bit unfortunate with the additional lifetime annotations but not really a problem.
It's a bit subtle that overriding nested_visit_map() will turn the visitor into a nested one. Maybe a name like nesting_enabled for the method would improve that?

[nikomatsakis]

@michaelwoerister

It's a bit subtle that overriding nested_visit_map() will turn the visitor into a nested one. Maybe a name like nesting_enabled for the method would improve that?

Sure, I can rename it. I'd prefer to leverage specialization to make a NestedVisitor trait or something but we can't quite do that nicely.

Also, I just pushed a PR that separates the name into the impl, which actually fixes the test cases that were blocked on this.

[michaelwoerister]

This is one of the cases where it would make sense to also store the kind of associated item in the ref.

[eddyb]

I think I agree. I'd almost go as far as having associated_item use the DefId of its container for the dep node, and then just use associated_items everywhere.

[michaelwoerister]

Also, I just pushed a PR that separates the name into the impl, which actually fixes the test cases that were blocked on this.

I just looked through the new commits. Very nice :)

[nikomatsakis]

@michaelwoerister @eddyb ready for re-review; I removed AssociatedItemRef and consolidated everything into ImplItemRef (the same should be done for traits, but I'd prefer to leave it for later).

I think there is one thing missing from this PR before it can land though: we are not hashing all of the things in the ImplItemRef along with the impl, I suspect, so we need to do that, and I need to make some ICH tests for it.

[nikomatsakis]

Oh, I should add -- it didn't quite work as well as the old approach across crates, but I think that's a separate issue really. I filed #37720 describing it.

[bors]

☔️ The latest upstream changes (presumably #37730) made this pull request unmergeable. Please resolve the merge conflicts.

[eddyb]

Lovely! This might actually solve my trouble with this ordering thing, looks like it's self-contained now.

[eddyb]

r=me with a rebase.

[nikomatsakis]

@bors r+

[bors]

📌 Commit ab79438 has been approved by nikomatsakis

[nikomatsakis]

@bors r=eddyb

[bors]

💡 This pull request was already approved, no need to approve it again.

• There's another pull request that is currently being tested, blocking this pull request: #37732

[bors]

📌 Commit ab79438 has been approved by eddyb

[nikomatsakis]

@bors r=eddyb

[bors]

📌 Commit c938007 has been approved by eddyb

[bors]

⌛️ Testing commit c938007 with merge 35e8924...

[bors]

☀️ Test successful - auto-linux-32-nopt-t, auto-linux-32-opt, auto-linux-64-cargotest, auto-linux-64-cross-freebsd, auto-linux-64-debug-opt, auto-linux-64-nopt-t, auto-linux-64-opt, auto-linux-64-opt-rustbuild, auto-linux-64-x-android-t, auto-linux-cross-opt, auto-linux-musl-64-opt, auto-mac-32-opt, auto-mac-64-opt, auto-mac-64-opt-rustbuild, auto-mac-cross-ios-opt, auto-win-gnu-32-opt, auto-win-gnu-32-opt-rustbuild, auto-win-gnu-64-opt, auto-win-msvc-32-opt, auto-win-msvc-64-cargotest, auto-win-msvc-64-opt, auto-win-msvc-64-opt-rustbuild
Approved by: eddyb
Pushing 35e8924 to master...