Add expansion info to crate metadata #43847

ibabushkin · 2017-08-13T16:16:43Z

This implements the serialization and deserialization of hygiene info in crate metadata, preserving macro expansion information across crates, addressing some of the points mentioned in #40847. There is some work left to do, however: Apparently there is code that hides macro backtraces from external crates and some of the decoding logic might need to be expanded or moved to accomodate for all current use cases I might have missed.

r? @jseyfried

ibabushkin · 2017-08-13T16:24:20Z

src/libsyntax_pos/hygiene.rs

@@ -129,13 +138,98 @@ impl HygieneData {
        }
        HYGIENE_DATA.with(|data| f(&mut *data.borrow_mut()))
    }
+
+    pub fn safe_with<T, F: FnOnce(&HygieneData) -> T>(f: F) -> T {
+        // FIXME(twk): not sure how this would behave...


I couldn't find any info on how multiple declarations of thread_local!s behave, so this might need to be moved for the expected behaviour.

Okay, a lot of symptoms point to having two thread locals being the issue. However, replacing them with one leads to absolutely uncontrolled memory consumption.

ibabushkin · 2017-08-14T14:49:24Z

So currently, the on-demand loading mechanism for hygiene data is still incorrect (though I have a local fix for that). Moreover, some tests still fail. My current theory is that encoding the hygiene data of a crate doesn't force all of it's dependencies' hygiene data to be decoded first.

One could fix this either by forcing a complete decoding procedure for each dependency upon encoding of by simply making decoding of that particular metadata strict (which I'd like to avoid).

bors · 2017-08-16T06:44:49Z

☔ The latest upstream changes (presumably #43710) made this pull request unmergeable. Please resolve the merge conflicts.

ibabushkin · 2017-08-16T11:10:57Z

For reference, to reproduce the crash on the backtrace tests in a more comfortable environment:

RUST_LOG=debug ./build/x86_64-unknown-linux-gnu/stage1/bin/rustc -g --crate-type=lib src/test/run-pass/backtrace.rs

I'm starting my search from there.

ibabushkin · 2017-08-19T18:47:06Z

Okay, the implementation now doesn't cause any crashes and is okay resource-usage-wise. What remains to do now is to remove leftover cruft, and to clean up functionality in other places. For instance, cross-crate macro backtraces would probably benefit from these changes, but the integration remains to be implemented. Finally, some tests might be in order. Given all these points, I think now is a good time to review what we've got so far.

jseyfried · 2017-08-21T19:51:26Z

Started reviewing, looks good so far. I think we should be able to test using nested macro_rules! (at least to observe that this PR behaves differently from master) and/or nested macros (with #![feature(decl_macro)]).

ibabushkin · 2017-08-21T21:52:52Z

Alright, I'll fiddle something like that together.

ibabushkin · 2017-08-25T19:23:41Z

Quick update: in the meantime, some recent changes are breaking the functionality again - I'll try to find out what happens.

bors · 2017-08-30T14:00:27Z

☔ The latest upstream changes (presumably #43968) made this pull request unmergeable. Please resolve the merge conflicts.

arielb1 · 2017-09-05T17:48:26Z

What's the status of this PR? Are you still interested in it @ibabushkin?

ibabushkin · 2017-09-05T19:23:34Z

On 2017-09-05, Ariel Ben-Yehuda wrote: What's the status of this PR? Are you still interested in it @ibabushkin? -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #43847 (comment)

Yes, absolutely. I'm sorry for the stall - some bugs (re?)appeared and some other things got in the way :/

aidanhs · 2017-09-13T21:57:27Z

Hi @ibabushkin, just a friendly ping to make sure this PR doesn't get lost! :)

arielb1 · 2017-09-19T20:42:50Z

What's the status of this PR? Are you still interested in it @ibabushkin?

ibabushkin · 2017-09-19T21:04:23Z

Yes, absolutely. I'm sorry for the delays piling up - I fear I'll only be able to get back to this early next month :/. Would you prefer me to close the PR in the meantime and reopen it when it is ready?

carols10cents · 2017-09-25T15:29:23Z

Would you prefer me to close the PR in the meantime and reopen it when it is ready?

That's up to you @ibabushkin! We're happy to hold off on the reminder pings until October if you want to leave this open :)

aidanhs · 2017-10-05T18:59:27Z

Hi @ibabushkin, you mentioned early October, so just a friendly ping to make sure you're still aware of this PR!

ibabushkin · 2017-10-07T16:14:29Z

I'm taking up the work again! :) Thank you all for your patience.

This change adds an encoded version of hygiene related datastructures to the metadata of a crate. It is then decoded on-demand when spans and AST items from a foreign crate are accessed. Some changes might still be necessary to implement the correct behaviour, however.

Sadly, this isn't the reason the tests fail.

However, now that it behaves as expected, the memory usage is unacceptably high. A solution would be more lazyness during encoding or a pruning scheme.

kennytm · 2017-10-14T06:43:33Z

src/test/compile-fail-fulldeps/macro-spans.rs

+#[macro_use]
+extern crate macro_spans_lib;
+
+abc!(Name); //~ ERROR recursive type `Name::Name` has infinite size


This test failed.

[00:56:33] ---- [compile-fail] compile-fail-fulldeps/macro-spans.rs stdout ---- [00:56:33] [00:56:33] error: /checkout/src/test/compile-fail-fulldeps/macro-spans.rs:16: expected error not found: recursive type `Name::Name` has infinite size [00:56:33] [00:56:33] error: 0 unexpected errors found, 1 expected errors not found [00:56:33] status: exit code: 101 [00:56:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail-fulldeps/macro-spans.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused" [00:56:33] not found errors (from test file): [ [00:56:33] Error { [00:56:33] line_num: 16, [00:56:33] kind: Some( [00:56:33] Error [00:56:33] ), [00:56:33] msg: "recursive type `Name::Name` has infinite size" [00:56:33] } [00:56:33] ] [00:56:33] [00:56:33] thread '[compile-fail] compile-fail-fulldeps/macro-spans.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:1084:12 [00:56:33] note: Run with `RUST_BACKTRACE=1` for a backtrace. [00:56:33] [00:56:33] [00:56:33] failures: [00:56:33] [compile-fail] compile-fail-fulldeps/macro-spans.rs [00:56:33] [00:56:33] test result: FAILED. 52 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

ibabushkin · 2017-10-14T08:48:19Z

I am aware :) - fixing it is not quite trivial, working on it.

…

On 2017-10-14, kennytm wrote: kennytm commented on this pull request. > +// Copyright 2017 The Rust Project Developers. See the COPYRIGHT +// file at the top-level directory of this distribution and at +// http://rust-lang.org/COPYRIGHT. +// +// Licensed under the Apache License, Version 2.0 <LICENSE-APACHE or +// http://www.apache.org/licenses/LICENSE-2.0> or the MIT license +// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your +// option. This file may not be copied, modified, or distributed +// except according to those terms. + +// aux-build:macro_spans_lib.rs + +#[macro_use] +extern crate macro_spans_lib; + +abc!(Name); //~ ERROR recursive type `Name::Name` has infinite size This test failed. ``` [00:56:33] ---- [compile-fail] compile-fail-fulldeps/macro-spans.rs stdout ---- [00:56:33] [00:56:33] error: /checkout/src/test/compile-fail-fulldeps/macro-spans.rs:16: expected error not found: recursive type `Name::Name` has infinite size [00:56:33] [00:56:33] error: 0 unexpected errors found, 1 expected errors not found [00:56:33] status: exit code: 101 [00:56:33] command: "/checkout/obj/build/x86_64-unknown-linux-gnu/stage2/bin/rustc" "/checkout/src/test/compile-fail-fulldeps/macro-spans.rs" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps" "--target=x86_64-unknown-linux-gnu" "--error-format" "json" "-C" "prefer-dynamic" "-o" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu" "-Crpath" "-O" "-Lnative=/checkout/obj/build/x86_64-unknown-linux-gnu/native/rust-test-helpers" "-L" "/checkout/obj/build/x86_64-unknown-linux-gnu/test/compile-fail-fulldeps/macro-spans.stage2-x86_64-unknown-linux-gnu.compile-fail.libaux" "-A" "unused" [00:56:33] not found errors (from test file): [ [00:56:33] Error { [00:56:33] line_num: 16, [00:56:33] kind: Some( [00:56:33] Error [00:56:33] ), [00:56:33] msg: "recursive type `Name::Name` has infinite size" [00:56:33] } [00:56:33] ] [00:56:33] [00:56:33] thread '[compile-fail] compile-fail-fulldeps/macro-spans.rs' panicked at 'explicit panic', /checkout/src/tools/compiletest/src/runtest.rs:1084:12 [00:56:33] note: Run with `RUST_BACKTRACE=1` for a backtrace. [00:56:33] [00:56:33] [00:56:33] failures: [00:56:33] [compile-fail] compile-fail-fulldeps/macro-spans.rs [00:56:33] [00:56:33] test result: FAILED. 52 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out ``` -- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: #43847 (review)

ibabushkin · 2017-10-15T21:57:15Z

It seems that the error display logic broke in the meantime, which is rather odd, but hints at more serious problems. I am currently considering starting this feature from scratch.

alexcrichton · 2017-10-19T14:42:37Z

ping @ibabushkin, was curious how the progress is here!

ibabushkin · 2017-10-19T21:21:07Z

@alexcrichton To be honest, I am currently hung up debugging the test failures - and I suspect more systematic problems with this changeset. I asked in #rust-internals about approaches to a more comfortable debugging/logging setup, but so far I wasn't really successful.

carols10cents · 2017-10-30T15:51:57Z

@alexcrichton To be honest, I am currently hung up debugging the test failures - and I suspect more systematic problems with this changeset. I asked in #rust-internals about approaches to a more comfortable debugging/logging setup, but so far I wasn't really successful.

@jseyfried do you have any ideas on how @ibabushkin could debug the test failures here?

bors · 2017-11-01T09:45:04Z

☔ The latest upstream changes (presumably #45674) made this pull request unmergeable. Please resolve the merge conflicts.

carols10cents · 2017-11-13T16:38:58Z

Hi @ibabushkin, thank you for this PR! Due to inactivity, merge conflicts, and the possibility you raised that there might be systematic problems with this changeset, I'm going to close this.

Please do continue to reach out when you have time to work on this to get help (try pinging @jseyfried specifically on IRC?) and either reopen this PR or start a new PR when you're ready!

rust-highfive assigned jseyfried Aug 13, 2017

ibabushkin commented Aug 13, 2017

View reviewed changes

ibabushkin closed this Aug 13, 2017

ibabushkin reopened this Aug 13, 2017

carols10cents added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Aug 14, 2017

ibabushkin force-pushed the cross-crate-expand branch from 1c29647 to 8ddc088 Compare August 15, 2017 21:00

ibabushkin force-pushed the cross-crate-expand branch 3 times, most recently from d6e9dee to 3a5c9f1 Compare August 19, 2017 17:10

alexcrichton added S-waiting-on-author Status: This is awaiting some action (such as code changes or more information) from the author. and removed S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Aug 24, 2017

ibabushkin added 3 commits October 7, 2017 18:33

Added an explanation to slightly tricky decoding logic.

e6a6c64

Fixed build after rebase

b47a791

ibabushkin added 8 commits October 7, 2017 18:35

Actually encode syntax context info for spans.

7bb56bf

Fixed the on-demand decoding mechanism for hygiene data.

214fb56

Fixed one possible error source when encoding hygiene metadata.

64934ba

Sadly, this isn't the reason the tests fail.

Fixed broken functionality.

754bf65

However, now that it behaves as expected, the memory usage is unacceptably high. A solution would be more lazyness during encoding or a pruning scheme.

Implemented first steps of a new, leaner hygiene data encoding scheme.

b653026

Fixed on-demand hygiene info encoding.

e3e790a

Added serialization instances after rebase.

3927f51

Removed some cruft added for macro expansion serialization.

909808f

ibabushkin force-pushed the cross-crate-expand branch from 05a6684 to 909808f Compare October 7, 2017 16:47

ibabushkin added 2 commits October 11, 2017 20:20

Fixed submodules. Kinda.

6cb141a

(kind of) fixed build.

b79b450

kennytm reviewed Oct 14, 2017

View reviewed changes

carols10cents closed this Nov 13, 2017

eddyb mentioned this pull request Mar 30, 2018

macro_rules macros should not be serialized through strings in crate metadata. #49511

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add expansion info to crate metadata #43847

Add expansion info to crate metadata #43847

ibabushkin commented Aug 13, 2017

ibabushkin Aug 13, 2017

ibabushkin Aug 17, 2017

ibabushkin commented Aug 14, 2017 •

edited

bors commented Aug 16, 2017

ibabushkin commented Aug 16, 2017

ibabushkin commented Aug 19, 2017

jseyfried commented Aug 21, 2017

ibabushkin commented Aug 21, 2017

ibabushkin commented Aug 25, 2017

bors commented Aug 30, 2017

arielb1 commented Sep 5, 2017

ibabushkin commented Sep 5, 2017 via email

aidanhs commented Sep 13, 2017

arielb1 commented Sep 19, 2017

ibabushkin commented Sep 19, 2017

carols10cents commented Sep 25, 2017

aidanhs commented Oct 5, 2017

ibabushkin commented Oct 7, 2017

kennytm Oct 14, 2017

ibabushkin commented Oct 14, 2017 via email

ibabushkin commented Oct 15, 2017

alexcrichton commented Oct 19, 2017

ibabushkin commented Oct 19, 2017

carols10cents commented Oct 30, 2017

bors commented Nov 1, 2017

carols10cents commented Nov 13, 2017

Add expansion info to crate metadata #43847

Add expansion info to crate metadata #43847

Conversation

ibabushkin commented Aug 13, 2017

ibabushkin Aug 13, 2017

Choose a reason for hiding this comment

ibabushkin Aug 17, 2017

Choose a reason for hiding this comment

ibabushkin commented Aug 14, 2017 • edited

bors commented Aug 16, 2017

ibabushkin commented Aug 16, 2017

ibabushkin commented Aug 19, 2017

jseyfried commented Aug 21, 2017

ibabushkin commented Aug 21, 2017

ibabushkin commented Aug 25, 2017

bors commented Aug 30, 2017

arielb1 commented Sep 5, 2017

ibabushkin commented Sep 5, 2017 via email

aidanhs commented Sep 13, 2017

arielb1 commented Sep 19, 2017

ibabushkin commented Sep 19, 2017

carols10cents commented Sep 25, 2017

aidanhs commented Oct 5, 2017

ibabushkin commented Oct 7, 2017

kennytm Oct 14, 2017

Choose a reason for hiding this comment

ibabushkin commented Oct 14, 2017 via email

ibabushkin commented Oct 15, 2017

alexcrichton commented Oct 19, 2017

ibabushkin commented Oct 19, 2017

carols10cents commented Oct 30, 2017

bors commented Nov 1, 2017

carols10cents commented Nov 13, 2017

ibabushkin commented Aug 14, 2017 •

edited