macros: stackless expansion #36214

jseyfried · Sep 2, 2016

After this PR, macro expansion cannot overflow the stack unless the expanded crate is too deep to fold.
Everything but the stackless placeholder expansion commit is also groundwork for macro modularization.

cc #35896
r? @nrc or @eddyb

durka · Sep 3, 2016

Hmm, this doesn't seem to be working. I tried it on my most-recursing macro, brainmunch. Having built master and this branch:

[root@li1424-173 brainmunch]# time ../rust/stackless/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -O -Z time-passes src/main.rs | grep expansion
time: 8.082; rss: 68MB  expansion

real    0m9.118s
user    0m8.993s
sys     0m0.100s
[root@li1424-173 brainmunch]# time ../rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -o master -O -Z time-passes src/main.rs | grep expansion
time: 8.501; rss: 67MB  expansion

real    0m9.575s
user    0m9.450s
sys     0m0.103s

This branch is first, then master. I would have expected a more impressive time difference and a smaller memory footprint if the expansion is really stackless. Furthermore, I removed #![recursion_limit="5000"] from main.rs and both builds immediately bailed having reached the recursion limit.

eddyb · Sep 3, 2016

@durka If you don't get a stack overflow with a high recursion limit before this PR, then there's not much to see.
You couldn't notice a large memory or time difference, all that's happening AFAICT is that very deep macro recursion would not cause stack overflows anymore because the real stack isn't used.

jseyfried · Sep 3, 2016

@durka This PR doesn't change recursion_limit checking -- if macro expansion is more recursive than the recursion limit, it is still an error.

It would probably be a good idea to raise the default recursion limit or even remove it altogether, but that's for a separate PR. Note that if we remove the recursion limit altogether, I think we should limit the total memory usage or total number of expansions so that if there really is unbounded recursion, the user will still get a nice error message (not a hang or an out of memory error). I'd be happy to help if you want to do this.

To test this branch, you can set the recursion limit to something that can't be reached in a reasonable amount of time and space, like 2^32.

This PR doesn't improve performance (I actually measured ~5% regression, but that will be made up by a later PR) nor does it improve memory usage -- after this PR, the heap will have all the information that used to be on the stack and more.

durka · Sep 3, 2016

Ah I see, I got too excited.

durka · Sep 3, 2016

Further investigation: I lengthened the test case to stack overflow on master after 1m27s, and it finished compiling on this branch after 5 minutes. Intentionally lowering the recursion limit produces comparable time-to-error for both branches.

bors · Sep 3, 2016

☔️ The latest upstream changes (presumably #35957) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · Sep 5, 2016

nrc reviewed Sep 5, 2016

View changes

nrc · Sep 5, 2016

src/libsyntax/ext/expand.rs

+                    syntax_pos::DUMMY_SP => PathBuf::new(),
+                    _ => PathBuf::from(fld.cx.parse_sess.codemap().span_to_filename(inner)),
+                };
+                fld.cx.directory.pop();


This doesn't seem right - in the dummy span case you're popping an empty PathBuf

nrc · Sep 5, 2016

nrc reviewed Sep 5, 2016

View changes

nrc · Sep 5, 2016

src/libsyntax/ext/expand.rs

-            if valid_ident {
-                fld.cx.mod_push(item.ident);
+            let directory = fld.cx.directory.clone();
+            if item.span.contains(inner) {


This logic could do with a comment

bors · Sep 5, 2016

☔️ The latest upstream changes (presumably #36240) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · Sep 6, 2016

r=me with that boilerplate factored away

jseyfried · Sep 6, 2016

@bors r=nrc

bors · Sep 6, 2016

📌 Commit 400a4f2 has been approved by nrc

bors · Sep 6, 2016

⌛️ Testing commit 400a4f2 with merge d75c64d...

bors · Sep 6, 2016

💔 Test failed - auto-win-msvc-64-cargotest

jseyfried · Sep 6, 2016

@bors r=nrc

bors · Sep 6, 2016

📌 Commit dc01d96 has been approved by nrc

jseyfried · Sep 7, 2016

@bors r=nrc

bors · Sep 7, 2016

📌 Commit 9ac91fa has been approved by nrc

bors · Sep 8, 2016

⌛️ Testing commit 9ac91fa with merge 4a26286...

bors · Sep 8, 2016

jseyfried · Sep 13, 2016

@DanielJCampbell This refactoring might help with expansion visualization / stepwise expansion -- feel free to ping me on IRC if you'd like to discuss.

DanielJCampbell · Sep 14, 2016

Yeah, I will do that - I have a working solution but it's based on Rust from 6 weeks ago, looks like plenty has changed since then.

rust-highfive assigned nrc Sep 2, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch 2 times, most recently from 6497f39 to d0695b0 Sep 2, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch 3 times, most recently from a9fa986 to 99c896f Sep 4, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from 3f9b910 to dc01d96 Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from dc01d96 to 323ee43 Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from 323ee43 to 9ac91fa Sep 7, 2016

bors merged commit 9ac91fa into rust-lang:master Sep 8, 2016
2 checks passed

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

homu Test successful
Details

bluss added the relnotes label Sep 8, 2016

jseyfried deleted the jseyfried:stackless_expansion branch Sep 8, 2016

This was referenced Sep 21, 2016

Merged

macros: allow attribute invocations at the crate root #36618

Closed

Module level attributes are being parsed before plugins are registered #36617

Mark-Simulacrum referenced this pull request Nov 16, 2016
Merged
limit the length of types in monomorphization #37789

jseyfried referenced this pull request Feb 7, 2017
Closed
Tracking issue for "macro naming and modularisation" (RFC #1561) #35896

jseyfried referenced this pull request Jul 26, 2017
Closed
My code fails to build with „unexpected end of macro invocation“ since nightly 2017-07-02 #43057

rust-lang/rust

Join GitHub today

macros: stackless expansion #36214

Conversation

jseyfried commented Sep 2, 2016 • edited

rust-highfive assigned nrc Sep 2, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch 2 times, most recently from 6497f39 to d0695b0 Sep 2, 2016

This comment has been minimized.

durka commented Sep 3, 2016

This comment has been minimized.

eddyb commented Sep 3, 2016

This comment has been minimized.

jseyfried commented Sep 3, 2016 • edited

This comment has been minimized.

durka commented Sep 3, 2016

This comment has been minimized.

durka commented Sep 3, 2016 • edited

This comment has been minimized.

bors commented Sep 3, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch 3 times, most recently from a9fa986 to 99c896f Sep 4, 2016

nrc reviewed Sep 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

nrc reviewed Sep 5, 2016 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

bors commented Sep 5, 2016

jseyfried added some commits Aug 31, 2016

This comment has been minimized.

nrc commented Sep 6, 2016

This comment has been minimized.

jseyfried commented Sep 6, 2016

This comment has been minimized.

bors commented Sep 6, 2016

This comment has been minimized.

bors commented Sep 6, 2016

bors added a commit that referenced this pull request Sep 6, 2016

This comment has been minimized.

bors commented Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from 3f9b910 to dc01d96 Sep 6, 2016

This comment has been minimized.

jseyfried commented Sep 6, 2016

This comment has been minimized.

bors commented Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from dc01d96 to 323ee43 Sep 6, 2016

jseyfried added some commits Sep 2, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from 323ee43 to 9ac91fa Sep 7, 2016

This comment has been minimized.

jseyfried commented Sep 7, 2016

This comment has been minimized.

bors commented Sep 7, 2016

This comment has been minimized.

bors commented Sep 8, 2016

bors added a commit that referenced this pull request Sep 8, 2016

This comment has been minimized.

bors commented Sep 8, 2016

Hide details View details bors merged commit 9ac91fa into rust-lang:master Sep 8, 2016 2 checks passed

2 checks passed

bluss added the relnotes label Sep 8, 2016

jseyfried deleted the jseyfried:stackless_expansion branch Sep 8, 2016

This comment has been minimized.

jseyfried commented Sep 13, 2016

This comment has been minimized.

DanielJCampbell commented Sep 14, 2016

This was referenced Sep 21, 2016

Mark-Simulacrum referenced this pull request Nov 16, 2016

jseyfried referenced this pull request Feb 7, 2017

jseyfried referenced this pull request Jul 26, 2017

jseyfried commented Sep 2, 2016 •

edited

jseyfried force-pushed the jseyfried:stackless_expansion branch 2 times, most recently from `6497f39` to `d0695b0` Sep 2, 2016

jseyfried commented Sep 3, 2016 •

edited

durka commented Sep 3, 2016 •

edited

jseyfried force-pushed the jseyfried:stackless_expansion branch 3 times, most recently from `a9fa986` to `99c896f` Sep 4, 2016

nrc reviewed Sep 5, 2016

View changes

nrc reviewed Sep 5, 2016

View changes

jseyfried force-pushed the jseyfried:stackless_expansion branch from `3f9b910` to `dc01d96` Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from `dc01d96` to `323ee43` Sep 6, 2016

jseyfried force-pushed the jseyfried:stackless_expansion branch from `323ee43` to `9ac91fa` Sep 7, 2016

bors merged commit `9ac91fa` into rust-lang:master Sep 8, 2016
2 checks passed