macros: stackless expansion #36214

jseyfried · 2016-09-02T11:30:02Z

After this PR, macro expansion cannot overflow the stack unless the expanded crate is too deep to fold.
Everything but the stackless placeholder expansion commit is also groundwork for macro modularization.

cc #35896
r? @nrc or @eddyb

durka · 2016-09-03T05:05:32Z

Hmm, this doesn't seem to be working. I tried it on my most-recursing macro, brainmunch. Having built master and this branch:

[root@li1424-173 brainmunch]# time ../rust/stackless/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -O -Z time-passes src/main.rs | grep expansion
time: 8.082; rss: 68MB  expansion

real    0m9.118s
user    0m8.993s
sys     0m0.100s
[root@li1424-173 brainmunch]# time ../rust/build/x86_64-unknown-linux-gnu/stage1/bin/rustc -o master -O -Z time-passes src/main.rs | grep expansion
time: 8.501; rss: 67MB  expansion

real    0m9.575s
user    0m9.450s
sys     0m0.103s

This branch is first, then master. I would have expected a more impressive time difference and a smaller memory footprint if the expansion is really stackless. Furthermore, I removed #![recursion_limit="5000"] from main.rs and both builds immediately bailed having reached the recursion limit.

eddyb · 2016-09-03T05:17:34Z

@durka If you don't get a stack overflow with a high recursion limit before this PR, then there's not much to see.
You couldn't notice a large memory or time difference, all that's happening AFAICT is that very deep macro recursion would not cause stack overflows anymore because the real stack isn't used.

jseyfried · 2016-09-03T05:18:46Z

@durka This PR doesn't change recursion_limit checking -- if macro expansion is more recursive than the recursion limit, it is still an error.

It would probably be a good idea to raise the default recursion limit or even remove it altogether, but that's for a separate PR. Note that if we remove the recursion limit altogether, I think we should limit the total memory usage or total number of expansions so that if there really is unbounded recursion, the user will still get a nice error message (not a hang or an out of memory error). I'd be happy to help if you want to do this.

To test this branch, you can set the recursion limit to something that can't be reached in a reasonable amount of time and space, like 2^32.

This PR doesn't improve performance (I actually measured ~5% regression, but that will be made up by a later PR) nor does it improve memory usage -- after this PR, the heap will have all the information that used to be on the stack and more.

durka · 2016-09-03T05:23:51Z

Ah I see, I got too excited.

durka · 2016-09-03T05:57:37Z

Further investigation: I lengthened the test case to stack overflow on master after 1m27s, and it finished compiling on this branch after 5 minutes. Intentionally lowering the recursion limit produces comparable time-to-error for both branches.

bors · 2016-09-03T10:23:16Z

☔ The latest upstream changes (presumably #35957) made this pull request unmergeable. Please resolve the merge conflicts.

nrc · 2016-09-05T03:48:11Z

src/libsyntax/ext/expand.rs

+                    syntax_pos::DUMMY_SP => PathBuf::new(),
+                    _ => PathBuf::from(fld.cx.parse_sess.codemap().span_to_filename(inner)),
+                };
+                fld.cx.directory.pop();


This doesn't seem right - in the dummy span case you're popping an empty PathBuf

bors · 2016-09-05T04:51:57Z

☔ The latest upstream changes (presumably #36240) made this pull request unmergeable. Please resolve the merge conflicts.

…ack` with a single field `directory: PathBuf`.

…tures`.

…xpander`.

nrc · 2016-09-06T01:48:29Z

r=me with that boilerplate factored away

jseyfried · 2016-09-06T04:12:55Z

@bors r=nrc

bors · 2016-09-06T04:12:56Z

📌 Commit 400a4f2 has been approved by nrc

bors · 2016-09-06T04:14:51Z

⌛ Testing commit 400a4f2 with merge d75c64d...

@nrc

macros: stackless expansion After this PR, macro expansion cannot overflow the stack unless the expanded crate is too deep to fold. Everything but the stackless placeholder expansion commit is also groundwork for macro modularization. r? @nrc or @eddyb

bors · 2016-09-06T05:32:35Z

💔 Test failed - auto-win-msvc-64-cargotest

jseyfried · 2016-09-06T06:09:12Z

@bors r=nrc

bors · 2016-09-06T06:09:13Z

📌 Commit dc01d96 has been approved by nrc

jseyfried · 2016-09-07T22:55:33Z

@bors r=nrc

bors · 2016-09-07T22:55:34Z

📌 Commit 9ac91fa has been approved by nrc

bors · 2016-09-08T02:02:52Z

⌛ Testing commit 9ac91fa with merge 4a26286...

@nrc

macros: stackless expansion After this PR, macro expansion cannot overflow the stack unless the expanded crate is too deep to fold. Everything but the stackless placeholder expansion commit is also groundwork for macro modularization. r? @nrc or @eddyb

bors · 2016-09-08T05:30:46Z

jseyfried · 2016-09-13T19:46:02Z

@DanielJCampbell This refactoring might help with expansion visualization / stepwise expansion -- feel free to ping me on IRC if you'd like to discuss.

DanielJCampbell · 2016-09-14T08:24:54Z

Yeah, I will do that - I have a working solution but it's based on Rust from 6 weeks ago, looks like plenty has changed since then.

rust-highfive assigned nrc Sep 2, 2016

jseyfried force-pushed the stackless_expansion branch 2 times, most recently from 6497f39 to d0695b0 Compare September 2, 2016 13:38

jseyfried force-pushed the stackless_expansion branch 3 times, most recently from a9fa986 to 99c896f Compare September 5, 2016 01:23

nrc reviewed Sep 5, 2016
View reviewed changes

jseyfried added 15 commits September 5, 2016 04:52

In Parser and ExtCtxt, replace fields filename and `mod_path_st…

e1e5c14

…ack` with a single field `directory: PathBuf`.

Improve expand_type.

234d68b

Remove syntax::config::strip_unconfigured, add `syntax::config::fea…

9b3bc7a

…tures`.

Add Invocation and Expansion, remove MacroGenerable.

de2e678

Refactor with_exts_frame from a macro to a function.

3cba93f

Generalize Invocation to include modifiers/decorators.

fca80c9

Refactor away expand_item.

8be8cf8

Refactor out expand_item (with better semantics than before).

2a83574

Clean up module processing.

503a10b

Refactor expand_* into expander.fold_*.

4ed2c0e

Refactor SyntaxEnv.

79fa9eb

Refactor expand_invoc(.., fld) -> self.expand_invoc(..).

7a3ae57

Add module ext::placeholders with placeholder() and `PlaceholderE…

c07ff8d

…xpander`.

Implement stackless expansion.

d986bbe

Load macros from extern crates in the InvocationCollector fold.

2c88b4b

jseyfried force-pushed the stackless_expansion branch from 3f9b910 to dc01d96 Compare September 6, 2016 06:08

jseyfried force-pushed the stackless_expansion branch from dc01d96 to 323ee43 Compare September 6, 2016 08:15

jseyfried added 4 commits September 7, 2016 22:12

Refactor code out of the folder implementation for StripUnconfigured.

3af0c65

Strip unconfigured nodes in the InvocationCollector fold.

d76bf3e

Implement stackless placeholder expansion.

2d75904

Improve directory computation during invocation collection.

9ac91fa

jseyfried force-pushed the stackless_expansion branch from 323ee43 to 9ac91fa Compare September 7, 2016 22:49

bors merged commit 9ac91fa into rust-lang:master Sep 8, 2016

bluss added the relnotes Marks issues that should be documented in the release notes of the next release. label Sep 8, 2016

jseyfried deleted the stackless_expansion branch September 8, 2016 20:27

This was referenced Sep 21, 2016

macros: allow attribute invocations at the crate root #36618

Merged

Module level attributes are being parsed before plugins are registered #36617

Closed

Mark-Simulacrum mentioned this pull request Nov 16, 2016

limit the length of types in monomorphization #37789

Merged

jseyfried mentioned this pull request Feb 7, 2017

Tracking issue for "macro naming and modularisation" (RFC #1561) #35896

Closed

jseyfried mentioned this pull request Jul 26, 2017

My code fails to build with „unexpected end of macro invocation“ since nightly 2017-07-02 #43057

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macros: stackless expansion #36214

macros: stackless expansion #36214

jseyfried commented Sep 2, 2016 •

edited

Loading

durka commented Sep 3, 2016

eddyb commented Sep 3, 2016

jseyfried commented Sep 3, 2016 •

edited

Loading

durka commented Sep 3, 2016

durka commented Sep 3, 2016 •

edited

Loading

bors commented Sep 3, 2016

nrc Sep 5, 2016

jseyfried Sep 5, 2016

bors commented Sep 5, 2016

nrc commented Sep 6, 2016

jseyfried commented Sep 6, 2016

bors commented Sep 6, 2016

bors commented Sep 6, 2016

bors commented Sep 6, 2016

jseyfried commented Sep 6, 2016

bors commented Sep 6, 2016

jseyfried commented Sep 7, 2016

bors commented Sep 7, 2016

bors commented Sep 8, 2016

bors commented Sep 8, 2016

jseyfried commented Sep 13, 2016

DanielJCampbell commented Sep 14, 2016

macros: stackless expansion #36214

macros: stackless expansion #36214

Conversation

jseyfried commented Sep 2, 2016 • edited Loading

durka commented Sep 3, 2016

eddyb commented Sep 3, 2016

jseyfried commented Sep 3, 2016 • edited Loading

durka commented Sep 3, 2016

durka commented Sep 3, 2016 • edited Loading

bors commented Sep 3, 2016

nrc Sep 5, 2016

Choose a reason for hiding this comment

jseyfried Sep 5, 2016

Choose a reason for hiding this comment

bors commented Sep 5, 2016

nrc commented Sep 6, 2016

jseyfried commented Sep 6, 2016

bors commented Sep 6, 2016

bors commented Sep 6, 2016

bors commented Sep 6, 2016

jseyfried commented Sep 6, 2016

bors commented Sep 6, 2016

jseyfried commented Sep 7, 2016

bors commented Sep 7, 2016

bors commented Sep 8, 2016

bors commented Sep 8, 2016

jseyfried commented Sep 13, 2016

DanielJCampbell commented Sep 14, 2016

jseyfried commented Sep 2, 2016 •

edited

Loading

jseyfried commented Sep 3, 2016 •

edited

Loading

durka commented Sep 3, 2016 •

edited

Loading