No more syntex #940

fitzgen · 2017-08-31T23:12:09Z

There are a few commits in this PR, but the big one is the commit that goes syntex -> quote for code generation. I've included a copy of its commit message below.

I tried to verify that it works with the stylo build still, but there were some issues, and then I checked with master, and that wasn't working either. So now I'm C-Reducing the failures on master and figured that this is at least ready for feedback in the meantime.

r? @emilio

The syntex crate is unmaintained. It is slow to build, and additionally it requires that we pre-process src/codegen/mod.rs before we build the bindgen crate.

The quote crate provides similar quasi-quoting functionality, is maintained, and builds faster. It doesn't have a typed API or builders, however; it only deals with tokens.

Before this commit:

$ cargo clean; cargo build
<snip>
    Finished dev [unoptimized + debuginfo] target(s) in 98.75 secs

After this commit:

$ cargo clean; cargo build
<snip>
    Finished dev [unoptimized + debuginfo] target(s) in 46.26 secs

Build time is cut in half! But what about run time?

Before this commit:

Generated Stylo bindings in: Duration { secs: 3, nanos: 521105668 }

After this commit:

Generated Stylo bindings in: Duration { secs: 3, nanos: 548797242 }

So it appears to be about 20ms slower at generating Stylo bindings, but I suspect this is well within the noise.

Finally, this also lets us remove that nasty mem::transmute inside bindgen::ir::BindgenContext::gen that was used for the old syntex context. Now BindgenContext doesn't have a lifetime parameter either. This should make it easier to revisit doing our analyses in parallel with rayon, since that context was one of the things that made it hard for BindgenContext to implement Sync.

Fixes #925

highfive · 2017-08-31T23:12:14Z

Warning

These commits modify unsafe code. Please review it carefully!

emilio

So, looks pretty reasonable. If I understand it well, we're running rustfmt on the bindings now, and we need to have the latest rustfmt-nightly installed locally to run tests so that they pass on CI.

Is that right? If so, I think we should document that.

emilio · 2017-09-04T10:53:44Z

src/codegen/helpers.rs

+    pub fn link_name(name: &str) -> quote::Tokens {
+        // This is ridiculous, but rustfmt doesn't seem to be formatting
+        // `link_name` attributes very well, so we jump through these formatting
+        // hoops to make it work.


Can you file an issue upstream and reference it here?

I've filed a handful of bugs upstream, and they've all been fixed now, so this comment might actually be out of date. I'll double check.

emilio · 2017-09-04T10:54:49Z

src/codegen/helpers.rs

-                .lit()
-                .byte_str(string),
-        )
+        let b = quote::ByteStr(&string);


fitzgen · 2017-09-05T16:00:19Z

If I understand it well, we're running rustfmt on the bindings now, and we need to have the latest rustfmt-nightly installed locally to run tests so that they pass on CI.

Is that right? If so, I think we should document that.

Yes, it will be automatically installed by the Once callback in the rustfmt function in the tests file.

I'll document it in CONTRIBUTING.md as well.

bors-servo · 2017-09-05T17:06:44Z

☔ The latest upstream changes (presumably #926) made this pull request unmergeable. Please resolve the merge conflicts.

fitzgen · 2017-09-05T17:25:14Z

Added note about rustfmt in CONTRIBUTING.md
Removed the funky formatting for link_name -- new rustfmt has fixed the bug we were running into

re-r? @emilio

fitzgen · 2017-09-05T17:25:33Z

Oops, just saw I need to rebase now.

fitzgen · 2017-09-05T17:36:20Z

Ok, rebased.

emilio

Looks ok, but I don't feel like updating nightly / rustfmt on behalf of the user is right... :/

emilio · 2017-09-06T14:36:24Z

CONTRIBUTING.md

+
+```
+$ rustup update nightly
+$ rustup run nightly cargo install -f rustfmt-nightly


What happens when rustfmt changes the default formatting? All our CI breaks until we update them? That seems not great :(

Nope, because both the generated bindings and the expectations are run through rustfmt before being compared.

emilio · 2017-09-06T14:37:25Z

tests/tests.rs

+        // Because `rustfmt` needs to match its exact nightly version, we update
+        // both at the same time.
+
+        let status = process::Command::new("rustup")


It definitely feels wrong to update the nightly in behalf of the user.

Unfortunately, this is the only way for the latest rustfmt to work :-/

emilio · 2017-09-06T14:37:38Z

tests/tests.rs

+        assert!(status.success(), "should run `rustup update nightly` OK");
+
+        let status = process::Command::new("rustup")
+            .args(&["run", "nightly", "cargo", "install", "-f", "rustfmt-nightly"])


emilio · 2017-09-06T14:39:55Z

(Of course this is just for contributors, but still)

emilio · 2017-09-06T14:45:43Z

Oh, also @glyn reported when building on top of this branch that he got unexpected test expectation changes, so we should probably get that sorted out.

fitzgen · 2017-09-06T18:49:45Z

@glyn, do you have the latest rustfmt installed? Can you give more info about the expectation changes you were seeing?

bors-servo · 2017-09-06T19:41:30Z

☔ The latest upstream changes (presumably #953) made this pull request unmergeable. Please resolve the merge conflicts.

glyn · 2017-09-07T09:43:56Z

@fitzgen Yes, I believe I was using the latest rustfmt. Details in this gist.

fitzgen · 2017-09-07T17:46:11Z

@fitzgen Yes, I believe I was using the latest rustfmt. Details in this gist.

Aha! You need to use rustfmt-nightly, which is where rustfmt development has moved to. The plain rustfmt crate is unmaintained now.

The `syntex` crate is unmaintained. It is slow to build, and additionally it requires that we pre-process `src/codegen/mod.rs` before we build the `bindgen` crate. The `quote` crate provides similar quasi-quoting functionality, is maintained, and builds faster. It doesn't have a typed API or builders, however; it only deals with tokens. Before this commit: ``` $ cargo clean; cargo build <snip> Finished dev [unoptimized + debuginfo] target(s) in 98.75 secs ``` After this commit: ``` $ cargo clean; cargo build <snip> Finished dev [unoptimized + debuginfo] target(s) in 46.26 secs ``` Build time is cut in half! But what about run time? Before this commit: ``` Generated Stylo bindings in: Duration { secs: 3, nanos: 521105668 } ``` After this commit: ``` Generated Stylo bindings in: Duration { secs: 3, nanos: 548797242 } ``` So it appears to be about 20ms slower at generating Stylo bindings, but I suspect this is well within the noise. Finally, this also lets us remove that nasty `mem::transmute` inside `bindgen::ir::BindgenContext::gen` that was used for the old `syntex` context. Now `BindgenContext` doesn't have a lifetime parameter either. This should make it easier to revisit doing our analyses in parallel with `rayon`, since that context was one of the things that made it hard for `BindgenContext` to implement `Sync`. Fixes rust-lang#925

Was previously printing seconds, but claiming it was milliseconds.

;)

The latest `rustfmt` has fixed the formatting bugs we were running into.

fitzgen · 2017-09-07T18:17:13Z

@bors-servo r=emilio

bors-servo · 2017-09-07T18:17:13Z

📌 Commit 46f74c1 has been approved by emilio

bors-servo · 2017-09-07T19:02:10Z

⌛ Testing commit 46f74c1 with merge b13f076...

@emilio

No more syntex There are a few commits in this PR, but the big one is the commit that goes `syntex` -> `quote` for code generation. I've included a copy of its commit message below. I tried to verify that it works with the stylo build still, but there were some issues, and then I checked with master, and that wasn't working either. So now I'm C-Reducing the failures on master and figured that this is at least ready for feedback in the meantime. r? @emilio ---------------------------- The `syntex` crate is unmaintained. It is slow to build, and additionally it requires that we pre-process `src/codegen/mod.rs` before we build the `bindgen` crate. The `quote` crate provides similar quasi-quoting functionality, is maintained, and builds faster. It doesn't have a typed API or builders, however; it only deals with tokens. Before this commit: ``` $ cargo clean; cargo build <snip> Finished dev [unoptimized + debuginfo] target(s) in 98.75 secs ``` After this commit: ``` $ cargo clean; cargo build <snip> Finished dev [unoptimized + debuginfo] target(s) in 46.26 secs ``` Build time is cut in half! But what about run time? Before this commit: ``` Generated Stylo bindings in: Duration { secs: 3, nanos: 521105668 } ``` After this commit: ``` Generated Stylo bindings in: Duration { secs: 3, nanos: 548797242 } ``` So it appears to be about 20ms slower at generating Stylo bindings, but I suspect this is well within the noise. Finally, this also lets us remove that nasty `mem::transmute` inside `bindgen::ir::BindgenContext::gen` that was used for the old `syntex` context. Now `BindgenContext` doesn't have a lifetime parameter either. This should make it easier to revisit doing our analyses in parallel with `rayon`, since that context was one of the things that made it hard for `BindgenContext` to implement `Sync`. Fixes #925

bors-servo · 2017-09-07T19:39:34Z

☀️ Test successful - status-travis
Approved by: emilio
Pushing b13f076 to master...

glyn · 2017-09-08T08:12:58Z

@fitzgen rm ~/.cargo/bin/rustfmt fixed the problem, thanks.

highfive assigned emilio Aug 31, 2017

highfive added the S-awaiting-review label Aug 31, 2017

fitzgen force-pushed the no-syntex branch 2 times, most recently from d5f4753 to d130bf6 Compare September 1, 2017 19:51

emilio reviewed Sep 4, 2017

View reviewed changes

glyn mentioned this pull request Sep 4, 2017

bitfield methods not generated #816

Closed

highfive added the S-needs-rebase label Sep 5, 2017

fitzgen force-pushed the no-syntex branch from 43806e3 to 21f8bf8 Compare September 5, 2017 17:35

emilio reviewed Sep 6, 2017

View reviewed changes

highfive added the S-needs-rebase label Sep 6, 2017

fitzgen added 8 commits September 7, 2017 10:46

Add the TypeKind name to the IR graphviz output

66743be

Run rustfmt on the test actual output and expectations

7346811

Print time phases for the stylo sanity test

4f0aa3e

Fix units and formatting of time phases

6e4383a

Was previously printing seconds, but claiming it was milliseconds.

We can afford the exra cycles to use double precision floats here

a93d0fb

;)

Add a timer for rustfmt'ing the generated bindings

17f13b3

Explicitly install the latest rustfmt in CI

efdb1c5

fitzgen added 4 commits September 7, 2017 10:52

Also print rustfmt's stderr for expectations

cca4037

Note that we need rustfmt to run tests now

231bccb

Use the quote! macro for link_name attributes

04f020b

The latest `rustfmt` has fixed the formatting bugs we were running into.

Just check for rustfmt; don't install it

46f74c1

fitzgen force-pushed the no-syntex branch from 21f8bf8 to 46f74c1 Compare September 7, 2017 18:02

emilio approved these changes Sep 7, 2017

View reviewed changes

highfive added S-awaiting-merge and removed S-awaiting-review S-needs-rebase labels Sep 7, 2017

bors-servo merged commit 46f74c1 into rust-lang:master Sep 7, 2017

highfive removed the S-awaiting-merge label Sep 7, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No more syntex #940

No more syntex #940

fitzgen commented Aug 31, 2017

highfive commented Aug 31, 2017

emilio left a comment

emilio Sep 4, 2017

fitzgen Sep 5, 2017

emilio Sep 4, 2017

fitzgen commented Sep 5, 2017

bors-servo commented Sep 5, 2017

fitzgen commented Sep 5, 2017

fitzgen commented Sep 5, 2017

fitzgen commented Sep 5, 2017

emilio left a comment

emilio Sep 6, 2017

fitzgen Sep 6, 2017

emilio Sep 6, 2017

fitzgen Sep 6, 2017

emilio Sep 6, 2017

emilio commented Sep 6, 2017

emilio commented Sep 6, 2017

fitzgen commented Sep 6, 2017

bors-servo commented Sep 6, 2017

glyn commented Sep 7, 2017

fitzgen commented Sep 7, 2017

fitzgen commented Sep 7, 2017

bors-servo commented Sep 7, 2017

bors-servo commented Sep 7, 2017

bors-servo commented Sep 7, 2017

glyn commented Sep 8, 2017

No more syntex #940

No more syntex #940

Conversation

fitzgen commented Aug 31, 2017

highfive commented Aug 31, 2017

emilio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

fitzgen commented Sep 5, 2017

bors-servo commented Sep 5, 2017

fitzgen commented Sep 5, 2017

fitzgen commented Sep 5, 2017

fitzgen commented Sep 5, 2017

emilio left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

emilio commented Sep 6, 2017

emilio commented Sep 6, 2017

fitzgen commented Sep 6, 2017

bors-servo commented Sep 6, 2017

glyn commented Sep 7, 2017

fitzgen commented Sep 7, 2017

fitzgen commented Sep 7, 2017

bors-servo commented Sep 7, 2017

bors-servo commented Sep 7, 2017

bors-servo commented Sep 7, 2017

glyn commented Sep 8, 2017