Use syn and quote crates for the `match_token!` macro… #217

SimonSapin · 2016-10-03T16:46:38Z

… instead of the parser and quasi-quoting from Rust’s (unstable) libsyntax.

This has significantly worse diagnostics when encountering unexpected syntax (e.g. no indication of which line has the offending code) but this removes all usage of unstable compiler internals that constantly need to be fixed when updating the compiler.

Fixes #216.

r? @nox

This change is

SimonSapin · 2016-10-03T16:49:11Z

Cargo.toml

@@ -33,7 +33,7 @@ harness = false
 [features]
 unstable = ["tendril/unstable", "string_cache/unstable"]
 heap_size = ["heapsize", "heapsize_plugin"]
-codegen = ["html5ever_macros"]
+codegen = []


This feature is now unused, but removing it would be a breaking change. So let’s keep it until we have some other reason to make a breaking change.

Can you add a comment with that in the file so we don't accidentally bump the version without removing the feature?

SimonSapin · 2016-10-03T16:49:11Z

src/tree_builder/rules.rs

@@ -7,7 +7,7 @@
 // option. This file may not be copied, modified, or distributed
 // except according to those terms.

-//! The tree builder rules, as a single, enormous nested match expression.


include! doesn’t like top-level attributes.

SimonSapin · 2016-10-03T16:49:11Z

src/tree_builder/rules.rs

@@ -770,7 +770,7 @@ impl<Handle, Sink> TreeBuilderStep
                    Done
                }

-                // FIXME: This should be unreachable, but match_token! requires a
+                // FIXME: This should be unreachable, but match_token requires a


Yes, the parser is that stupid. It would find match_token! in a comment and try to parse it as a macro invocation.

Note: this comment referred to my hand-written parser. syn doesn’t have this problem.

emilio

Are there any tests for this macro?

emilio · 2016-10-04T11:50:24Z

macros/match_token.rs

+}
+
+pub fn expand_match_tokens(from: &Path, to: &Path) {
+//    use std::fmt::Write;


Do you plan to keep these comments for any specific reasons? Or is it fine to remove them?

They were temporary, for debugging. I forgot to remove them before committing. Removed now.

SimonSapin · 2016-10-04T11:53:58Z

Are there any tests for this macro?

Not directly, but there are many tree builder tests that exercise the generated code.

Ygg01 · 2016-10-05T20:10:39Z

How easy will this be the modify?

nox · 2016-10-11T09:41:30Z

syn 0.9.0 is out with many improvements, could the hand-written parser use it? I'm really wary of merging something that can also end up transforming comments.

Ygg01 · 2016-10-11T10:23:30Z

@nox: syn is for derive macros, only, as far as I can tell. See discussion here: #216

nox · 2016-10-11T10:26:25Z

@Ygg01 https://github.com/dtolnay/syn/releases/tag/0.9.0

Ygg01 · 2016-10-11T11:00:54Z

@nox: I'm a newbie when it comes to procedural macros, but those notes don't show me anything that would make full procedural macros parsable by syn? I hate to be an energy vampire, so could you just explain what you meant?

Did you mean we could parse tags as:

match mode {
    InitialMode => 
    #[derive(tags)]
    match token {
         CharacterTokens(NotSplit, text) => SplitWhitespace(text),
         ...
         token => {
                    if !self.opts.iframe_srcdoc {
                        self.unexpected(&token);
                        self.set_quirks_mode(Quirks);
                    }
                    Reprocess(BeforeHtml, token)
          }
    }
}

bors-servo · 2016-10-17T15:17:18Z

☔ The latest upstream changes (presumably #218) made this pull request unmergeable. Please resolve the merge conflicts.

SimonSapin · 2016-10-17T20:05:40Z

Alright! I pushed a rewrite that uses the syn and quote crates, the provide real parser and quasi-quoting.

r? @nox

I’ll rebase and fix conflicts tomorrow.


        Use a hand-written parser a string formatting for the `match_token!` …

…macro… … instead of the parser and quasi-quoting from Rust’s (unstable) libsyntax. This has significantly worse diagnostics when encountering unexpected syntax (e.g. no indication of which line has the offending code) but this removes all usage of unstable compiler internals that constantly need to be fixed when updating the compiler. Fixes #216.


        Use an actual Rust parser for the match_token! macro.

nox · 2016-10-18T11:42:24Z

@bors-servo r+

bors-servo · 2016-10-18T11:42:25Z

📌 Commit dd9f2cc has been approved by nox

bors-servo · 2016-10-18T11:42:26Z

⌛ Testing commit dd9f2cc with merge b8354e4...


        Auto merge of #217 - servo:diy-macros, r=nox

@nox

Use syn and quote crates for the `match_token!` macro… … instead of the parser and quasi-quoting from Rust’s (unstable) libsyntax. This has significantly worse diagnostics when encountering unexpected syntax (e.g. no indication of which line has the offending code) but this removes all usage of unstable compiler internals that constantly need to be fixed when updating the compiler. Fixes #216. r? @nox  --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/html5ever/217)

bors-servo · 2016-10-18T11:50:29Z

☀️ Test successful - status-travis


        Add parse_token_trees.

Token trees are much simpler than an AST. This makes then easier to work with when they are sufficent for the task at hand, such as for example [expanding a procedural macro]( servo/html5ever#217).


        Add parse_token_trees.

Token trees are much simpler than an AST. This makes them easier to work with when they are sufficient for the task at hand, such as for example [expanding a procedural macro]( servo/html5ever#217).


        Add parse_token_trees.

Token trees are much simpler than an AST. This makes them easier to work with when they are sufficient for the task at hand, such as for example [expanding a procedural macro]( servo/html5ever#217).

SimonSapin reviewed Oct 3, 2016

View changes

SimonSapin mentioned this pull request Oct 3, 2016

Macros to generate tree building rules don't work in current Rust #216

Closed

jdm mentioned this pull request Oct 3, 2016

Fixed compilation errors in macros. Whitespace handling in Tree Builder. #205

Closed

SimonSapin force-pushed the diy-macros branch from f70a0bf to 7a86f58 Oct 4, 2016

emilio reviewed Oct 4, 2016

View changes

SimonSapin force-pushed the diy-macros branch from 7a86f58 to b178d28 Oct 4, 2016

SimonSapin changed the title ~~Use a hand-written parser a string formatting for the `match_token!` macro…~~ Use syn and quote crates for the `match_token!` macro… Oct 17, 2016

SimonSapin force-pushed the diy-macros branch from 548a0d5 to 63bf17f Oct 18, 2016

Use an actual Rust parser for the match_token! macro.

Loading status checks…

dd9f2cc

SimonSapin force-pushed the diy-macros branch from 63bf17f to dd9f2cc Oct 18, 2016

SimonSapin deleted the diy-macros branch Oct 18, 2016

SimonSapin mentioned this pull request Oct 20, 2016

Add parse_token_trees. dtolnay/syn#52

Merged

servo / html5ever

Use syn and quote crates for the `match_token!` macro… #217

Use syn and quote crates for the `match_token!` macro… #217

SimonSapin commented Oct 3, 2016 •

edited by larsbergstrom

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

emilio left a comment

This comment has been minimized.

This comment has been minimized.

SimonSapin commented Oct 4, 2016

Ygg01 commented Oct 5, 2016

nox commented Oct 11, 2016

Ygg01 commented Oct 11, 2016

nox commented Oct 11, 2016

Ygg01 commented Oct 11, 2016

bors-servo commented Oct 17, 2016

SimonSapin commented Oct 17, 2016

nox commented Oct 18, 2016

bors-servo commented Oct 18, 2016

bors-servo commented Oct 18, 2016

bors-servo commented Oct 18, 2016

servo / html5ever

Join GitHub today

Use syn and quote crates for the `match_token!` macro… #217

Use syn and quote crates for the `match_token!` macro… #217

Conversation

SimonSapin commented Oct 3, 2016 • edited by larsbergstrom

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

emilio left a comment

This comment has been minimized.

This comment has been minimized.

SimonSapin commented Oct 4, 2016

Ygg01 commented Oct 5, 2016

nox commented Oct 11, 2016

Ygg01 commented Oct 11, 2016

nox commented Oct 11, 2016

Ygg01 commented Oct 11, 2016

bors-servo commented Oct 17, 2016

SimonSapin commented Oct 17, 2016

nox commented Oct 18, 2016

bors-servo commented Oct 18, 2016

bors-servo commented Oct 18, 2016

bors-servo commented Oct 18, 2016

SimonSapin commented Oct 3, 2016 •

edited by larsbergstrom