Rewrite the parser to be fully incremental

[SimonSapin]

(Commits before December come from https://github.com/SimonSapin/css-parsing-tests , imported with git subtree.)

Previously, the tokenizer was an iterator of "component values", where a component value is either a token or a function/block that contains a Vec of nested component values. This meant that parsing code (e.g. for a property value in Servo) could not get nesting wrong, but the tokenizer would not yield anything before the end of a top-level block.

This makes the tokenizer fully incremental: it yields tokens is it finds them, including }, ], and ) tokens that indicate the end of a block or function.

On top of that, Parser (which is what every parsing code should use) does some tricks to deal with nesting without but still be incremental. See src/parser.rs.

Token::ParenthesisBlock, Token::SquareBracketBlock, Token::CurlyBracketBlock, and Token::Function now indicate the start of a block/function, but do not contain the contents/arguments. When getting one of them from Parser::next or Parser::next_including_whitespace, the user can call Parser::parse_nested_block to get a new Parser for the contents/arguments that borrows the first one mutably (so that you have to drop it before using the first one again.) The inner parser "ends" at the end of the corresponding block or function. Whether or not the inner parser is consumed to its end, or is even created, the following call to Parser::next or Parser::next_including_whitespace on the "parent" parser will skip as necessary and continue after the end of the the block/function.

This makes it easier for users to write correct code, but it comes at a cost of complexity in cssparser itself. The code for dealing with nested parsers can be subtly wrong. For example, parser.rs has code like this:

if let Some(block_type) = self.state.at_start_of.take() {
    consume_until_end_of_block(block_type, self.tokenizer())
}

When first writing it, I had forgotten .take(). This compiled without error and of course gave completely incorrect parsing results.

An alternative is to make Parser much much simpler: Parser::next would do little more than call the tokenizer, and users would have to deal with block nesting themselves. (There would be helpers like Parser::skip_until_end_of(BlockType) to deal with error cases.) Users would have to be more careful about always calling Parser::push_back on unexpected tokens (in case it’s an end-of-block token), but they really should be doing that anyway. This is what Gecko does.

So, are "nested" parsers a good idea?

[SimonSapin]

The CSS Syntax spec was changed to enable CSS Variables / custom properties named e.g. --foo.

[SimonSapin]

f? @pcwalton and anyone with opinions on writing parsers.