Make `Tokenizer` public

[xojoc]

[SimonSapin]

Why? What's missing in Parser?

(In general, when requesting a change, it's helpful to explain the motivation in the request.)

[xojoc]

Parser.next skips block tokens and I couln't find a way to parse them. I tried to use parse_nested_block but I just get a lot of lifetime errors.
If this change is inaceptable, could you please give me an example of how to parse blocks (and yes I've already read the Servo code.)

[SimonSapin]

It’s not necessarily unacceptable but it should be justified and this doesn’t seem like a strong enough justification.

The intended usage is that when Parser::next returns a block or function, you can parse its content with Parser::parse_nested_block and a closure that takes a new Parser object as a parameter. The closure should use that object, not the one parse_nested_block was called on. The inner parser will "end" (next will return Err(())) at the end of the block or function. Examples:

https://github.com/servo/rust-cssparser/blob/72cf5e4ea8/src/rules_and_declarations.rs#L389
https://github.com/servo/rust-cssparser/blob/72cf5e4ea8/src/color.rs#L68-L70

This takes care of block nesting (and spec-compliant error recovery) for you. When using the tokenizer directly, you’d have to keep track yourself of a stack of opened blocks to find the matching closing token.

I’d consider making the tokenizer public if there is a use case for opting out completely of this block nesting handling.

[SimonSapin]

@xojoc, does my comment above help?

[xojoc]

Yes, thank you :) I just needed to get more familiar with Rust.