CSSTokenizer

[adamziel]

Overview

Ships a CSSTokenizer class that tokenizes CSS syntax. It follows the CSS Syntax Level 3 spec, giving Data Liberation tooling a reliable foundation for discovering and rewriting URLs without leaning on brittle regular expressions or partial parsers.

This branch is tested using the @rmenke/css-tokenizer-tests test corpus and an extra set of tests designed to cover nuances.

Design choices:

On-the-fly normalization

The CSS Spec requires the following normalization step:

Replace any U+000D CARRIAGE RETURN (CR) code points, U+000C FORM FEED (FF)
code points, or pairs of U+000D CARRIAGE RETURN (CR) followed by U+000A LINE
FEED (LF) in input by a single U+000A LINE FEED (LF) code point.
Replace any U+0000 NULL or surrogate code points in input with U+FFFD REPLACEMENT
CHARACTER (�).

This processor delays normalization as much as possible, rather than preprocessing
the entire input upfront. This avoids the upfront allocation cost for clean CSS
and preserves original byte positions for accurate raw token extraction. A part
of the normalization is performed on-the-fly as the tokens are consumed. The rest
of it is done once the token value is requested.

No EOF token

The EOF token is a CSS parsing concept, not CSS tokenization concept. Therefore,
this processor does not produce it.

UTF-8 handling

Only UTF-8 strings are supported. In case an invalid UTF-8 sequence is encountered,
it is replaced with a U+FFFD REPLACEMENT CHARACTER (�) using the maximal subpart
approach described in https://www.unicode.org/versions/Unicode9.0.0/ch03.pdf,
section 3.9 Best Practices for Using U+FFFD.

Usage

The next_token() method is the main entry point for tokenizing a CSS string.
It will consume the next token from the input stream and return true if a token
was found. Otherwise, it will return false:

$css = 'width: 10px;';
$processor = CSSProcessor::create( $css );
while ( $processor->next_token() ) {
    echo $processor->get_normalized_token();
}
// Outputs:
// width: 10px;

[adamziel]

I've applied your feedback @dmsnell, thank you! This PR seems in a pretty good place now. I'll draft a second one with set_token_value() before merging this, just to make sure we have the right APIs.

[adamziel]

I think we're good API-wise. Even if not, we can always adjust since it's the php-toolkit repo.

[adamziel]

CC @sirreal – you may like this