CSS Processor bad-string-token should report null value

[sirreal]

In CSS tokenization, some tokens have associated data (values, units, integer/number flags…). The bad-string-token has no associated data and WordPress\DataLiberation\CSS\CSSProcessor::get_token_value() should return null on bad string tokens.

It currently returns the part of the string up to the breaking codepoint. For example:

$p = WordPress\DataLiberation\CSS\CSSProcessor::create( "'str\ning'" );
$p->next_token();
assert( $p->get_token_type() === WordPress\DataLiberation\CSS\CSSProcessor::TOKEN_BAD_STRING );
assert( $p->get_token_value() === null, 'Expected null token value, got ' . var_export( $p->get_token_value(), true ) );

Will throw AssertionError: Expected null token value, got 'str'.

There's a relevant note in the parsing section about the preserved tokens:

Note: The tokens <}-token>s, <)-token>s, <]-token>, <bad-string-token>, and <bad-url-token> are always parse errors, but they are preserved in the token stream by this specification to allow other specs, such as Media Queries, to define more fine-grained error-handling than just dropping an entire declaration or block.

---

Note that bad-url-token returns null token value, as expected.

[sirreal]

@adamziel Do you know whether this was intentional? As far as I can tell, <bad-string-token> should not have an associated value.