Speed up the tokenizer and eliminate double-tokenization at parse boundaries

[bartveneman]

Summary

Three commits, all driven by CPU-profiling real parses of bootstrap.css and tailwind.css
rather than guessing.

Parse+Walk throughput (ops/sec):

File	Before	After	Change
Large	16058	16565	+3.2%
Bootstrap	210	225	+7.1%
Tailwind	16	17	+6.3%

Tokenizer tuning (commits 1-2)

• Reorder the dispatch chain by measured token frequency. Identifiers are the single most common non-whitespace token (~15-21% of all tokens across both files), but the chain checked them 11th - behind CDO/CDC (<!--/-->, a token that occurred zero times in either file; it's legacy HTML-comment compatibility) and the at-keyword/hash checks, both individually rarer than identifiers. Moved the identifier check up front and CDO to dead last. Also consolidated three separate ch === CHAR_HYPHEN checks (CDC, hyphen-led identifier, hyphen-led signed number) into one block sharing a single lookahead read, while preserving exact precedence (CDC must still win for -->, since bare -- is valid identifier-start for custom properties).
• Hoist this.pos (and this._line/this._line_offset where relevant) to local variables inside hot loops, syncing back to the instance once at the end instead of touching this on every character. Applied to every consume_* method that showed measurable self-time in the profiler: consume_ident_or_function (20.1% self-time), consume_number (4.7%), the inline whitespace-skip in next_token_fast, and consume_whitespace; also consume_hash/consume_at_keyword for consistency despite their low frequency.

consume_string, consume_unicode_range, and consume_hex_escape were deliberately left untouched - none showed measurable self-time in the profiler (well under 0.4% combined) and their escape/newline interactions are more tangled, so there's nothing to gain versus real risk to a hand-tuned hot path.

Eliminate double-tokenization at selector/declaration-value boundaries (commit 3)

The parser did a coarse boundary scan by fully tokenizing up to a selector's { or a declaration value's terminator, only to hand that exact span to a dedicated sub-parser (SelectorParser, ValueNodeParser) that re-tokenizes it from scratch to build the real AST. The coarse pass only ever needed to know where the construct ends, not the individual tokens in it.

Replaced both boundary scans with raw character-level scans (Lexer.skip_to_unquoted, Lexer.skip_to_declaration_stop) that skip ordinary content via a single lookup-table check per character, while still routing through consume_string/a shared _skip_comment for correctness on quotes and comments, so escapes, nesting, and on_comment callbacks stay byte-for-byte identical to the old tokenized scan.

Getting this right required matching several non-obvious side effects of the old token-based scan exactly (verified by diffing serialized ASTs between old and new implementations across handwritten edge cases and the full text of bootstrap/tailwind's regular and minified builds - byte identical in both) - see the commit message for the specific quirks (a pre-tokenization side effect on : consumption, a loop-exit edge case, and an empty-value-before-} quirk).

Approaches measured and rejected along the way

Idea	Result
Sticky-regex identifier scanning	35-40% slower than the manual char-loop, even accounting for real identifiers averaging ~11-12 chars
Pre-converting the source to a Uint16Array	No net win - the conversion pass costs as much as the savings from several scans
Pre-scanning for <!-- and gating the CDO check behind a boolean flag	Slightly slower - branch prediction already makes an always-false check nearly free

Verification

• Full test suite (1386 tests) passes unchanged, including the dedicated CDO/CDC precedence test (--> → CDC, not identifier -- + delimiter >) and the custom-property tokenizer test (--custom-prop → single IDENT token).
• tsc --noEmit and oxlint/oxfmt all clean.
• Commit 3 additionally verified via full-file AST comparison (every node's type/start/length/is_important) across bootstrap.css, bootstrap.min.css, tailwind.css, and tailwind.min.css - byte-identical output between old and new implementations.
• Measured via repeated full end-to-end parse benchmarks (bootstrap.css + tailwind.css together): commits 1-2 measured ~5.9% faster combined; commit 3 measured ~7.9% faster than commits 1-2's baseline (the selector-boundary change alone was ~3.83% of that; the declaration-value scan accounts for most of the remainder).

https://claude.ai/code/session_017XjfDK7or6VrnJ41vBhxrK

---

Generated by Claude Code

[github-actions]

⚠️ Package Size Increase

📦 Package	📏 Base Size	📏 Source Size	📈 Size Change
@projectwallace/css-parser	39 kB	40.6 kB	+1.6 kB

[bartveneman]

CI is green except Audit packages, same pre-existing issue as #278: this branch doesn't touch package.json/pnpm-lock.yaml, and the two high findings are the transitive brace-expansion DoS advisory via tailwindcss > purgecss > glob > minimatch, already tracked separately in #258 (still open). Not fixing it here to keep this PR scoped to the tokenizer changes.

The size-increase comment above is expected — the dispatch consolidation and local-variable hoisting add a small amount of code in exchange for the ~6% measured speedup described in the PR body.

---

Generated by Claude Code