Step 1c: HTML tokenizer — html5lib conformance suite

[thomasnemer]

Parent: #20

Goal

Vendor the html5lib tokenizer test suite and write a conformance harness. Fix any edge cases to achieve >98% pass rate.

Prerequisites

• Step 1b (Step 1b: HTML tokenizer — script/raw text states and character references #55) — full tokenizer with all states and entities

File Changes

• crates/ie-html/tests/html5lib-tokenizer/*.json — vendored test fixtures
• crates/ie-html/tests/tokenizer_conformance.rs — conformance test harness

Implementation

• Vendor html5lib-tests tokenizer JSON files
• Conformance harness: parse each test, run tokenizer, compare tokens
• Adjacent Character tokens coalesced for comparison
• Report pass/fail with descriptive names
• Fix any failing edge cases in tokenizer

Acceptance Criteria

• 98% html5lib tokenizer conformance pass rate

• All unit tests still pass
• Clippy clean

[thomasnemer]

Context from #55 implementation

The full ~80-state WHATWG tokenizer is now complete:

API

• Tokenizer::new(input) — iterator-based, yields Token values
• Tokenizer::set_state(state) — tree builder feedback for RcData/RawText/ScriptData/PlainText
• All token types: Doctype, StartTag (with attributes), EndTag, Character, Comment, Eof

Character references

• Named: phf::Map from vendored entities.json (2231 entries), longest_match algorithm
• Numeric: decimal (&#N;) and hex (&#xN;) with Windows-1252 replacements
• Resolved in Data, RcData, and attribute value states

For conformance suite (#56)

• The tokenizer handles all WHATWG states
• html5lib test format: JSON with input, expected tokens, optional initialState and lastStartTag
• Adjacent Character tokens should be coalesced for comparison
• Some tests may need initialState mapped to TokenizerState enum values
• lastStartTag should set last_start_tag_name for appropriate end tag checks