Step 1b: HTML tokenizer — script/raw text states and character references

[thomasnemer]

Parent: #20

Goal

Add the remaining WHATWG tokenizer states: RcData, RawText, ScriptData (with all escape sub-states), PlainText, and character reference resolution (named, decimal, hex). This completes the full ~80-state tokenizer.

Prerequisites

• Step 1a (Step 1a: HTML tokenizer — core states and token types #54) — core states working

File Changes

• crates/ie-html/src/tokenizer.rs — add remaining states
• crates/ie-html/src/entities.rs — new file, entity lookup
• crates/ie-html/build.rs — new file, codegen for entity table
• crates/ie-html/data/entities.json — vendored WHATWG named character references
• crates/ie-html/Cargo.toml — add phf, serde_json build-dep

Implementation

Additional tokenizer states

• RcData, RcDataLessThanSign, RcDataEndTagOpen, RcDataEndTagName
• RawText, RawTextLessThanSign, RawTextEndTagOpen, RawTextEndTagName
• ScriptData + all escape states (~15 states)
• PlainText
• CharacterReference, NumericCharacterReference, NamedCharacterReference, etc.

Character reference resolution

• Vendor entities.json from WHATWG spec
• build.rs: generate phf::Map from entities.json
• Named entities: longest match, attribute value special rules
• Numeric: decimal (&#N;) and hex (&#xN;), Windows-1252 replacements

Tests

• Script content: <script>var x = 1 < 2;</script> → correct tokens
• Named entity: & → Character('&')
• Numeric entity: < → Character('<')
• Hex entity: < → Character('<')
• Entity in attribute: <a href="?a=1&amp;b=2"> → correct attribute value
• RCDATA: <title>&amp; stuff</title> → entities resolved
• RawText: <style>.a { }</style> → raw characters
• State switching via set_state()

Acceptance Criteria

• All core tests from 1a still pass
• Script/raw text/entity tests pass
• Clippy clean

[thomasnemer]

Context from #54 implementation

The core tokenizer is now working with ~40 states:

Token types (token.rs)

• Token::Doctype { name, public_id, system_id, force_quirks }
• Token::StartTag { name, attributes: Vec<Attribute>, self_closing }
• Token::EndTag { name }
• Token::Character(char), Token::Comment(String), Token::Eof
• Attribute { name, value }

Tokenizer (tokenizer.rs)

• Tokenizer::new(input) — creates iterator-based tokenizer
• Tokenizer::set_state(state) — tree builder feedback
• impl Iterator<Item = Token> — yields one token at a time
• Internal: pending_tokens: VecDeque<Token>, current_tag: Option<TagBuilder>
• Helper methods: push_to_tag_name, push_to_attr_name, push_to_attr_value
• starts_with(s) / starts_with_ci(s) for lookahead

States implemented

All core states: Data, TagOpen/EndTagOpen/TagName, all attribute states, SelfClosingStartTag, BogusComment, MarkupDeclarationOpen, all comment states, all doctype states, CDataSection.

Stubs remaining for Step 1b

RcData, RawText, ScriptData (+ all escape sub-states), PlainText, CharacterReference (+ all numeric/named sub-states). These are currently placeholders that emit characters as-is.

Fields reserved for Step 1b

• return_state — used by character reference states
• temp_buffer — used by char ref and script data states
• last_start_tag_name — for "appropriate end tag" check in raw text states