Tokenizer

The tokenizer is the first half of the parser: it turns text into doctypes, tags, comments, processing instructions and runs of characters, following section 13.2.5 of the HTML Standard. You can run it on its own, without building a tree, by giving it a Sink of your own. That's the fastest way through a page when you only need, say, every link's href and don't care about the tree.

On this page: A first sink · preprocess · Tokenizer · Sink · Tag · Attrs · Doctype · TextMode · Deadlines and positions · How fast it reads

Everything here is in parsec::tokenizer (crates/parsec/src/tokenizer.rs).

A first sink

use parsec::tokenizer::{Doctype, Sink, Tag, TextMode, Tokenizer, preprocess};

#[derive(Default)]
struct Links(Vec<String>);

impl Sink for Links {
    fn doctype(&mut self, _: &Doctype<'_>) {}
    fn start_tag(&mut self, tag: &Tag<'_>) -> Option<TextMode> {
        if tag.name == "a" {
            if let Some((_, href)) = tag.attrs().find(|&(name, _)| name == "href") {
                self.0.push(href.to_string());
            }
        }
        None
    }
    fn end_tag(&mut self, _: &Tag<'_>) {}
    fn comment(&mut self, _: &str) {}
    fn processing_instruction(&mut self, _: &str, _: &str) {}
    fn text(&mut self, _: &str) {}
    fn eof(&mut self) {}
}

fn main() {
    let html = r#"<p><A HREF="/one">1</a> <a href='/two?x&amp;y'>2</a>"#;
    let input = preprocess(html);
    let mut links = Links::default();
    Tokenizer::new(&input).run(&mut links);
    assert_eq!(links.0, ["/one", "/two?x&y"]);
}

Names come lowercased and references decoded, as the spec says. What the tokenizer alone can't do is the tree builder's job: it won't tell you a tag was implied or closed, and it won't switch into raw text after <script> unless your sink asks it to. More on that under Text modes.

preprocess

pub fn preprocess(input: &str) -> Cow<'_, str>

The spec's input stream preprocessing: every \r\n and every lone \r becomes \n. Input with no \r comes back borrowed, untouched. Always run your input through it before Tokenizer::new: the tokenizer doesn't treat \r as whitespace, so a carriage return left in a tag would end up in an attribute name.

use parsec::tokenizer::preprocess;
use std::borrow::Cow;

assert_eq!(preprocess("a\r\nb\rc\n\r"), "a\nb\nc\n\n");
assert!(matches!(preprocess("no returns"), Cow::Borrowed(_)));

Tokenizer

pub struct Tokenizer<'a> { /* private */ }

It reads the whole input at once, borrowing it for 'a.

Method	Does
new(input: &'a str) -> Tokenizer<'a>	A tokenizer at the start of input, in the data state. input should already have been through preprocess.
run<S: Sink>(&mut self, sink: &mut S)	Tokenizes to the end, calling sink for each token, and calls sink.eof() last. If a deadline passes first, it returns early, without calling eof.
set_text_mode(&mut self, mode: TextMode)	Starts in RCDATA, RAWTEXT, script data, PLAINTEXT or a CDATA section instead of the data state. Call it before run.
set_last_start_tag(&mut self, name: &str)	The tag name an end tag has to match to close RCDATA, RAWTEXT or script data, as when tokenizing the inside of a <title> you didn't see open. Give it lowercase.
set_deadline(&mut self, deadline: std::time::Instant)	Makes run stop once deadline has passed. It looks at the clock once every 4,096 turns of its main loop.
timed_out(&self) -> bool	Whether the last run stopped at the deadline.
position(&self) -> usize	How many bytes of the input it has consumed. After a full run, the input's length; after a timeout, how far it got.

run is meant to be called once. Parse errors are handled as the spec says and not reported.

Note

Error reporting with names and positions is planned, not shipped (#7).

Sink

pub trait Sink {
    fn doctype(&mut self, doctype: &Doctype<'_>);
    fn start_tag(&mut self, tag: &Tag<'_>) -> Option<TextMode>;
    fn end_tag(&mut self, tag: &Tag<'_>);
    fn comment(&mut self, text: &str);
    fn processing_instruction(&mut self, target: &str, data: &str);
    fn text(&mut self, text: &str);
    fn eof(&mut self);
    fn cdata_allowed(&self) -> bool { false }
}

Where the tokens go. Every reference you're handed is only good for the length of the call; copy what you want to keep.

Method	Called for
doctype	<!DOCTYPE ...>. See Doctype.
start_tag	A start tag. Return Some(mode) to switch the tokenizer into a text mode, as the tree builder does after <title> or <script>; return None otherwise.
end_tag	An end tag. The spec gives end tags no use for attributes or />, so read only name.
comment	A comment's text, without <!-- and -->. Bogus comments arrive here too: <!x> gives x, and <?xml ...?> gives ?xml ...?.
processing_instruction	<?target data?> or <?target data>, as the current HTML Standard parses them. The target starts with an ASCII letter or _ and holds letters, digits, - and _; anything else, and the targets xml and xml-stylesheet in any case, make a bogus comment instead.
text	A run of characters. Runs are split wherever the input needed a change (at a character reference, a NUL, a </ that didn't end anything), so one stretch of text can arrive as several calls. Join adjacent runs if you need whole text. A NUL in the data state arrives as "\0", which the tree builder later drops.
eof	The end of the input, once.
cdata_allowed	Asked at <![CDATA[. Return true to read a CDATA section, whose contents arrive through text; the tree builder says yes when the current node is SVG or MathML. With the default false, it's a bogus comment, as in HTML content.

Tag

pub struct Tag<'a> {
    pub name: &'a str,
    pub self_closing: bool,
    /* private */
}

impl<'a> Tag<'a> {
    pub fn attrs(&self) -> Attrs<'a>;
}

• name is lowercased.
• self_closing is whether the tag ended in />. In HTML that only matters for void elements and foreign content; <div/> is just <div>.
• attrs() gives the attributes in source order, as (name, value) pairs: names lowercased, values decoded, and repeated names dropped so that the first one wins. Each string is a slice of your input when nothing in it had to change, and points into the tokenizer's buffer otherwise.

You can't build a Tag yourself; you only receive them.

Attrs

pub struct Attrs<'a> { /* private */ }
impl<'a> Iterator for Attrs<'a> { type Item = (&'a str, &'a str); }
impl ExactSizeIterator for Attrs<'_> {}

The iterator Tag::attrs returns. It knows its length, so tag.attrs().len() is free.

Doctype

#[derive(Debug)]
pub struct Doctype<'a> {
    pub name: Option<&'a str>,
    pub public_id: Option<&'a str>,
    pub system_id: Option<&'a str>,
    pub force_quirks: bool,
}

The spec's DOCTYPE token. name is lowercased. Each part is None when it was missing, which isn't the same as present and empty. force_quirks is the spec's force-quirks flag, set for malformed doctypes; the tree builder uses it and the identifiers to pick the quirks mode.

TextMode

#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TextMode { Rcdata, Rawtext, ScriptData, Plaintext, Cdata }

The states other than data that the tokenizer can be put in.

Mode	Used for, in HTML	Inside it
Rcdata	<title>, <textarea>	Tags are text; character references are decoded.
Rawtext	<style>, <xmp>, <iframe>, <noembed>, <noframes>, and <noscript> with scripting on	Tags and references are text.
ScriptData	<script>	Like RAWTEXT, plus the spec's rules for <!-- and <script> inside scripts.
Plaintext	<plaintext>	Everything to the end is text; nothing closes it.
Cdata	Only for starting a tokenizer in a CDATA section, as the html5lib tests do. During a run, cdata_allowed decides instead.

The first four end at the matching end tag, </title> and so on, which the tokenizer recognises by the last start tag that switched it (or the one you set with set_last_start_tag).

Text modes

The tokenizer doesn't know which elements hold raw text; the tree builder tells it. If you run the tokenizer alone and want <script> and <style> contents to come through as text rather than as tags, return the mode from start_tag:

use parsec::tokenizer::{Doctype, Sink, Tag, TextMode, Tokenizer, preprocess};

#[derive(Default)]
struct Tags(Vec<String>);

impl Sink for Tags {
    fn doctype(&mut self, _: &Doctype<'_>) {}
    fn start_tag(&mut self, tag: &Tag<'_>) -> Option<TextMode> {
        self.0.push(tag.name.to_string());
        match tag.name {
            "script" => Some(TextMode::ScriptData),
            "style" | "xmp" | "iframe" | "noembed" | "noframes" => Some(TextMode::Rawtext),
            "title" | "textarea" => Some(TextMode::Rcdata),
            "plaintext" => Some(TextMode::Plaintext),
            _ => None,
        }
    }
    fn end_tag(&mut self, tag: &Tag<'_>) { self.0.push(format!("/{}", tag.name)); }
    fn comment(&mut self, _: &str) {}
    fn processing_instruction(&mut self, _: &str, _: &str) {}
    fn text(&mut self, _: &str) {}
    fn eof(&mut self) {}
}

fn main() {
    let input = preprocess("<script>if (a<b) document.write('<p>')</script><p>");
    let mut tags = Tags::default();
    Tokenizer::new(&input).run(&mut tags);
    // The <p> inside the script is text, not a tag.
    assert_eq!(tags.0, ["script", "/script", "p"]);
}

Deadlines and positions

set_deadline and position are what parse_with's budget is built on. You can use them the same way:

use parsec::tokenizer::{Doctype, Sink, Tag, TextMode, Tokenizer, preprocess};
use std::time::{Duration, Instant};

struct Count(usize);
impl Sink for Count {
    fn doctype(&mut self, _: &Doctype<'_>) {}
    fn start_tag(&mut self, _: &Tag<'_>) -> Option<TextMode> { self.0 += 1; None }
    fn end_tag(&mut self, _: &Tag<'_>) {}
    fn comment(&mut self, _: &str) {}
    fn processing_instruction(&mut self, _: &str, _: &str) {}
    fn text(&mut self, _: &str) {}
    fn eof(&mut self) {}
}

fn main() {
    let html = "<p>x".repeat(1_000_000);
    let input = preprocess(&html);
    let mut tokenizer = Tokenizer::new(&input);
    tokenizer.set_deadline(Instant::now() + Duration::from_nanos(1));
    let mut count = Count(0);
    tokenizer.run(&mut count);
    if tokenizer.timed_out() {
        assert!(tokenizer.position() < input.len());
        println!("stopped at byte {} after {} tags", tokenizer.position(), count.0);
    }
}

position counts bytes of the preprocessed input. There are no line and column numbers yet, and no positions on tokens; source positions for nodes and attributes are planned (#8).

How fast it reads

The tokenizer works on the whole input, so it can look ahead instead of keeping the spec's temporary buffer. It scans runs of text with memchr, and reads plain tags (lowercase names, values with no references or NULs) in a single pass. Text, tag names, attribute names and values are handed to your sink as slices of the input wherever nothing in them changed, so most of a page reaches you without a copy. cargo run --release --example tokenize FILE times it alone on a file.

See also

• Parsing, when you want the tree after all.
• Architecture, for how the tree builder drives it.
• Benchmarks, for timing it on your own pages.