Parsing

Parsing turns a string of HTML into a Document. Parsec has three entry points: parse for a whole page, parse_with for a whole page with a time budget and errors instead of panics, and parse_fragment for markup that belongs inside an element, as innerHTML does. All three live at the crate root and are defined in crates/parsec/src/tree.rs.

On this page: parse · parse_with and Options · ParseError · MAX_INPUT · parse_fragment · Quirks mode · What scripting changes · Input · How long a document lives

None of them reports parse errors. The tokenizer and tree builder recover from every mistake the way the HTML Standard says to, silently.

Note

Reporting errors with their spec names, line and column is planned, not shipped (#7).

`parse`

pub fn parse(html: &str, scripting: bool) -> Document

Parses a whole document.

html: the page, already decoded to UTF-8. See Input below.
scripting: true parses as a browser with JavaScript on, false as one with it off. See What scripting changes.
Returns the document. Its root, node 0, is a Kind::Document node.
Panics if html.len() is over MAX_INPUT bytes, with the message {len} bytes of HTML is over the 536870912 byte limit.
Has no time budget.

Warning

parse panics on oversize input. If your input comes from somewhere you don't control, use parse_with, which returns an error instead.

let doc = parsec::parse("<p>one<p>two", false);
assert_eq!(doc.inner_html(doc.root()), "<html><head></head><body><p>one</p><p>two</p></body></html>");

`parse_with` and `Options`

pub fn parse_with(html: &str, options: Options) -> Result<Document, ParseError>

#[derive(Debug, Clone, Copy, Default)]
pub struct Options {
    pub scripting: bool,
    pub budget: Option<std::time::Duration>,
}

parse_with parses a whole document with the options you give and returns an error instead of panicking.

Options::scripting is the same flag parse takes. Default: false.
Options::budget is how long the parse may take. Default: None, no limit. A budget so large that adding it to the start time overflows the clock (Duration::MAX, say) also means no limit.

Here's every way a call can end:

flowchart TD
    call["parse_with(html, options)"] --> size{"html.len() over MAX_INPUT?"}
    size -->|"yes"| large["Err(ParseError::TooLarge)"]
    size -->|"no"| clock["Start the clock, if options.budget is set"]
    clock --> run["Tokenize and build the tree"]
    run --> check{"Deadline passed?<br/>checked every 4,096 steps"}
    check -->|"no, more input"| run
    check -->|"yes"| timeout["Err(ParseError::Timeout)<br/>the partial tree is dropped"]
    check -->|"no, input finished"| ok["Ok(Document)"]

Options derives Default, so you can set just the field you care about:

use parsec::{Options, ParseError, parse_with};
use std::time::Duration;

let options = Options {
    budget: Some(Duration::from_millis(250)),
    ..Options::default()
};
match parse_with("<p>fine</p>", options) {
    Ok(doc) => assert!(doc.text(doc.root()).contains("fine")),
    Err(ParseError::Timeout { parsed, .. }) => println!("gave up after {parsed} bytes"),
    Err(ParseError::TooLarge { len }) => println!("{len} bytes is too much"),
}

How the budget is measured

The clock starts when parse_with is called, after the size check. The tokenizer compares the time with the deadline once every 4,096 turns of its main loop, and tree building happens inside that loop, so the budget covers both. Two consequences worth knowing:

The budget is approximate. A parse can run a little past it, and a short input can finish even with a budget of zero, because it never reaches its first check.
On timeout the partly built document is dropped. You get the error, not a partial tree.

Limits and hostile input says when you'd want a budget and how big.

`ParseError`

#[derive(Debug, Clone, PartialEq, Eq)]
pub enum ParseError {
    TooLarge { len: usize },
    Timeout { budget: std::time::Duration, parsed: usize },
}

Why parse_with returned no document. It implements std::error::Error and Display.

Variant	When	`Display`
`TooLarge { len }`	`html.len()` is over `MAX_INPUT`. `len` is the input's length in bytes.	`{len} bytes of HTML is over the 536870912 byte limit`
`Timeout { budget, parsed }`	The budget ran out. `budget` is the one you set; `parsed` is how many bytes the tokenizer had read.	`parsing took longer than {budget:?} (gave up after {parsed} bytes)`

parsed counts bytes of the input after preprocessing (see Input). If your input had \r\n line endings, each one counts as one byte there, so parsed can be smaller than the matching offset in your original string.

use parsec::{Options, ParseError, parse_with};
use std::time::Duration;

let html = "<div><b>x</b> y</div>".repeat(200_000);
let tight = Options { budget: Some(Duration::from_nanos(1)), ..Options::default() };
match parse_with(&html, tight) {
    Err(ParseError::Timeout { parsed, .. }) => assert!(parsed < html.len()),
    other => panic!("expected a timeout, got {:?}", other.map(|d| d.len())),
}

`MAX_INPUT`

pub const MAX_INPUT: usize = 1 << 29; // 536,870,912 bytes, 512 MiB

The most input a document can hold. Parsec stores strings as offsets with a few flag bits on top (see Architecture), and this cap keeps every offset in range. The check is on html.len() in bytes, before anything else happens.

parse and parse_fragment panic over the limit.
parse_with returns ParseError::TooLarge.

use parsec::{MAX_INPUT, Options, ParseError, parse_with};

let too_big = "a".repeat(MAX_INPUT + 1);
assert_eq!(
    parse_with(&too_big, Options::default()).err(),
    Some(ParseError::TooLarge { len: MAX_INPUT + 1 })
);

`parse_fragment`

pub fn parse_fragment(html: &str, ns: Ns, context: &str, scripting: bool) -> Document

Parses html as the contents of an element named context in namespace ns, the HTML Standard's fragment parsing algorithm, which is what setting innerHTML runs.

html: the markup.
ns: the context element's namespace, Ns::Html, Ns::Svg or Ns::MathMl. Ns::Xml is accepted but means nothing to the HTML parser; don't use it.
context: the context element's name, such as "div", "tr", "textarea", "svg" or "foreignObject". Known HTML tags are recognised in any case: "TR" and "tr" behave the same.
scripting: as for parse.
Returns a document whose root, node 0, is a Kind::Fragment node. The parsed nodes are the root's children.
Panics as parse does, over MAX_INPUT bytes.
Has no time budget, and there's no parse_fragment_with yet.

use parsec::{Kind, Ns, parse_fragment};

// A row's worth of cells only makes sense inside a <tr>.
let doc = parse_fragment("<td>a<td>b", Ns::Html, "tr", false);
assert_eq!(doc.kind(doc.root()), Kind::Fragment);
assert_eq!(doc.inner_html(doc.root()), "<td>a</td><td>b</td>");

// Parsed in a <div> instead, the cells are dropped and only text is left.
let doc = parse_fragment("<td>a<td>b", Ns::Html, "div", false);
assert_eq!(doc.inner_html(doc.root()), "ab");

What the context decides

The context element sets things up exactly as the spec's fragment case does:

The tokenizer's starting state. For an HTML context, title and textarea start in RCDATA, so tags are text and references are decoded; style, xmp, iframe, noembed and noframes start in RAWTEXT; script in script data; plaintext in PLAINTEXT; and noscript in RAWTEXT when scripting is on. Any other context starts in the data state.
The insertion mode, through the spec's "reset the insertion mode appropriately": a tr context parses as in a row, table as in a table, select as in a select, html as before the head, and so on.
A template context parses as template contents.
A form context sets the form element pointer, so a nested <form> is ignored.
An SVG or MathML context parses in foreign content.

use parsec::{Ns, parse_fragment};

// RCDATA: the tags are text.
let doc = parse_fragment("a<b>x</b>&amp;", Ns::Html, "textarea", false);
assert_eq!(doc.inner_html(doc.root()), "a&lt;b&gt;x&lt;/b&gt;&amp;");

// SVG: names get SVG's case, and <b> breaks back out into HTML.
let doc = parse_fragment("<clippath/><b>x</b>", Ns::Svg, "svg", false);
assert_eq!(doc.inner_html(doc.root()), "<clipPath></clipPath><b>x</b>");

// An html context gets a head and a body.
let doc = parse_fragment("<p>x", Ns::Html, "html", false);
assert_eq!(doc.inner_html(doc.root()), "<head></head><body><p>x</p></body>");

Things to know about fragment documents

quirks() is always Quirks::No. A fragment never sees a doctype, and Parsec has no outer document whose mode it could borrow, so <p><table> and selector case rules behave as in a no-quirks page.
The context element and the <html> element the algorithm uses internally stay in the document's arena, detached. doc.len() counts them, so an empty fragment has len() == 3. They aren't reachable from the root.
:root never matches in a fragment, because no element's parent is a Document node. See Selectors.

Quirks mode

A whole-document parse decides the document's mode from its doctype, as the spec's initial insertion mode does, and Document::quirks() reports it as a Quirks value:

Quirks::Quirks when there's no doctype at all, when the doctype has force-quirks set or a name other than html, or when its public or system identifier is on the spec's list of legacy ones (-//W3C//DTD HTML 4.01 Transitional// without a system identifier, for example).
Quirks::Limited for the XHTML 1.0 Frameset and Transitional public identifiers, and for HTML 4.01 Frameset and Transitional when a system identifier is present.
Quirks::No otherwise, including <!DOCTYPE html>.

The mode changes the tree in the places the spec says (in quirks mode a <table> doesn't close an open <p>), and it changes selector matching: in Quirks::Quirks documents, ids and class names compare without ASCII case. Quirks::Limited doesn't change either.

use parsec::{Quirks, parse};

assert_eq!(parse("<p>", false).quirks(), Quirks::Quirks);
assert_eq!(parse("<!DOCTYPE html><p>", false).quirks(), Quirks::No);
assert_eq!(
    parse(r#"<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"><p>"#, false).quirks(),
    Quirks::Limited
);

// In quirks mode the table goes inside the paragraph.
assert_eq!(
    parse("<p><table>", false).inner_html(0),
    "<html><head></head><body><p><table></table></p></body></html>"
);
assert_eq!(
    parse("<!DOCTYPE html><p><table>", false).inner_html(0),
    "<!DOCTYPE html><html><head></head><body><p></p><table></table></body></html>"
);

What `scripting` changes

Parsec never runs scripts. The flag says which browser to imitate, one with JavaScript on or one with it off, and the HTML Standard only lets that change one element, <noscript>:

Scripting off: <noscript> content is parsed as markup. In the <head>, only <link>, <meta>, <style> and the like stay inside it; anything else closes it.
Scripting on: <noscript> content is raw text, like <style>. Its child is one text node holding the markup as written.

The document remembers the flag, and the serializer writes a <noscript>'s text without escaping when it was parsed with scripting on, so the markup round-trips. A fragment with a noscript context follows the same rule.

use parsec::parse;

let html = "<body><noscript><b>x</b></noscript>";

// Off: <b> is an element.
let off = parse(html, false);
assert_eq!(off.text(0), "x");

// On: the whole thing is one text node, written back unescaped.
let on = parse(html, true);
assert_eq!(on.text(0), "<b>x</b>");
assert_eq!(on.inner_html(0), "<html><head></head><body><noscript><b>x</b></noscript></body></html>");

If you're extracting content, false is usually what you want: <noscript> fallbacks then become real elements your selectors can see.

Input

Parsec takes text, not bytes. Here's what that means in practice.

Decode first. html is a &str, so it's valid UTF-8 by construction. Parsec doesn't detect encodings from a BOM or <meta charset>; that's #5, with the decoders in #25. For now, decode with whatever you have: String::from_utf8_lossy for UTF-8, or a crate such as encoding_rs for the rest.
Strip a byte order mark, as the warning below explains.
Line endings. The spec's input preprocessing turns every \r\n and every lone \r into \n, and Parsec does that before tokenizing. Text you read back has \n only. An input with no \r at all skips the rewrite.
NUL characters follow the spec: dropped from text in the body, turned into U+FFFD in foreign content, attribute values and comments.
Whole input only. There's no streaming or chunked API yet (#6); the whole page has to be in memory.

Important

A browser drops a leading U+FEFF, the byte order mark, while decoding. Parsec sees it as text, so a page that starts with a BOM before its doctype parses in quirks mode with a stray character in the body. Remove it yourself: html.strip_prefix('\u{FEFF}').unwrap_or(html).

use parsec::{Quirks, parse};

let raw = "\u{FEFF}<!DOCTYPE html><p>a\r\nb</p>";

// As it is, the BOM is text and the doctype comes too late.
assert_eq!(parse(raw, false).quirks(), Quirks::Quirks);

// Stripped, it parses as the browser would.
let doc = parse(raw.strip_prefix('\u{FEFF}').unwrap_or(raw), false);
assert_eq!(doc.quirks(), Quirks::No);
assert_eq!(doc.text(doc.root()), "a\nb");

How long a document lives

Document owns its data. It keeps its own copy of the preprocessed input, and text and attribute values that needed no decoding point into that copy rather than being copied again. Nothing in the returned document borrows from your html string, so you can drop the string straight after parsing. Architecture explains the layout.

Navigation Menu

Repository navigation

Parsing

`parse`

`parse_with` and `Options`

How the budget is measured

`ParseError`

`MAX_INPUT`

`parse_fragment`

What the context decides

Things to know about fragment documents

Quirks mode

What `scripting` changes

Input

How long a document lives

See also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

Parsing

parse

parse_with and Options

How the budget is measured

ParseError

MAX_INPUT

parse_fragment

What the context decides

Things to know about fragment documents

Quirks mode

What scripting changes

Input

How long a document lives

See also

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Clone this wiki locally

`parse`

`parse_with` and `Options`

`ParseError`

`MAX_INPUT`

`parse_fragment`

What `scripting` changes