Repository navigation
WASM
Parsec on npm is parsec-html: the parser compiled to WebAssembly, with TypeScript types, for Node and for browsers. Node names follow the DOM wherever the DOM has a name for the thing, so code written against a browser's tree reads the same. It also has XPath 1.0, where every node was written in the source, the parse errors, and pages as bytes in any encoding, found the way a browser finds it.
On this page: Install · A first page · Loading it · Parsing · Bytes · Document · Node · XPath · Spans · Parse errors · Memory · Size · Working on it
Note
Parsec isn't released. The package goes to npm with 0.1.0; until then you build it from the repository (see Working on it).
import { parse } from "parsec-html";
const doc = parse(`
<ul>
<li class=item data-sku=A1><a href=/a>Kettle</a>
<li class=item data-sku=B2><a href=/b>Toaster</a>
</ul>`);
for (const item of doc.select("li.item")) {
const link = item.selectFirst("a");
console.log(item.getAttribute("data-sku"), link.textContent, link.getAttribute("href"));
}
doc.selectFirst("[data-sku=B2]").remove();
console.log(doc.body.outerHTML);The package holds two builds, and package.json picks between them:
-
Node gets one that loads its
.wasmitself, soimport { parse } from "parsec-html"(orrequire) is all you need. - Browsers and bundlers get the web build, which you start once:
import init, { parse } from "parsec-html";
await init();| Call | |
|---|---|
parse(html, options?) |
a whole document, as a browser builds it |
parseFragment(html, context?, options?) |
the markup as the contents of a context element ("div" if you leave it out), as innerHTML parses it: parseFragment("<td>1", "tr")
|
parseBytes(bytes, options?) |
a whole document from a Uint8Array, decoded as a browser decodes it; see Bytes
|
new ByteParser(options?) |
the same for bytes that arrive in pieces: feed(bytes), then finish()
|
options is { scripting?: boolean, maxDepth?: number, spans?: boolean, errors?: boolean }: scripting parses <noscript> as a browser with JavaScript on does, maxDepth keeps the tree at most that deep, as Chromium does at 512, spans records where each node and attribute was written (Spans), and errors keeps the parse errors (Parse errors). Input over 512 MiB throws.
Hand over what fetch gave you and Parsec finds the encoding by the HTML Standard's sniffing: a byte order mark, then transportEncoding (the charset from the Content-Type, by any label the Encoding Standard knows; one it doesn't is passed over, as browsers do), then a <meta> in the first 1,024 bytes, then UTF-8 if the bytes are valid UTF-8 (the standard's autodetection step, as Firefox and Chrome take it), then defaultEncoding, windows-1252 unless you give another. A <meta> the parser meets later, naming another encoding, makes it parse again in that one, once, as a browser reloads.
import { parseBytes, ByteParser } from "parsec-html";
const res = await fetch(url);
const charset = /charset=([^;]+)/i.exec(res.headers.get("content-type") ?? "")?.[1];
const doc = parseBytes(new Uint8Array(await res.arrayBuffer()), { transportEncoding: charset });
doc.characterSet; // "Shift_JIS"
doc.encodingConfidence; // "certain"
// Or as it arrives, with the same result however the bytes are cut:
const parser = new ByteParser({ transportEncoding: charset, spans: true });
for await (const chunk of res.body) parser.feed(chunk);
const streamed = parser.finish();The options are ByteParseOptions: ParseOptions and transportEncoding?: string and defaultEncoding?: string. A defaultEncoding the Encoding Standard doesn't know throws. characterSet is the Encoding Standard's name for the encoding, as the DOM's document.characterSet; it's "UTF-8" for a string, as DOMParser says. encodingConfidence is "certain" when a byte order mark, the transport encoding or a <meta> the parser met decided it, "tentative" when only the prescan, the UTF-8 detection or the default did, and undefined for a string.
A ByteParser finishes once; feed or finish after that throws. Until the encoding is known it holds the bytes back (a byte order mark or a transport encoding decides at once, a whole <meta> as soon as it's read, and otherwise 1,024 bytes), and while the encoding is a guess it keeps every byte, in case a late <meta> means parsing again. Pass the transport encoding when you have it.
Spans and error offsets for bytes are byte offsets into the Uint8Array, in whatever encoding it was: bytes.subarray(span.start, span.end) is the tag as it was sent, a Shift_JIS kanji counting two. line and column count characters of the decoded page.
The 82 html5lib-tests encoding/ cases that don't need a script engine run through both, in Node and in Firefox, parseBytes whole and ByteParser in pieces of 1 to 7 bytes, and all pass. The Encoding page has the decoders' own conformance.
| Member | Gives |
|---|---|
root |
the document node (nodeType 9), or a fragment's root (11), which holds its nodes |
documentElement, head, body
|
those elements, or undefined
|
html |
the whole document as markup |
text |
every text node's text, in tree order |
select(selector) |
every matching element, in document order |
selectFirst(selector) |
the first match, found without looking further, or undefined
|
xpath(expr) |
an XPath 1.0 query from the document node; see XPath |
errors |
the parse errors, an array of ParseError
|
characterSet |
the encoding bytes were decoded from ("Shift_JIS"), or "UTF-8" for a string; see Bytes
|
encodingConfidence |
"certain" or "tentative", or undefined for a string |
| Member | Gives |
|---|---|
nodeType |
1 element, 3 text, 8 comment, 9 document, 10 doctype, 11 fragment |
nodeName |
DIV for an HTML element, foreignObject for SVG's, #text, #comment, #document, a doctype's name |
localName |
an element's name, lowercase for HTML; undefined for anything else |
parentNode, firstChild, lastChild, nextSibling, previousSibling
|
the neighbours, or undefined
|
childNodes, children
|
every child, or only the elements |
content |
a template's contents, which selectors and childNodes don't reach, as in a browser |
select, selectFirst
|
as on the document, under this node; as querySelectorAll, the node itself isn't counted, and combinators look past it |
matches(selector) |
whether this element matches |
getAttribute(name), hasAttribute, getAttributeNames()
|
names as written (xlink:href), looked up in lowercase on HTML elements |
setAttribute(name, value), removeAttribute(name)
|
edits, as the DOM's; removeAttribute says whether there was one |
id |
the id attribute, or ""
|
textContent, outerHTML, innerHTML
|
text and markup, read-only for now |
remove() |
takes the node out; it stays readable, and can be put back |
appendChild(child), insertBefore(child, reference)
|
move a node of the same document; reference must be one of this node's children |
isSameNode(other) |
whether two handles name the same node |
xpath(expr) |
an XPath 1.0 query with this node as the context |
span, endSpan, attributeSpan(name)
|
where the node, its end tag or one attribute was written; see Spans |
A selector that doesn't parse throws an Error saying where; so does an edit that would put a node inside itself or move one between documents.
Not there yet: setting innerHTML or text, and creating nodes.
const doc = parse("<ul id=l><li class=a>one<li>two</ul>");
doc.xpath("//li"); // [Node, Node]
doc.xpath("//li/@class"); // ["a"]
doc.xpath("count(//li)"); // 2
doc.xpath("boolean(//table)"); // false
doc.xpath("//li")[1].xpath("string(preceding-sibling::li)"); // "one"The TypeScript type is XPathResult: (Node | string)[] | string | number | boolean. A node-set is an array in document order, of Nodes for elements, text, comments and the document, and of strings for attributes, which have no node of their own here. A query that doesn't parse, or uses a $variable, throws; there's no way to pass variables from JavaScript yet. Names match lowercase and in no namespace, as lxml matches HTML: //div finds every div, //DIV none.
The first query after a change indexes the document, once, and later queries reuse the index until the next edit. A removed node can be queried too: its subtree is a tree of its own. The engine and its limits are on XPath.
Parse with { spans: true } and a node's span says where it was written:
const src = "<p class=note>Hi</p>\n<div>x</div>";
const doc = parse(src, { spans: true });
const p = doc.selectFirst("p");
src.slice(p.span.start, p.span.end); // "<p class=note>"
p.endSpan; // </p>, from 16 to 20
p.attributeSpan("class"); // class=note, from 3 to 13
doc.selectFirst("div").span.line; // 2A Span has start and end, indexes into the string you parsed in UTF-16 units, as JavaScript counts, so src.slice(span.start, span.end) is the source exactly, emoji and other characters past the Basic Multilingual Plane included. It also has the line and column of start, both from 1. The column counts characters, not UTF-16 units: an emoji moves the index by two and the column by one.
For a document from bytes, start and end are byte offsets into them instead; see Bytes.
The parser counts UTF-8 bytes. For a string with anything beyond ASCII, the package keeps a copy of it and, the first time a span or error is read, a table of UTF-16 units per 64 bytes, so each conversion after that looks at 64 bytes at most. ASCII strings need neither, and nothing is kept without spans or errors.
span is an element's start tag from < to >, or a whole text node, comment or doctype; endSpan is the end tag that closed an element. All three are undefined without { spans: true }, for what the parser implied (<html>, <head>, <body>, <tbody> nobody wrote), and for what an edit made.
const doc = parse("<p>a\n</b>", { errors: true });
for (const e of doc.errors) console.log(`${e.line}:${e.column}: ${e.code}`);
// 1:1: expected-doctype-but-got-start-tag
// 2:1: unexpected-end-tagEach ParseError has code, the HTML Standard's name for it (or html5lib's, for the errors of tree construction, which the standard leaves unnamed), offset, an index into the string as a span's start is, and line and column. Without { errors: true } the array is empty. They're records of what the parser recovered from, not exceptions: the document still parsed.
A Document and each Node you're handed hold the parsed tree alive between them. The package is built with weak references, so the garbage collector frees them once you drop them; doc.free(), or using doc = parse(...), frees one sooner.
The WebAssembly is 1,071 KB, 504 KB gzipped. Bytes in added 201 KB of that (142 KB gzipped), nearly all of it encoding_rs's decoding tables for the Chinese, Japanese and Korean encodings: main before it built to 871 KB, 362 KB gzipped. That's under the 150 KB gzipped over which it would have gone behind a separate entry point, so it's in the one build. XPath and spans added 110 KB (46 KB gzipped) and parse errors 21 KB (7 KB): main before them built to 739 KB, 308 KB gzipped. It's built with opt-level = "s" and wasm-opt -Os: "z" saved 59 KB but parsed the 13 pages in bench/ 40% slower (131 ms against 186 ms in Node 22, measured at a load of 5.8). CI writes the sizes into each run's summary.
You need Rust with the wasm32-unknown-unknown target and wasm-pack.
cd bindings/wasm
wasm-pack test --node # the API, in Node
wasm-pack test --headless --firefox --test browser # the same, in Firefox
./build.sh # the package, in pkg/
mkdir -p node_modules && ln -s ../pkg node_modules/parsec-html
node --test tests/package.test.mjs # the package as installedParsec is Apache-2.0 and not yet released; these pages describe main. Found something wrong here? Open an issue or ask in Discussions.
Repository · Issues · Discussions · Roadmap · Security policy · Licence