Repository navigation
Python
Parsec's Python package is parsec-html, and the module you import is parsec_html. Its API is selectolax's, so moving over is mostly a change of import, and it adds what selectolax can't do: one parsed page shared by many threads, free-threaded Python with the GIL off, selectors selectolax can't run, such as :lang(), :dir() and :valid, XPath 1.0 that answers the way lxml's does, where every node was written in your input, and the parse errors.
On this page: Install · A first page · Threads · Moving from selectolax · HTMLParser · Node · Attributes · XPath · Positions · Parse errors · Errors · Speed · Wheels · Working on it
Note
Parsec isn't released. Wheels go to PyPI with 0.1.0; until then you build from git, which needs Rust 1.88 or newer.
pip install "git+https://github.com/parsec-html/parsec#subdirectory=bindings/python"It runs on CPython 3.9 and later, free-threaded 3.14t and later included (PyO3, which the package is built with, doesn't support free-threaded 3.13). The package ships type stubs, so your editor and mypy know every signature on this page.
Why parsec_html and not parsec? There's already a parsec on PyPI (a parser-combinator library), and two packages that install the same module name overwrite each other. parsec_html matches the name you pip install, so nothing clashes.
from parsec_html import HTMLParser
page = HTMLParser("""
<title>Shop</title>
<ul>
<li class=item data-sku=A1><a href=/a>Kettle</a> <span class=price>£24</span>
<li class=item data-sku=B2><a href=/b>Toaster</a> <span class=price>£31</span>
</ul>
""")
for item in page.css("li.item"):
link = item.css_first("a")
print(item.attrs["data-sku"], link.text(), link.attributes["href"],
item.css_first(".price").text())
page.css_first("li[data-sku=B2]").decompose()
print(page.body.html)A parsed page belongs to no thread. Read it from as many as you like, and edit it from any of them: readers share it, and an edit waits for them, then has it to itself. Parsing and css() let go of the GIL while they work, so on ordinary CPython two threads parsing or querying really do run at once.
On a free-threaded build (3.14t, 3.15t) the module declares that it doesn't need the GIL, so importing it leaves the GIL off:
import sys, threading
from parsec_html import HTMLParser
page = HTMLParser(open("big.html").read())
counts = []
threads = [threading.Thread(target=lambda: counts.append(len(page.css("a")))) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()
assert not sys._is_gil_enabled() # on 3.14tselectolax documents its Lexbor parsers as not thread-safe: one parser per thread. Underneath, Parsec's Document is Send and Sync, checked when it compiles; see Thread safety and memory safety.
# from selectolax.lexbor import LexborHTMLParser as HTMLParser
from parsec_html import HTMLParserThese were checked against selectolax 1.0.2 by running the same calls through both. Everything below this table works the way selectolax's does, except where it says so.
| selectolax 1.0.2 | parsec_html | |
|---|---|---|
<a href>'s attributes
|
{'href': None} |
{'href': ''}. Parsec's tree doesn't yet record whether a value was written. |
| A selector that doesn't parse | SelectolaxError |
SelectorError, a ValueError
|
A node after decompose()
|
freed; reading it gives an empty element | still there, detached: readable, and can be inserted again |
attrs.items(), keys(), values()
|
iterators | lists |
iter(), traverse()
|
generators | iterators over a list taken when you call them |
bytes in |
UTF-8, or sniffed with encoding=True
|
UTF-8, bad bytes replaced with U+FFFD; sniffing comes with #5 |
| Class names |
LexborHTMLParser, LexborNode, LexborAttributes
|
HTMLParser, Node, Attributes
|
:lang(), :dir(), :valid, :in-range, :scope
|
SelectolaxError: "Can't parse CSS selector" |
match |
Not there yet: clone(), raw_html, raw_value, html_pretty and inner_html_pretty, select() and its filters, scripts_contain, script_srcs_contain, template_fragments, text_lexbor, mem_id, tag_id, parser.inner_html, the :lexbor-contains() pseudo-class, and create_tag() and extract_html_comment() at module level.
HTMLParser(html, is_fragment=False, fragment_tag="div", fragment_namespace="html",
scripting=False, budget=None, max_depth=None, url=None,
positions=False, errors=False)| Argument | |
|---|---|
html |
str, or bytes read as UTF-8 |
is_fragment |
parse as the contents of a fragment_tag element, as innerHTML does: HTMLParser("<td>1", is_fragment=True, fragment_tag="tr")
|
fragment_namespace |
"html", "svg" or "math" (or their namespace URLs), for the fragment's context element |
scripting |
parse <noscript> as a browser with JavaScript on does |
budget |
give up after this many seconds and raise TimeoutError; see Limits and hostile input
|
max_depth |
keep the tree at most this deep, as Chromium does at 512 |
url |
the page's address, so :target and :local-link match |
positions |
record where each node and attribute was written; see Positions |
errors |
keep the parse errors, for errors; see Parse errors
|
| Member | Gives |
|---|---|
root |
the <html> element; a fragment's first node; None for an empty fragment |
head, body
|
those elements, or None (always None for a fragment) |
html |
the whole document as markup |
text(deep=True, separator="", strip=False, skip_empty=False) |
the text, as Node.text
|
css(query) |
every matching element, in document order |
css_first(query, default=None, strict=False) |
the first match, or default; with strict, a ValueError if there's more than one |
css_matches(selector) |
whether anything matches |
any_css_matches(selectors) |
whether any of them matches anything |
tags(name) |
every element of that name |
xpath(query, **variables) |
an XPath 1.0 query from the document node; see XPath |
errors |
the parse errors, a list of ParseError
|
line_col(offset) |
the (line, column) of an offset into the input, counted as source_range counts |
create_node(tag) |
a new element, not yet in the tree |
merge_text_nodes() |
joins each run of adjacent text nodes into one |
strip_tags(tags, recursive=False) |
removes every element matching each selector in tags, contents and all |
unwrap_tags(tags, delete_empty=False) |
replaces each match with its contents |
A node is its parser plus a position in the tree, so it stays valid after any edit, and two Node objects for the same node are equal and hash the same.
| Member | Gives |
|---|---|
tag |
the element's name (div, foreignObject); -text, -comment, -doctype, -document for those |
id |
the id attribute, or None
|
attributes |
a new dict of the attributes, names as written (xlink:href, viewBox) |
attrs |
the live Attributes mapping |
html |
the node and its contents as markup (outerHTML) |
inner_html |
the contents as markup; set it to replace them, parsed in this element's context |
text(deep=True, separator="", strip=False, skip_empty=False) |
the text nodes' text, joined by separator; deep=False takes only the node's own text children; strip strips each piece; skip_empty drops whitespace-only ones |
text_content |
a text node's text, else None
|
comment_content |
a comment's text, stripped, else None
|
parent, next, prev, first_child, last_child, child
|
the neighbours, text and comments included; None at the edges (a fragment's top-level nodes have no parent) |
iter(include_text=False, skip_empty=False) |
the children: elements only, unless include_text, which adds text and comments |
traverse(include_text=False, skip_empty=False) |
the node and everything under it, in tree order, filtered the same way |
css(query), css_first(...), css_matches(...), any_css_matches(...)
|
as on the parser, but under this node, and counting the node itself, as selectolax does |
xpath(query, **variables) |
an XPath query with this node as the context, as lxml's element.xpath()
|
source_range, end_tag_range, position, attribute_range(name)
|
where the node was written; see Positions |
decompose(), remove()
|
take the node out of the tree; it stays readable |
unwrap(delete_empty=False) |
replaces the element with its contents; an empty one stays unless delete_empty
|
unwrap_tags(...), strip_tags(...)
|
as on the parser, under this node |
insert_before(value, move=False), insert_after(...), insert_child(...), replace_with(...)
|
put value beside, inside (last) or in place of the node: a str or bytes goes in as text, never parsed; a Node goes in as a copy, from this page or another, or itself with move=True
|
merge_text_nodes() |
as on the parser, under this node |
is_element_node, is_text_node, is_comment_node, is_document_node, is_empty_text_node
|
what kind of node it is |
parser |
the HTMLParser it belongs to |
To insert markup rather than text, parse it and pass the node: div.insert_child(HTMLParser("<b>new</b>", is_fragment=True).root).
node.attrs reads and writes the element's attributes: attrs["class"], attrs["data-x"] = "1" (None sets it empty), del attrs["hidden"], "id" in attrs, len(attrs), iteration over the names, keys(), values(), items(), get(key, default=None), and sget(key, default=""). On HTML elements names are looked up in lowercase, as getAttribute does. A missing key raises KeyError.
xpath() runs XPath 1.0, on the parser from the document node and on a node from that node, and gives back what lxml gives back, so code written for lxml or parsel ports as it is:
page.xpath("//li[@class='item']/a/text()") # ['Kettle', 'Toaster']
page.xpath("//li/@data-sku") # ['A1', 'B2']
page.xpath("count(//li)") # 2.0
page.xpath("boolean(//table)") # False
item.xpath("string(span[@class='price'])") # '£24'
page.xpath("//li[@data-sku=$sku]", sku="B2") # [<Node li>]| The query gives | You get |
|---|---|
| a node-set | a list in document order: elements, comments and the document as Nodes, text nodes and attributes as str
|
| a string | str |
| a number | float |
| a boolean | bool |
Keyword arguments bind $variables: a str, bytes, int, float or bool. A query that doesn't parse, or uses a $variable you didn't pass, raises XPathError, a ValueError. Names match as lxml matches them in HTML, lowercase and in no namespace: //div finds every div, //DIV none. A <template>'s contents aren't in the tree a query walks. The XPath page has the engine's limits; a query that would run for hours stops with an XPathError.
The first query after the page changes indexes it, once, and later queries reuse the index until the next edit. The index lives on the parser, so threads share it, and a query lets go of the GIL while it runs, as css() does. A node you've removed can still be queried: its subtree is a tree of its own, so //li from it finds the lis under it.
What's different from lxml: no namespaces= (HTML names have no prefixes), no smart_strings (a string is a plain str, without .getparent()), and no compiled etree.XPath object; each call compiles its query, which takes microseconds.
Parse with positions=True and every node the parser read knows where it was in your input:
src = "<p class=note>Hi</p>\n<div>x</div>"
page = HTMLParser(src, positions=True)
p = page.css_first("p")
p.source_range # (0, 14): the start tag, <p class=note>
p.end_tag_range # (16, 20): </p>
p.attribute_range("class") # (3, 13)
page.css_first("div").position # (2, 1)
src[slice(*p.source_range)] # '<p class=note>'| Member | Gives |
|---|---|
source_range |
(start, end) of an element's start tag, from < to >, or of a whole text node, comment or doctype |
end_tag_range |
(start, end) of the end tag that closed the element, or None if it had none |
attribute_range(name) |
(start, end) of one attribute, from its name to the end of its value; the name is looked up as attrs looks it up |
position |
the (line, column) of source_range's start |
HTMLParser.line_col(offset) |
the (line, column) of any offset, in the same units |
Ranges index the input as you passed it, as a (start, end) pair like re.Match.span() gives, so slicing it gives the source: code point indexes into a str (src[start:end], café and emoji included), byte offsets into bytes (data[start:end], even when some of those bytes weren't UTF-8 and were read as U+FFFD). Lines and columns count from 1, as editors and lxml's sourceline do, and a column counts characters. Each (line, column) is worked out from the input when you ask, so it costs a pass over the text before it.
The parser itself counts UTF-8 bytes. For a str with anything beyond ASCII, the package keeps a copy of the text and, on the first range you read, a table of code points per 64 bytes, so each conversion after that looks at 64 bytes at most. ASCII text and bytes need neither, and nothing is kept without positions=True or errors=True.
Everything is None without positions=True, and for nodes nobody wrote: the <html>, <head>, <body> and <tbody> the parser implied, and anything an edit made. Without it the parse is as fast as ever; with it, each node keeps a few bytes more.
Parse with errors=True and parser.errors lists the parse errors the parser recovered from, in source order. Each is a ParseError (a record, not an exception: the page still parsed):
page = HTMLParser("<p>a\n</b>", errors=True)
for e in page.errors:
print(e) # 1:1: expected-doctype-but-got-start-tag
# 2:1: unexpected-end-tag
page.errors[1].code, page.errors[1].offset # ('unexpected-end-tag', 5)code is the HTML Standard's name for the error (eof-in-tag, duplicate-attribute), or, for one tree construction finds, which the standard leaves unnamed, the name html5lib uses. offset indexes the input as source_range does, so src[e.offset:] starts where the error was found; line and column count from 1. Without errors=True the list is empty. Parsing has the detail.
| Raised | When |
|---|---|
parsec_html.SelectorError (a ValueError) |
a selector doesn't parse; the message says where |
parsec_html.XPathError (a ValueError) |
an XPath query doesn't parse, uses a $variable you didn't pass, or runs past the work limit |
TimeoutError |
a parse ran past its budget
|
ValueError |
input over 512 MiB; an edit that would put a node inside itself; css_first(strict=True) with more than one match |
TypeError |
html that isn't str or bytes; inserting something that isn't text or a Node; an XPath variable that isn't a string, number or bool |
bindings/python/bench.py times parsec_html against selectolax's Lexbor backend in one process on the 13 real pages bench/ uses, 24 MB of Wikipedia, MDN, GitHub, the BBC and the rest: parsing, the same 32 selectors the Rust benchmark runs, and a scraper's loop that reads every link's href and text.
Two runs on an Apple M2 Max with CPython 3.14 and selectolax 1.0.2, medians of 20 runs each, at a load of 2.7 to 3.1:
| parsec_html | selectolax | ||
|---|---|---|---|
| Parse all 13 pages | 66.8, 65.6 ms | 105.6, 105.9 ms | 1.6× |
| The 32 selectors over all 13 pages | 283.1, 283.5 ms | 781.4, 793.8 ms | 2.8× |
Every link's href and text |
42.6, 45.0 ms | 50.0, 49.8 ms | 1.1 to 1.2× |
Page by page, parsing is 1.5 to 2.8 times faster (MDN and the one-page HTML spec are the narrowest, near 1.5×, and the New York Times the widest). Selector by selector it's ahead on all 32, from 1.04× on [class] and 1.07× on * to 29× on p:nth-child(2n+1), which costs selectolax 350 ms. The close ones are the selectors that match most of a page: there the time goes on making a Python object for each of 400,000 matches, which both libraries have to do. The match counts agree except on [href], where Lexbor also counts SVG's xlink:href, as the Rust Benchmarks explain.
To run it yourself:
bench/fetch-pages.sh
cd bindings/python && maturin develop --release && pip install selectolax
python bench.pyThe release builds a wheel for each CPython from 3.9 to 3.15 and for free-threaded 3.14t and 3.15t, on Linux (glibc and musl) and macOS, x86-64 and ARM, and Windows x86-64 and ARM. One more wheel uses the stable ABI that 3.15 brought for both builds of CPython (abi3t), so a newer Python than any of those still installs. That one is slower at making Python objects (31 ms against 22 ms for 400,000 nodes), which is why the per-version wheels come first.
cd bindings/python
pip install maturin pytest
maturin develop --release
pytest tests
bench/fetch-pages.sh && pip install selectolax && python bench.pyCI builds and tests the wheel on Linux, macOS and Windows with 3.9, 3.14 and 3.14t, and on Linux with 3.15 and the abi3t wheel on 3.15 and 3.15t.
Parsec is Apache-2.0 and not yet released; these pages describe main. Found something wrong here? Open an issue or ask in Discussions.
Repository · Issues · Discussions · Roadmap · Security policy · Licence