Repository navigation
Python
Parsec's Python package is parsec-html, and the module you import is parsec_html. Its API is selectolax's, so moving over is mostly a change of import, and it adds what selectolax can't do: one parsed page shared by many threads, free-threaded Python with the GIL off, selectors selectolax can't run, such as :lang(), :dir() and :valid, XPath 1.0 that answers the way lxml's does, where every node was written in your input, the parse errors, and bytes in any encoding, found the way a browser finds it.
On this page: Install · A first page · Threads · Moving from selectolax · HTMLParser · Node · Attributes · XPath · Bytes and encodings · Positions · Parse errors · Errors · Speed · Wheels · Working on it
Note
Parsec isn't released. Wheels go to PyPI with 0.1.0; until then you build from git, which needs Rust 1.88 or newer.
pip install "git+https://github.com/parsec-html/parsec#subdirectory=bindings/python"It runs on CPython 3.9 and later, free-threaded 3.14t and later included (PyO3, which the package is built with, doesn't support free-threaded 3.13). The package ships type stubs, so your editor and mypy know every signature on this page.
Why parsec_html and not parsec? There's already a parsec on PyPI (a parser-combinator library), and two packages that install the same module name overwrite each other. parsec_html matches the name you pip install, so nothing clashes.
from parsec_html import HTMLParser
page = HTMLParser("""
<title>Shop</title>
<ul>
<li class=item data-sku=A1><a href=/a>Kettle</a> <span class=price>£24</span>
<li class=item data-sku=B2><a href=/b>Toaster</a> <span class=price>£31</span>
</ul>
""")
for item in page.css("li.item"):
link = item.css_first("a")
print(item.attrs["data-sku"], link.text(), link.attributes["href"],
item.css_first(".price").text())
page.css_first("li[data-sku=B2]").decompose()
print(page.body.html)A parsed page belongs to no thread. Read it from as many as you like, and edit it from any of them: readers share it, and an edit waits for them, then has it to itself. Parsing and css() let go of the GIL while they work, so on ordinary CPython two threads parsing or querying really do run at once.
On a free-threaded build (3.14t, 3.15t) the module declares that it doesn't need the GIL, so importing it leaves the GIL off:
import sys, threading
from parsec_html import HTMLParser
page = HTMLParser(open("big.html").read())
counts = []
threads = [threading.Thread(target=lambda: counts.append(len(page.css("a")))) for _ in range(8)]
for t in threads: t.start()
for t in threads: t.join()
assert not sys._is_gil_enabled() # on 3.14tselectolax documents its Lexbor parsers as not thread-safe: one parser per thread. Underneath, Parsec's Document is Send and Sync, checked when it compiles; see Thread safety and memory safety.
# from selectolax.lexbor import LexborHTMLParser as HTMLParser
from parsec_html import HTMLParserThese were checked against selectolax 1.0.2 by running the same calls through both. Everything below this table works the way selectolax's does, except where it says so.
| selectolax 1.0.2 | parsec_html | |
|---|---|---|
<a href>'s attributes
|
{'href': None} |
{'href': ''}. Parsec's tree doesn't yet record whether a value was written. |
| A selector that doesn't parse | SelectolaxError |
SelectorError, a ValueError
|
A node after decompose()
|
freed; reading it gives an empty element | still there, detached: readable, and can be inserted again |
attrs.items(), keys(), values()
|
iterators | lists |
iter(), traverse()
|
generators | iterators over a list taken when you call them |
bytes in |
UTF-8, or sniffed with encoding=True
|
sniffed as a browser sniffs them; see Bytes and encodings |
| Class names |
LexborHTMLParser, LexborNode, LexborAttributes
|
HTMLParser, Node, Attributes
|
:lang(), :dir(), :valid, :in-range, :scope
|
SelectolaxError: "Can't parse CSS selector" |
match |
Not there yet: clone(), raw_html, raw_value, html_pretty and inner_html_pretty, select() and its filters, scripts_contain, script_srcs_contain, template_fragments, text_lexbor, mem_id, tag_id, parser.inner_html, the :lexbor-contains() pseudo-class, and create_tag() and extract_html_comment() at module level.
HTMLParser(html, is_fragment=False, fragment_tag="div", fragment_namespace="html",
scripting=False, budget=None, max_depth=None, url=None,
positions=False, errors=False,
transport_encoding=None, default_encoding=None)| Argument | |
|---|---|
html |
str, or bytes decoded as a browser decodes a page; see Bytes and encodings
|
is_fragment |
parse as the contents of a fragment_tag element, as innerHTML does: HTMLParser("<td>1", is_fragment=True, fragment_tag="tr")
|
fragment_namespace |
"html", "svg" or "math" (or their namespace URLs), for the fragment's context element |
scripting |
parse <noscript> as a browser with JavaScript on does |
budget |
give up after this many seconds and raise TimeoutError; see Limits and hostile input
|
max_depth |
keep the tree at most this deep, as Chromium does at 512 |
url |
the page's address, so :target and :local-link match |
positions |
record where each node and attribute was written; see Positions |
errors |
keep the parse errors, for errors; see Parse errors
|
transport_encoding |
for bytes: the encoding the transport layer gave, such as the charset of an HTTP Content-Type, by any label the Encoding Standard knows; one it doesn't is passed over |
default_encoding |
for bytes: the encoding when nothing names one and they aren't valid UTF-8, "windows-1252" unless given |
| Member | Gives |
|---|---|
root |
the <html> element; a fragment's first node; None for an empty fragment |
head, body
|
those elements, or None (always None for a fragment) |
html |
the whole document as markup |
text(deep=True, separator="", strip=False, skip_empty=False) |
the text, as Node.text
|
css(query) |
every matching element, in document order |
css_first(query, default=None, strict=False) |
the first match, or default; with strict, a ValueError if there's more than one |
css_matches(selector) |
whether anything matches |
any_css_matches(selectors) |
whether any of them matches anything |
tags(name) |
every element of that name |
xpath(query, **variables) |
an XPath 1.0 query from the document node; see XPath |
errors |
the parse errors, a list of ParseError
|
encoding |
the Encoding Standard's name for what bytes were decoded from ("UTF-8", "Shift_JIS"), what a browser's document.characterSet says; None for a str
|
encoding_confidence |
"certain" or "tentative", as the HTML Standard says; None for a str
|
line_col(offset) |
the (line, column) of an offset into the input, counted as source_range counts |
create_node(tag) |
a new element, not yet in the tree |
merge_text_nodes() |
joins each run of adjacent text nodes into one |
strip_tags(tags, recursive=False) |
removes every element matching each selector in tags, contents and all |
unwrap_tags(tags, delete_empty=False) |
replaces each match with its contents |
A node is its parser plus a position in the tree, so it stays valid after any edit, and two Node objects for the same node are equal and hash the same.
| Member | Gives |
|---|---|
tag |
the element's name (div, foreignObject); -text, -comment, -doctype, -document for those |
id |
the id attribute, or None
|
attributes |
a new dict of the attributes, names as written (xlink:href, viewBox) |
attrs |
the live Attributes mapping |
html |
the node and its contents as markup (outerHTML) |
inner_html |
the contents as markup; set it to replace them, parsed in this element's context |
text(deep=True, separator="", strip=False, skip_empty=False) |
the text nodes' text, joined by separator; deep=False takes only the node's own text children; strip strips each piece; skip_empty drops whitespace-only ones |
text_content |
a text node's text, else None
|
comment_content |
a comment's text, stripped, else None
|
parent, next, prev, first_child, last_child, child
|
the neighbours, text and comments included; None at the edges (a fragment's top-level nodes have no parent) |
iter(include_text=False, skip_empty=False) |
the children: elements only, unless include_text, which adds text and comments |
traverse(include_text=False, skip_empty=False) |
the node and everything under it, in tree order, filtered the same way |
css(query), css_first(...), css_matches(...), any_css_matches(...)
|
as on the parser, but under this node, and counting the node itself, as selectolax does |
xpath(query, **variables) |
an XPath query with this node as the context, as lxml's element.xpath()
|
source_range, end_tag_range, position, attribute_range(name)
|
where the node was written; see Positions |
decompose(), remove()
|
take the node out of the tree; it stays readable |
unwrap(delete_empty=False) |
replaces the element with its contents; an empty one stays unless delete_empty
|
unwrap_tags(...), strip_tags(...)
|
as on the parser, under this node |
insert_before(value, move=False), insert_after(...), insert_child(...), replace_with(...)
|
put value beside, inside (last) or in place of the node: a str or bytes goes in as text, never parsed; a Node goes in as a copy, from this page or another, or itself with move=True
|
merge_text_nodes() |
as on the parser, under this node |
is_element_node, is_text_node, is_comment_node, is_document_node, is_empty_text_node
|
what kind of node it is |
parser |
the HTMLParser it belongs to |
To insert markup rather than text, parse it and pass the node: div.insert_child(HTMLParser("<b>new</b>", is_fragment=True).root).
node.attrs reads and writes the element's attributes: attrs["class"], attrs["data-x"] = "1" (None sets it empty), del attrs["hidden"], "id" in attrs, len(attrs), iteration over the names, keys(), values(), items(), get(key, default=None), and sget(key, default=""). On HTML elements names are looked up in lowercase, as getAttribute does. A missing key raises KeyError.
xpath() runs XPath 1.0, on the parser from the document node and on a node from that node, and gives back what lxml gives back, so code written for lxml or parsel ports as it is:
page.xpath("//li[@class='item']/a/text()") # ['Kettle', 'Toaster']
page.xpath("//li/@data-sku") # ['A1', 'B2']
page.xpath("count(//li)") # 2.0
page.xpath("boolean(//table)") # False
item.xpath("string(span[@class='price'])") # '£24'
page.xpath("//li[@data-sku=$sku]", sku="B2") # [<Node li>]| The query gives | You get |
|---|---|
| a node-set | a list in document order: elements, comments and the document as Nodes, text nodes and attributes as str
|
| a string | str |
| a number | float |
| a boolean | bool |
Keyword arguments bind $variables: a str, bytes, int, float or bool. A query that doesn't parse, or uses a $variable you didn't pass, raises XPathError, a ValueError. Names match as lxml matches them in HTML, lowercase and in no namespace: //div finds every div, //DIV none. A <template>'s contents aren't in the tree a query walks. The XPath page has the engine's limits; a query that would run for hours stops with an XPathError.
The first query after the page changes indexes it, once, and later queries reuse the index until the next edit. The index lives on the parser, so threads share it, and a query lets go of the GIL while it runs, as css() does. A node you've removed can still be queried: its subtree is a tree of its own, so //li from it finds the lis under it.
What's different from lxml: no namespaces= (HTML names have no prefixes), no smart_strings (a string is a plain str, without .getparent()), and no compiled etree.XPath object; each call compiles its query, which takes microseconds.
A str is text already, so it's parsed as it is. bytes are what came over the network, and Parsec decodes them the way a browser does, by the HTML Standard's encoding sniffing:
- A byte order mark, if the bytes start with one.
-
transport_encoding, if you pass one the Encoding Standard knows. This is thecharsetfrom the HTTPContent-Type; a browser passes over one it can't use (charset=utf8mb4, say), and so does Parsec. - A
<meta charset>or<meta http-equiv=Content-Type>in the first 1,024 bytes. - UTF-8, if the bytes have anything beyond ASCII and are all valid UTF-8. This is the standard's autodetection step, narrowed to the one guess Firefox's and Chrome's detectors both make; legacy text is almost never valid UTF-8 by accident.
-
default_encoding, windows-1252 unless you say otherwise.
If the parser then meets a <meta> naming another encoding (one past the first 1,024 bytes, say), it parses the page again in that one, once, as a browser reloads.
import urllib.request
from parsec_html import HTMLParser
with urllib.request.urlopen(url) as r:
page = HTMLParser(r.read(), transport_encoding=r.headers.get_content_charset())
page.encoding # 'Shift_JIS'
page.encoding_confidence # 'certain'encoding_confidence is "certain" when a byte order mark, the transport encoding or a <meta> the parser met decided it, and "tentative" when only the prescan, the UTF-8 detection or the default did. Labels are the Encoding Standard's, so "sjis", "latin1" and " UTF8 " all work, and latin1 means windows-1252, as it does on the web.
The default a browser uses depends on its language. encoding_for_locale("ja") gives "Shift_JIS", encoding_for_locale("ru-RU") "windows-1251" and encoding_for_locale("zh-Hant") "Big5", from the HTML Standard's table, ready to pass as default_encoding when you know who a site was written for.
A fragment (is_fragment=True) of bytes is sniffed the same way, without the second parse, which only a document has. transport_encoding or default_encoding with a str raises TypeError, as html5lib does: there's nothing to decode.
Note
Before this, bytes were read as UTF-8 whatever they were. UTF-8 still comes out as UTF-8, declared or not. What changed is bytes that aren't valid UTF-8 and don't say what they are: each bad byte used to become U+FFFD, and now the page decodes as windows-1252, as in a browser, so caf\xe9 reads café. Pass transport_encoding when you know better.
StreamParser parses bytes as they arrive. feed() each piece, then close() gives the HTMLParser, the same one HTMLParser(b"".join(pieces), ...) gives, however the bytes were cut, mid-character included:
from parsec_html import StreamParser
with urllib.request.urlopen(url) as r:
s = StreamParser(transport_encoding=r.headers.get_content_charset())
while chunk := r.read(65536):
s.feed(chunk)
page = s.close()It takes HTMLParser's keywords except the page and the fragment ones. The names feed and close are lxml's feed parser's and the standard library's html.parser's, so code written for either reads the same; close() returns the parsed page as lxml's returns its root. After close(), feed() and close() raise ValueError. Both let go of the GIL while they work.
Until the encoding is known, the parser holds the bytes back: a byte order mark or a transport encoding decides at once, a <meta> as soon as the prescan has read all of it, and otherwise it waits for 1,024 bytes. While the encoding is only a guess it keeps every byte, in case a late <meta> means parsing again, so pass transport_encoding when you have it.
The 82 html5lib-tests encoding/ cases that don't need a script engine run through both, HTMLParser whole and StreamParser in pieces of 1 to 7 bytes, and all 82 pass; the one left, in scripted/, writes its <meta> with document.write. The Encoding page has the decoders' own conformance.
Parse with positions=True and every node the parser read knows where it was in your input:
src = "<p class=note>Hi</p>\n<div>x</div>"
page = HTMLParser(src, positions=True)
p = page.css_first("p")
p.source_range # (0, 14): the start tag, <p class=note>
p.end_tag_range # (16, 20): </p>
p.attribute_range("class") # (3, 13)
page.css_first("div").position # (2, 1)
src[slice(*p.source_range)] # '<p class=note>'| Member | Gives |
|---|---|
source_range |
(start, end) of an element's start tag, from < to >, or of a whole text node, comment or doctype |
end_tag_range |
(start, end) of the end tag that closed the element, or None if it had none |
attribute_range(name) |
(start, end) of one attribute, from its name to the end of its value; the name is looked up as attrs looks it up |
position |
the (line, column) of source_range's start |
HTMLParser.line_col(offset) |
the (line, column) of any offset, in the same units |
Ranges index the input as you passed it, as a (start, end) pair like re.Match.span() gives, so slicing it gives the source: code point indexes into a str (src[start:end], café and emoji included), byte offsets into bytes (data[start:end]) in whatever encoding they were: a Shift_JIS kanji is two bytes of the range, a UTF-16 character two or four, a byte order mark is counted, and so is every byte the decoder replaced with U+FFFD. Lines and columns count from 1, as editors and lxml's sourceline do, and a column counts characters. Each (line, column) is worked out from the input when you ask, so it costs a pass over the text before it.
The parser itself counts UTF-8 bytes into the text it parsed. For a str with anything beyond ASCII, the package keeps a copy of the text and, on the first range you read, a table of code points per 64 bytes, so each conversion after that looks at 64 bytes at most. For bytes that aren't plain UTF-8, it decodes them once more after the parse (in one pass for single-byte encodings, a byte at a time for Shift_JIS and the other multi-byte ones, which takes two or three times as long as the parse), noting each place where the bytes and the text stop moving in step: one entry per character whose length differs (8 bytes each, so a page of Japanese text costs about 4 bytes per byte of it), none for ASCII. ASCII text and UTF-8 bytes need nothing, and nothing is kept without positions=True or errors=True.
Everything is None without positions=True, and for nodes nobody wrote: the <html>, <head>, <body> and <tbody> the parser implied, and anything an edit made. Without it the parse is as fast as ever; with it, each node keeps a few bytes more.
Parse with errors=True and parser.errors lists the parse errors the parser recovered from, in source order. Each is a ParseError (a record, not an exception: the page still parsed):
page = HTMLParser("<p>a\n</b>", errors=True)
for e in page.errors:
print(e) # 1:1: expected-doctype-but-got-start-tag
# 2:1: unexpected-end-tag
page.errors[1].code, page.errors[1].offset # ('unexpected-end-tag', 5)code is the HTML Standard's name for the error (eof-in-tag, duplicate-attribute), or, for one tree construction finds, which the standard leaves unnamed, the name html5lib uses. offset indexes the input as source_range does, so src[e.offset:] starts where the error was found; line and column count from 1. Without errors=True the list is empty. Parsing has the detail.
| Raised | When |
|---|---|
parsec_html.SelectorError (a ValueError) |
a selector doesn't parse; the message says where |
parsec_html.XPathError (a ValueError) |
an XPath query doesn't parse, uses a $variable you didn't pass, or runs past the work limit |
TimeoutError |
a parse ran past its budget
|
ValueError |
input over 512 MiB; an edit that would put a node inside itself; css_first(strict=True) with more than one match; a default_encoding the Encoding Standard doesn't know; feed() or close() on a closed StreamParser
|
TypeError |
html that isn't str or bytes; transport_encoding or default_encoding with a str; inserting something that isn't text or a Node; an XPath variable that isn't a string, number or bool |
bindings/python/bench.py times parsec_html against selectolax's Lexbor backend in one process on the 13 real pages bench/ uses, 24 MB of Wikipedia, MDN, GitHub, the BBC and the rest: parsing, the same 32 selectors the Rust benchmark runs, and a scraper's loop that reads every link's href and text.
Two runs on an Apple M2 Max with CPython 3.14 and selectolax 1.0.2, medians of 20 runs each, at a load of 2.7 to 3.1:
| parsec_html | selectolax | ||
|---|---|---|---|
| Parse all 13 pages | 66.8, 65.6 ms | 105.6, 105.9 ms | 1.6× |
| The 32 selectors over all 13 pages | 283.1, 283.5 ms | 781.4, 793.8 ms | 2.8× |
Every link's href and text |
42.6, 45.0 ms | 50.0, 49.8 ms | 1.1 to 1.2× |
Page by page, parsing is 1.5 to 2.8 times faster (MDN and the one-page HTML spec are the narrowest, near 1.5×, and the New York Times the widest). Selector by selector it's ahead on all 32, from 1.04× on [class] and 1.07× on * to 29× on p:nth-child(2n+1), which costs selectolax 350 ms. The close ones are the selectors that match most of a page: there the time goes on making a Python object for each of 400,000 matches, which both libraries have to do. The match counts agree except on [href], where Lexbor also counts SVG's xlink:href, as the Rust Benchmarks explain.
To run it yourself:
bench/fetch-pages.sh
cd bindings/python && maturin develop --release && pip install selectolax
python bench.pyThe release builds a wheel for each CPython from 3.9 to 3.15 and for free-threaded 3.14t and 3.15t, on Linux (glibc and musl) and macOS, x86-64 and ARM, and Windows x86-64 and ARM. One more wheel uses the stable ABI that 3.15 brought for both builds of CPython (abi3t), so a newer Python than any of those still installs. That one is slower at making Python objects (31 ms against 22 ms for 400,000 nodes), which is why the per-version wheels come first.
cd bindings/python
pip install maturin pytest
maturin develop --release
pytest tests
bench/fetch-pages.sh && pip install selectolax && python bench.pyCI builds and tests the wheel on Linux, macOS and Windows with 3.9, 3.14 and 3.14t, and on Linux with 3.15 and the abi3t wheel on 3.15 and 3.15t.
Parsec is Apache-2.0 and not yet released; these pages describe main. Found something wrong here? Open an issue or ask in Discussions.
Repository · Issues · Discussions · Roadmap · Security policy · Licence