v0.2.0

htmlarc 0.2.0 decodes legacy-charset pages correctly, brings several selectors in
line with the HTML and Selectors specs, and speeds up parsing and the Python sweeps.
The archive format is still v12, so archives built by 0.1.x open unchanged.

pip install htmlarc
cargo install htmlarc --locked

Charset decoding at ingest

htmlarc-convert used to decode every WARC or directory page as UTF-8, replacing
invalid bytes. Pages in a legacy charset lost their non-ASCII text:

<title>\xcf\xf0\xe8\xe2\xe5\xf2</title>, charset=windows-1251
0.1.1: <title>������</title>
0.2.0: <title>Привет</title>

On one Common Crawl segment that affected 1,068 of 34,465 HTML responses (3.1%). The
converter now checks, in order: a BOM, valid UTF-8, mostly-valid UTF-8, the
Content-Type charset, <meta> labels in the first 4,096 bytes, and finally
statistical detection with the URL's top-level domain as a hint.
ADR 0011
explains the order and compares it with browsers and other tools. Archives built
by 0.1.x keep their damage; rebuild them from the source files to fix it.

Written HTML (to_html, the CLI's .html output) is UTF-8, so it now declares
utf-8 in every <meta> charset label. A page decoded from windows-1251 no longer
shows as Привет when the written file is opened in a browser. Queries still
see the page's own label: meta[charset] returns windows-1251.

Behaviour changes

These can change query results:

• [class…] attribute selectors test the whole class value, as Selectors Level 4
  says. [class~="foo"] now matches exactly what .foo matches.
• Attribute values compare case-insensitively only for the attributes the HTML
  standard lists, such as type, lang and rel. a[href^='http://'] no longer
  matches HTTP://….
• The doctype is no longer an element sibling, so html:first-child and
  p:first-child after a doctype now match, as in lxml.
• :root is the top-level element (<html>), not the document node.
• A compound without a type selector, such as :not(p), no longer matches the
  doctype or comment nodes.
• The document root's Element.tag in Python is "[document]" (BeautifulSoup's
  name) instead of sys_root.
• 93 rare math entities decode to the code points the HTML standard lists:
  ≂̸ is now U+2242 U+0338, where 0.1.x dropped the combining mark
  (html-escape 0.3).

New

• The universal selector *.

Dependencies

• pyo3 0.30 fixes a possible memory leak in Python classes (a missing
  garbage-collector traversal) and slower function calls on CPython 3.11+.
• zstd 0.14.3 and arrow 60.1 are the minimum versions, past RustSec advisories that
  htmlarc's own use couldn't trigger.

Faster

Measured on an Apple M4 Pro. Each figure comes from its PR (#100–#104) and
compares against the code just before it.

• Parsing takes about a third less time (3,000 pages: 1.11 s to 0.73 s). It skips
  a tokenizer error check whose results were discarded, and WARC inflate is 1.6×
  faster with the zlib-rs backend. Converting 20,000 pages: about 10.5 s to 8.0 s.
• Selectors skip elements whose tag can't match before running the matcher. Over
  5,000 Common Crawl pages, a[href] went from 33 ms to 16 ms and h1, h2, h3
  from 39 ms to 6 ms.
• Python scan_* sweeps keep their worker threads and decompression contexts
  between calls. On a 9,567-page wiktionary archive, scan_count takes 25–65%
  less time and scan_table 54% less.

Licensed under AGPL-3.0-only, with separate commercial licensing available.