Repository navigation
Frostwork 0.2.0
Backward-incompatible changes
-
A document with no BOM, label or declaration is no longer assumed to be UTF-8: it is UTF-8 if it is
valid UTF-8, and otherwise its encoding is autodetected, as Firefox does. Pass the response URL as
urltoextract,extract_grouped,detect_encoding,Plan,Page.extractand
frostwork.encoding.autodetect_encodingso that its top-level domain informs the guess (.jp
favours Shift_JIS,.ruwindows-1251, …), as it does in browsers. -
In Rust,
extract,extract_grouped,detect_encoding,Plan::extract,
Page::extract_with_encodingandCompiledPage::extract_with_encodingtake the response URL as a new
url: Option<&str>argument after the encoding. PassNonewhen there is no URL. -
bgsound,embed,keygen,source,trackandwbrare now void elements, matching libxml2
2.15, the oracle Frostwork now targets. Markup after one of them is no longer its child, e.g.
<div><source><span>x</span></div>matchesdiv > spaninstead ofsource span.
Python API and tooling
-
frostwork.encodingexposes each step ofdetect_encoding()(bom_encoding(),
declared_encoding(),autodetect_encoding()), plusdecode(), which decodes as browsers do where
Python's codecs differ (e.g.euc_jp), andcodec_name(), which gives the Python codec name of any
labelresolve_label()resolves. It lets a library that keeps its own encoding API, such as w3lib,
give browser answers. Importing it registers Python codecs forx-user-definedandreplacement.
See PYTHON.md. -
frostwork.navigation()reads the base URL and meta refresh of a document as a browser does, and
Page.extract_with_navigation()reads them in the same pass as the fields; in Rust,navigation()
andPlan::extract_with_navigation(). See
PYTHON.md. -
The refusal reason of a selector with an unsupported pseudo-class or pseudo-element now names it,
e.g.pseudo-class `:hover` is unsupported, also when it sits next to a supported one
(li:nth-child(2):hover). -
frostwork.page.detect_encoding()andfrostwork.page.resolve_label()are deprecated; use
frostwork.detect_encoding()andfrostwork.resolve_label().
Correctness
-
The
<meta>prescan skips other tags together with their attributes, as browsers do, so a
<meta charset>inside an attribute value (<img alt="<meta charset=big5>">) no longer declares the
encoding. -
A selector deferred on an ancestor (
:has(),:contains(), a reverse position such as:last-child
or[last()]) followed by a descendant step that the ancestor itself could match, such as*or its
own tag, no longer returns that ancestor, e.g.li:last-child *returned the<li>as well as its
descendants, and//li[last()]//*/@classitsclass.