Repository navigation
v2.14.0 — faster XML reads and compressed input (thanks @bendrucker)
This release is @bendrucker's work. #212 halves the time to read XML and adds compressed input. Thank you, Ben, for the change, for the measurements behind every step of it, and for turning around the review fixes in two days.
Faster reads
Ben measured this branch against main over 68 benchmark cases, with interleaved runs and output checked against fingerprints. These are the geomean changes:
| DuckDB | Wall | CPU | Peak RSS |
|---|---|---|---|
| 2.0 | -50.6% | -42.5% | -7.6% |
| 1.5.6 | -51.9% | -44.6% | -7.8% |
Some real datasets on DuckDB 2.0:
- 5,000 IRS 990 files under
union_by_name: 21.99 s to 3.64 s. - ClinVar streamed: 12.23 s to 4.32 s, and peak memory 2.6 GB to 1.6 GB.
bench/CLIMB.md records each of the 22 accepted changes with its measurement, and the ideas he rejected. The main changes:
- Streaming (SAX) builds each record as a small DOM subtree and converts it with the DOM code, so both paths return the same rows.
- Bind-time inference over many files runs in parallel and merges in file order. The new
xml_inference_threadssetting overrides the thread count. - Values go straight into result vectors, and hand-written scanners replace
std::regex. - In-memory parses use one libxml2 parser context per thread. The extension still sets no libxml2 globals.
New: compressed input
read_xml, read_html and their _objects variants read gzip, and zstd once parquet is loaded. The new compression parameter overrides detection.
Behaviour changes
- A plain file whose name ends in
.gznow needscompression := 'none'. - Streaming reads honour
root_elementandmax_depth := 0, and keep whitespace between inline elements, to match DOM. - A
record_elementthat needs XPath and goes past libxml2's 10M-node cap raises an out-of-memory error. A bare element name reads every record. - On the SAX path, one record may span at most a quarter of
memory_limit. The span includes its start tag and expanded entity references. An inference sample stops at the same size, so a field first seen after that needscolumns. (#212, #215) - Under
ignore_errors, the rows a SAX read holds count againstmemory_limitand can spill to disk (#215). duck_blocks_to_htmldrops attribute keys that aren't valid HTML names, and escapes</scriptand<!--inmetadatacontent (#216).html_escape()keeps the text after a NUL byte (#216).
Hardening
Reviews of #212 and of the duck_block module found the following. Claude (Opus) did the reviews.
- A gzip file under 2 MB could hold 2 to 4 times
memory_limiton the SAX path. #215 finishes the record cap Ben added in #212. - A crafted
duck_blockcould inject script throughduck_blocks_to_html. #216 fixes this, and the docs now say the writer escapes but does not sanitize. - #217 tracks the remaining duck_block findings.
What's Changed
- feat: faster XML reads and compressed input by @bendrucker in #212
- fix(sax): bound the four paths past the streamed-record memory cap (#214) in #215
- fix(duck_block): stop block fields writing markup in duck_blocks_to_html in #216
- ci(canary): resolve duckdb main to a (commit, version) pair and restore the tests in #213
- docs(changelog): v2.14.0, and backfill v2.9.3 through v2.13.0 in #218
Full Changelog: v2.13.0...v2.14.0