Skip to content

v2.14.0 — faster XML reads and compressed input (thanks @bendrucker)

Choose a tag to compare

@teaguesterling teaguesterling released this 02 Nov 03:07
18c2223

This release is @bendrucker's work. #212 halves the time to read XML and adds compressed input. Thank you, Ben, for the change, for the measurements behind every step of it, and for turning around the review fixes in two days.

Faster reads

Ben measured this branch against main over 68 benchmark cases, with interleaved runs and output checked against fingerprints. These are the geomean changes:

DuckDB Wall CPU Peak RSS
2.0 -50.6% -42.5% -7.6%
1.5.6 -51.9% -44.6% -7.8%

Some real datasets on DuckDB 2.0:

  • 5,000 IRS 990 files under union_by_name: 21.99 s to 3.64 s.
  • ClinVar streamed: 12.23 s to 4.32 s, and peak memory 2.6 GB to 1.6 GB.

bench/CLIMB.md records each of the 22 accepted changes with its measurement, and the ideas he rejected. The main changes:

  • Streaming (SAX) builds each record as a small DOM subtree and converts it with the DOM code, so both paths return the same rows.
  • Bind-time inference over many files runs in parallel and merges in file order. The new xml_inference_threads setting overrides the thread count.
  • Values go straight into result vectors, and hand-written scanners replace std::regex.
  • In-memory parses use one libxml2 parser context per thread. The extension still sets no libxml2 globals.

New: compressed input

read_xml, read_html and their _objects variants read gzip, and zstd once parquet is loaded. The new compression parameter overrides detection.

Behaviour changes

  • A plain file whose name ends in .gz now needs compression := 'none'.
  • Streaming reads honour root_element and max_depth := 0, and keep whitespace between inline elements, to match DOM.
  • A record_element that needs XPath and goes past libxml2's 10M-node cap raises an out-of-memory error. A bare element name reads every record.
  • On the SAX path, one record may span at most a quarter of memory_limit. The span includes its start tag and expanded entity references. An inference sample stops at the same size, so a field first seen after that needs columns. (#212, #215)
  • Under ignore_errors, the rows a SAX read holds count against memory_limit and can spill to disk (#215).
  • duck_blocks_to_html drops attribute keys that aren't valid HTML names, and escapes </script and <!-- in metadata content (#216).
  • html_escape() keeps the text after a NUL byte (#216).

Hardening

Reviews of #212 and of the duck_block module found the following. Claude (Opus) did the reviews.

  • A gzip file under 2 MB could hold 2 to 4 times memory_limit on the SAX path. #215 finishes the record cap Ben added in #212.
  • A crafted duck_block could inject script through duck_blocks_to_html. #216 fixes this, and the docs now say the writer escapes but does not sanitize.
  • #217 tracks the remaining duck_block findings.

What's Changed

  • feat: faster XML reads and compressed input by @bendrucker in #212
  • fix(sax): bound the four paths past the streamed-record memory cap (#214) in #215
  • fix(duck_block): stop block fields writing markup in duck_blocks_to_html in #216
  • ci(canary): resolve duckdb main to a (commit, version) pair and restore the tests in #213
  • docs(changelog): v2.14.0, and backfill v2.9.3 through v2.13.0 in #218

Full Changelog: v2.13.0...v2.14.0