Skip to content

v4.2.0 — the compact duck_blocks line format; dense element_order from duck_blocks_normalize

Latest

Choose a tag to compare

@teaguesterling teaguesterling released this 20 Nov 20:36
c978e07

MINOR. The release adds a compact line format for duck_blocks, as three SQL macros and a vendorable C++ header, and it fixes two producer defects. The spec stays at 1.4.

Added

  • The compact line format (#86, #90, #91, #93, #94). It writes one JSON object per line, one line per element. The fields are one letter (t c l e a m), the types and attribute keys are short codes, and NULL and default fields are left out. The line number is the element_order. Measured on 16 duckdb_panduck fixtures, it is 0.53 of pandoc JSON raw and 0.92 with zstd, against 1.68 and 1.37 for COPY ... (FORMAT json) of the same rows.
    • duck_blocks_to_compact(blocks) and duck_blocks_from_compact(text) convert to and from a duck_block list. read_compact_blocks(path) reads one or more files as rows, and a glob works.
    • Type codes: a block is UPPERCASE, an inline is lowercase and a value is a symbol. All 50 types in the spec 1.4 vocabulary have a code. Any other type is written as kind:element_type.
    • The header line is {"@":"4.2.0"}, the version of the code tables, a constant. Codes are only ever added. A reader refuses a type code or an attribute code it does not know, so a file from a newer table fails loudly instead of losing a field.
    • The writer refuses a NULL level, encoding, attributes map or attribute value, rather than writing a default.
    • The readers stream. No macro has a subquery body, so the cost grows with the file (measured: 486,000 lines in 4 files read in 1.4 s). read_compact_blocks numbers lines in scan order, so it refuses to run with preserve_insertion_order off.
    • vendor/duck_block_compact.hpp holds the code tables, TABLE_VERSION and a C++ codec on DuckDB's bundled yyjson. The SQL macros are built from its tables at load. duckdb_panduck vendors it for its native .dblocks reader and writer. test/fixtures/compact/golden is the corpus both implementations are held to.

Fixed

  • duck_blocks_normalize keeps element_order dense (#89, duckdb_panduck#206). Folding a lone plain into its parent left a gap: a two-item tight list came back as 0, 1, 3, 5.
  • The legacy duck_block_heading(content, level) carries level 1 (#87, #88). It returned a NULL level, so its block failed duck_blocks_validate. It was the only one of 116 builder overloads that did.
  • test/check_vendorable.py compiles as C++17 (#92). DuckDB v2.0's own headers do not compile as C++11, so the check failed on every tree.

Measured

CI run 44802413402 on the merge commit c978e07, after a re-run of the canary jobs. In the first attempt, canary linux_arm64 failed to download a Yum repository's metadata while building its Docker image, and canary linux_amd64 was cancelled.

Build Tests
v2.0.0 pin: linux_amd64, windows_amd64 18 passed, 0 skipped
v2.0 next: linux_amd64, windows_amd64 18 passed, 0 skipped
v2.0.0 pin and v2.0 next: osx_arm64 16 passed, 2 skipped
DuckDB main canary: linux_amd64 18 passed, 0 skipped. This is the leg #94 was fixed for: before it, compact.test failed here
DuckDB main canary: osx_arm64 16 passed, 2 skipped
DuckDB main canary: windows_amd64 fails, in DuckDB. MSVC stops in DuckDB's own duckdb/storage/arena_allocator.hpp(160): error C2665 while compiling DuckDB main. Canary windows_amd64_mingw was cancelled
wasm_mvp, wasm_eh, wasm_threads, linux_arm64 build passes. I read the test line only for the rows above.

The build inputs (vcpkg.json, extension_config.cmake, CMakeLists.txt) are unchanged since v4.1.0.