Repository navigation
v4.2.0 — the compact duck_blocks line format; dense element_order from duck_blocks_normalize
LatestMINOR. The release adds a compact line format for duck_blocks, as three SQL macros and a vendorable C++ header, and it fixes two producer defects. The spec stays at 1.4.
Added
- The compact line format (#86, #90, #91, #93, #94). It writes one JSON object per line, one line per element. The fields are one letter (
tcleam), the types and attribute keys are short codes, and NULL and default fields are left out. The line number is theelement_order. Measured on 16 duckdb_panduck fixtures, it is 0.53 of pandoc JSON raw and 0.92 with zstd, against 1.68 and 1.37 forCOPY ... (FORMAT json)of the same rows.duck_blocks_to_compact(blocks)andduck_blocks_from_compact(text)convert to and from a duck_block list.read_compact_blocks(path)reads one or more files as rows, and a glob works.- Type codes: a block is UPPERCASE, an inline is lowercase and a value is a symbol. All 50 types in the spec 1.4 vocabulary have a code. Any other type is written as
kind:element_type. - The header line is
{"@":"4.2.0"}, the version of the code tables, a constant. Codes are only ever added. A reader refuses a type code or an attribute code it does not know, so a file from a newer table fails loudly instead of losing a field. - The writer refuses a NULL
level,encoding,attributesmap or attribute value, rather than writing a default. - The readers stream. No macro has a subquery body, so the cost grows with the file (measured: 486,000 lines in 4 files read in 1.4 s).
read_compact_blocksnumbers lines in scan order, so it refuses to run withpreserve_insertion_orderoff. vendor/duck_block_compact.hppholds the code tables,TABLE_VERSIONand a C++ codec on DuckDB's bundled yyjson. The SQL macros are built from its tables at load. duckdb_panduck vendors it for its native.dblocksreader and writer.test/fixtures/compact/goldenis the corpus both implementations are held to.
Fixed
duck_blocks_normalizekeepselement_orderdense (#89, duckdb_panduck#206). Folding a loneplaininto its parent left a gap: a two-item tight list came back as 0, 1, 3, 5.- The legacy
duck_block_heading(content, level)carries level 1 (#87, #88). It returned a NULL level, so its block failedduck_blocks_validate. It was the only one of 116 builder overloads that did. test/check_vendorable.pycompiles as C++17 (#92). DuckDB v2.0's own headers do not compile as C++11, so the check failed on every tree.
Measured
CI run 44802413402 on the merge commit c978e07, after a re-run of the canary jobs. In the first attempt, canary linux_arm64 failed to download a Yum repository's metadata while building its Docker image, and canary linux_amd64 was cancelled.
| Build | Tests |
|---|---|
| v2.0.0 pin: linux_amd64, windows_amd64 | 18 passed, 0 skipped |
| v2.0 next: linux_amd64, windows_amd64 | 18 passed, 0 skipped |
| v2.0.0 pin and v2.0 next: osx_arm64 | 16 passed, 2 skipped |
| DuckDB main canary: linux_amd64 | 18 passed, 0 skipped. This is the leg #94 was fixed for: before it, compact.test failed here |
| DuckDB main canary: osx_arm64 | 16 passed, 2 skipped |
| DuckDB main canary: windows_amd64 | fails, in DuckDB. MSVC stops in DuckDB's own duckdb/storage/arena_allocator.hpp(160): error C2665 while compiling DuckDB main. Canary windows_amd64_mingw was cancelled |
| wasm_mvp, wasm_eh, wasm_threads, linux_arm64 | build passes. I read the test line only for the rows above. |
The build inputs (vcpkg.json, extension_config.cmake, CMakeLists.txt) are unchanged since v4.1.0.