Support the Vortex file format

[alexey-milovidov]

Closes: #87327
Related: ClickHouse/rust_vendor#74

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Added support for reading and writing the Vortex columnar file format (the Vortex input and output format). This closes #87327.

Documentation entry for user-facing changes

The implementation uses the Rust vortex crate (v0.83.0) through a new C FFI crate rust/workspace/vortex (_ch_rust_vortex), following the same pattern as prql and polyglot. Data crosses the FFI boundary through the Arrow C Data Interface and is converted with the same ArrowColumnToCHColumn/CHColumnToArrowColumn code as the Arrow format. IO is delegated back to ClickHouse through callbacks: reads go through ClickHouse's own read buffers (range reads for seekable inputs, whole-file buffering otherwise), and the produced file is streamed into the output buffer. All work is driven by a single-threaded runtime on the calling thread — the library spawns no threads, and Rust panics are caught at the FFI boundary and turned into exceptions.

Features:

• Reading with projection pushdown: only the columns used by the query are read from the file.
• Schema inference and count()-only queries answered from file metadata without reading data.
• Writing with the library's default adaptive compression (BtrBlocks-style cascading encodings + zstd), including a valid empty file for empty results.
• Graceful errors on malformed and truncated files (fuzzer-friendly: no aborts, Rust panics become exceptions).

Limitations (documented in docs/reference/formats/Vortex.mdx):

• Map, Int128/UInt128/Int256/UInt256, IPv6, and Interval columns cannot be written (no corresponding Vortex type).
• String and FixedString are written as Vortex Binary (ClickHouse strings are arbitrary bytes, while Vortex requires Utf8 to be valid UTF-8).
• The format is disabled in MSan builds: the MSan-instrumented library (with origin tracking) is so large that linking unit_tests_dbms overflows the 2 GiB R_X86_64_PC32 relocation range (same approach as wasmtime and delta-kernel-rs).
• Reading and writing are single-threaded in this first version. On a 10M-row test table, reads with projection are on par with Parquet (~0.5 s), while writes are noticeably slower (the adaptive compressor samples many encodings per column) — parallelism can be added later.

The 127 new vendored Rust crates are added in ClickHouse/rust_vendor#74 (the contrib/rust_vendor submodule is bumped to that branch).

[mintlify]

Preview deployment for your docs. Learn more about Mintlify Previews.

Project	Status	Preview	Updated (UTC)
ClickHouse-docs	🟢 Ready	View Preview	Aug 1, 2026, 10:03 PM

[clickhouse-gh]

Workflow [PR], commit [e5634f1]

Summary: ✅

• Performance Comparison: Performance dashboard

---

AI Review

Summary

This PR adds Vortex input/output format support through a Rust FFI layer and the existing Arrow conversion bridge. The overall shape is reasonable, but one advertised surface is still broken: Null/Nothing is documented as supported even though the current write path and schema-inference path do not handle it.

Findings

• ⚠️ Majors
  • [src/Processors/Formats/Impl/VortexBlockOutputFormat.cpp:92] The new format still delegates writing to the generic CHColumnToArrowColumn mapper, and that mapper has no TypeIndex::Nothing branch in either getArrowType or fillArrowArray. That makes SELECT NULL FORMAT Vortex fail with UNKNOWN_TYPE instead of producing a Null field. On the read side, VortexSchemaReader::readSchema calls arrowSchemaToCHHeader with allow_arrow_null_type = false, so DESC file(...) on a Vortex Null field also throws instead of inferring Nothing.
  • Suggested fix: either add explicit Nothing/Nullable(Nothing) mapping to Arrow Null on write and enable arrow::Type::NA schema inference for Vortex, or remove the Null row from the supported-types contract and add a focused stateless test for the chosen behavior.

Final Verdict

• Status: ⚠️ Request changes
• Minimum required actions: align the Null/Nothing contract with the implementation by either supporting it end-to-end or documenting/rejecting it consistently, and add a focused test for that behavior.

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	86.50%	86.50%	+0.00%
Functions	91.90%	91.90%	+0.00%
Branches	78.80%	78.80%	+0.00%

Changed lines: Changed C/C++ lines covered: 296/353 (83.85%) · Uncovered code

Full report · Diff report

[alexey-milovidov]

🕵 @groeneai, investigate the two Hung check failed, possible deadlock found stress-test failures and provide a fix in a separate PR (if one is already in progress, link it here):

• Stress test (amd_msan): a fuzzer query is stuck in ActionsDAG::evaluatePartialResult / ActionsDAG::updateHeader while building an ExpressionStep in the planner — uninterruptible CPU-bound expression evaluation, possibly the same class as Uninterruptible CPU-bound function evaluation over a large value #112203.
• Stress test (arm_release): a fuzzer query over merge(REGEXP('.+'), '.*') joined to a constant subquery is stuck in ReadFromMerge::filterTablesAndCreateChildrenPlans during QueryPlan::optimize (optimizePrimaryKeyConditionAndLimit → applyFilters), single-threaded, tens of seconds in plan optimization over the stress database's many tables.

Both hangs are unrelated to this PR: the hung queries do not involve the Vortex format at all, and the amd_msan binary does not even include it (Vortex is disabled under MSan).

[groeneai]

Both hangs are the same defect, and it is not the one the amd_msan half is described as. Neither is caused by this PR.

amd_msan is not ActionsDAG::evaluatePartialResult. The hung query is

SELECT * FROM one AS A INNER JOIN system.one AS one ON A.dummy = one.dummy
             INNER JOIN merge('^') AS two ON A.dummy = two.dummy

and its stack (hung_check.log, thread 5193) is:

AnalysisTableExpressionData::ensureColumnMembershipSetsArePopulated
IdentifierResolver::tryResolveIdentifierFromTableExpression
... QueryAnalysisPass::run
ReadFromMerge::getModifiedQueryInfo
ReadFromMerge::createChildrenPlans          StorageMerge.cpp:849
ReadFromMerge::filterTablesAndCreateChildrenPlans
ReadFromMerge::applyFilters                 StorageMerge.cpp:1790
QueryPlanOptimizations::optimizePrimaryKeyConditionAndLimit
QueryPlan::optimize

The evaluatePartialResult / updateHeader / ExpressionStep frames do appear in that artifact, but in the Top messages not matching their format strings section, as the stack of a completed sleepEachRow exception (Code: 44. The argument of function sleepEachRow must be constant) raised on an unrelated HTTPHandler query. The reason they look like the hung query is that the CI report only renders the last 32 KiB of hung_check.log, and the hung row lives in the excluded head: 84% of the amd_msan log and 91% of the arm_release log are outside that window, and ReadFromMerge appears 0 times inside it. Reading the artifact directly instead of the report body is what separates the two.

So both rows are ReadFromMerge child-plan construction under QueryPlan::optimize, differing only in the leaf: amd_msan in ensureColumnMembershipSetsArePopulated, arm_release in FunctionFactory::tryGet under resolveFunction.

Cost vs interruptibility are two separate things, and only the first has a fix.

Cost: your #113140 is the right fix for it, and the amd_msan leaf is precisely the function its comment names as being rebuilt once per identifier. That accounts for the wall time.

Interruptibility: #113140 does not address it. Its 87 added lines contain no cancellation check (0 occurrences of isCancelled, checkTimeLimit, QueryStatus, ProcessList), and StorageMerge.cpp has none anywhere in its 2046 lines. Both rows show is_cancelled: 1 with max_execution_time = 10, yet ran 1757 s and 3994 s single-threaded. That is the gap: cancellation is observed between pipeline steps, and ReadFromMerge::applyFilters builds every child plan inside one optimizePrimaryKeyConditionAndLimit call with no checkpoint, so a cancelled query cannot leave. #113140 shrinks the window a lot but does not close it; a wide enough merge still exceeds any timeout uninterruptibly.

The phase counters confirm the hang is in the still-running phase rather than anything already finished: recorded phases sum to 6.6 s of 1757 s (amd_msan) and 29.4 s of 3994 s (arm_release), because QueryPlanOptimizeMicroseconds is only committed when its scope exits.

I am authoring the cancellation-checkpoint fix and will link the PR here when it opens. #112203 is a different mechanism (uninterruptible evaluation of one large value); this is uninterruptible plan construction, closer to #113003 and covered in general by #47272.

Unrelated to this PR, confirmed rather than assumed. Your MSan argument holds: rust/workspace/CMakeLists.txt in this PR sets ENABLE_VORTEX OFF when SANITIZE STREQUAL "memory", so the amd_msan binary cannot contain the format. For arm_release the diff touches 23 files, none of them under Storages/, Analyzer/, Planner/ or QueryPlan/, so it cannot appear on this stack; neither hung query references the format, and both shapes are recorded on other PRs.

[groeneai]

Fixing PR for the ReadFromMerge::createChildrenPlans plan-time cancellation hung check on my
CI finish ledger: #113415 (external, open).

I had a fix for this on a local branch, but @alexey-milovidov opened #113415 with the identical
guard at the identical call site, so I dropped mine rather than open a duplicate. My analysis and
the one coverage gap I measured are in
#113415 (comment).

[clickhouse-gh]

Vortex still inherits the generic CHColumnToArrowColumn mapping for IPv4, which emits plain Arrow UINT32. On the read side ArrowColumnToCHColumn only reconstructs IPv4 when it gets an explicit type hint (or the ORC-specific INT32 carrier), so schema-inferred Vortex readback still turns SELECT toIPv4(...) FORMAT Vortex into UInt32 on DESC file(...) / plain SELECT * FROM file(...).

This needs a Vortex-specific logical mapping for IPv4 (or an explicit rejection + documented limitation) plus a small stateless inferred-schema test.

[alexey-milovidov]

🕵 Vortex has no logical type for IP addresses, so a lossless Vortex-specific mapping is not possible — IPv4 goes through the generic Arrow UINT32 carrier, exactly like in Parquet and Arrow (where schema inference also reads it back as UInt32). Rejecting the write would make Vortex stricter than the sibling formats for no benefit, so I took the documented-limitation route: the format documentation now states that IPv4 is written as U32, that inference reads it back as UInt32, and shows the explicit-schema way to get IPv4 back (SELECT * FROM file('data.vortex', Vortex, 'ip IPv4')). Added the stateless test 04812_vortex_ipv4.sh covering inference (UInt32/Nullable(UInt32)), the plain and explicit-schema round trips (values verified), and the documented IPv6 write rejection.

[alexey-milovidov]

🕵 CI triage for the reds on a56a056dafe (all unrelated to this PR; origin/master has been merged in and CI restarted on e5634f18090):

• Stress test (arm_release) and Stress test (amd_tsan): the server stopped responding mid-stress — the ReadFromMerge::createChildrenPlans plan-time cancellation hang diagnosed earlier in this PR; the fix Honor query cancellation while building child plans of a Merge table #113415 merged to master after these runs and is now pulled into this branch by the master merge.
• Stress test (arm_tsan): Logical error: 'Not-ready Set is passed as the second argument for function 'notNullIn'' — known open issue Logical error: Not-ready Set is passed as the second argument for function 'A (STID: 0250-41a5) #107619 (fixes in flight: Fix "Not-ready Set" for GLOBAL IN over an object-storage _path / _file filter #112968, Fix logical error for a single-argument or in tryExtractCommonExpressions #113412).
• AST fuzzer (amd_tsan): Logical error: 'Invalid number of columns in chunk pushed to OutputPort. Expected 0, found 1' (STID: 2270-2caa) on a window-function query over remoteSecure with SAMPLE + QUALIFY. The existing open issues Logical error: Invalid number of columns in chunk pushed to OutputPort. Expected A, found B (STID: 2270-3452) #96656 and Logical error: Invalid number of columns in chunk pushed to OutputPort. Expected A, found B (STID: 2270-32ca) #103695 are the same error family but different stack IDs. @groeneai, investigate this failure: https://s3.amazonaws.com/clickhouse-test-reports/json.html?PR=112950&sha=a56a056dafe6b02647111a53b27d34cf38b0579a&name_0=PR&name_1=AST%20fuzzer%20%28amd_tsan%29 and provide a fix in a separate PR. If a fix is already in progress, link it here.

[clickhouse-gh]

Vortex now inherits the generic CHColumnToArrowColumn mapping here, but that mapper still has no TypeIndex::Nothing handling in either getArrowType or fillArrowArray. The advertised Null/Nothing support therefore does not actually work: SELECT NULL FORMAT Vortex (or any Nullable(Nothing) field) still throws UNKNOWN_TYPE, and a Null field cannot be schema-inferred on read either because VortexSchemaReader goes through arrowSchemaToCHHeader with allow_arrow_null_type = false.

Upstream vortex-arrow does support Arrow DataType::Null, so this needs either explicit Nothing/Nullable(Nothing) support on both paths or the supported-types contract/docs pared back, plus a focused stateless test for the chosen behavior.

[groeneai]

The AST fuzzer STID 2270-2caa failure on a56a056dafe is covered by a fix I already have open: #113469

I confirmed that on your exact query rather than by inference. A pre-fix binary reproduces your log line byte for byte (Expected 0, found 1, empty Header:, Chunk: Nullable(size = 2, UInt8(size = 2), UInt8(size = 2)), aborting in RemoteSource::prepare); the branch head returns the 8 correct rows and no error. Both arms asserted by build id.

Mechanism: cume_dist() OVER () references no column of the table expression, and defaultValueOfArgumentType(k) is constant-folded, so the outer query reads nothing from the shard. prepareBuildQueryPlanForTableExpression then synthesizes one column purely to learn the row count, and the window function is deferred to the initiator along with the projection that would have consumed that column, so nothing keeps it and it is pruned. The initiator's mergeable header ends up empty while the shard still streams the surviving Nullable(UInt8) constant, hence 1 column where 0 are expected.

The crash is the visible half. When no constant survives there is no error and rows are silently dropped instead: over two shards of 1000 rows, SELECT count(*) OVER () FROM remote(...) returned 1000 rows pre-fix, and 0 with prefer_localhost_replica = 0, against 2000 correct. The locally-read shard survives, every remote one is lost.

SAMPLE, QUALIFY, the four shards, and remoteSecure rather than remote are all incidental: dropping each of them still aborts pre-fix. What is load-bearing is more than one shard plus a window function referencing no column of the table. count(c0) OVER (), a single shard, and enable_analyzer = 0 are all clean pre-fix, so the fix is scoped to the analyzer path.

113469 covers this carrier; I am not claiming it clears the whole message class. Agreed that 96656 and 103695 are the same family with different stack IDs. One caveat on keying: within this family a single STID pools more than one query shape, so the shape identifies a carrier more reliably than the STID does.