Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats

[nihalzp]

The following queries now work.

-- Setup
SET allow_experimental_nullable_tuple_type = 1;
SET engine_file_truncate_on_insert = 1;

Arrow

INSERT INTO TABLE FUNCTION file('test.arrow', 'Arrow', 'c0 Nullable(Tuple(UInt32, String))') VALUES ((1, 'a')), (NULL), ((3, 'c'));
SELECT c0 FROM file('test.arrow', 'Arrow', 'c0 Nullable(Tuple(UInt32, String))');

Arrow

INSERT INTO TABLE FUNCTION file('test.arrowstream', 'ArrowStream', 'c0 Nullable(Tuple(UInt32, String))') VALUES ((1, 'a')), (NULL), ((3, 'c'));
SELECT c0 FROM file('test.arrowstream', 'ArrowStream', 'c0 Nullable(Tuple(UInt32, String))');

ORC

INSERT INTO TABLE FUNCTION file('test.orc', 'ORC', 'c0 Nullable(Tuple(UInt32, String))') VALUES ((1, 'a')), (NULL), ((3, 'c'));
SELECT c0 FROM file('test.orc', 'ORC', 'c0 Nullable(Tuple(UInt32, String))');

Legacy Parquet (Arrow-based reader):

INSERT INTO TABLE FUNCTION file('test.parquet', 'Parquet', 'c0 Nullable(Tuple(UInt32, String))') VALUES ((1, 'a')), (NULL), ((3, 'c'));
SELECT c0 FROM file('test.parquet', 'Parquet', 'c0 Nullable(Tuple(UInt32, String))') SETTINGS input_format_parquet_use_native_reader_v3 = 0;

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Support Nullable(Tuple) for Arrow, ArrowStream, ORC, legacy Parquet formats.

Documentation entry for user-facing changes

• Documentation is written (mandatory for new features)

[clickhouse-gh]

Workflow [PR], commit [7d950a7]

Summary: ✅

---

AI Review

Summary

This PR adds support for Nullable(Tuple(...)) round-trips in Arrow, ArrowStream, ORC, and legacy Parquet, including null-map propagation for nested tuple elements and broader stateless coverage. The core implementation looks solid, but one important edge-case remains under-tested: reading with LowCardinality(Nullable(Tuple(...))) type hints. Verdict: request changes until this gap is covered.

Missing context

• ⚠️ No CI report/log links were provided in the review request, so I could not validate runtime regressions/failures beyond code and tests in this checkout.

Findings

• ⚠️ Majors
  • [tests/queries/0_stateless/04064_tuple_inside_nullable_arrow_orc_roundtrip.sql:1+, tests/queries/0_stateless/04065_tuple_inside_nullable_parquet_roundtrip.sql:1+] Regression coverage still misses LowCardinality(Nullable(Tuple(...))) read-hint scenarios.
  • Impact: current changes explicitly touch LowCardinality + nullable handling paths; without dedicated tests this can silently regress.
  • Suggested fix: add at least one read-hint case per affected reader path (Arrow/ArrowStream/ORC and legacy Parquet) with LowCardinality(Nullable(Tuple(...))), including rows with both struct-level NULL and inner nullable values.

Tests

• ⚠️ Add regression tests for LowCardinality(Nullable(Tuple(...))) read hints across updated readers to lock in the behavior this PR targets.

ClickHouse Rules

Item	Status	Notes
Deletion logging	➖
Serialization versioning	➖
Core-area scrutiny	✅
No test removal	✅
Experimental gate	➖
No magic constants	✅
Backward compatibility	✅
SettingsChangesHistory.cpp	➖
PR metadata quality	✅
Safe rollout	✅
Compilation time	✅
No large/binary files	✅

Final Verdict

• Status: ⚠️ Request changes
• Minimum required actions:
  • Add targeted regression coverage for LowCardinality(Nullable(Tuple(...))) read-hint paths in the new format tests.

[alexey-milovidov]

The Stress test (arm_msan) failure is fixed by #101239, which should be merged first. After it is merged, please update the branch to include the fix.

[Avogar]

LGTM, but the comments about LowCardinality(Nullable) from AI review sounds valid to me, let's try to fix it and add a test

[Avogar]

LGTM. Let's just fix the test 02384_nullable_low_cardinality_as_dict_in_arrow in flaky check. It cannot be run in parallel with itself as it uses file table function with constant file name. Let's use unique file name using currentDataBase. Or better to rewrite the test to a bash test with clickhouse-local to avoid keeping trash files in user_files directory

[clickhouse-gh]

LLVM Coverage Report

Metric	Baseline	Current	Δ
Lines	84.00%	84.00%	+0.00%
Functions	90.90%	90.90%	+0.00%
Branches	76.50%	76.50%	+0.00%

Changed lines: 97.87% (138/141) · Uncovered code

Full report · Diff report