Document Union nullability contract

[connortsui20]

Rationale

The current develop contract is sufficient to start the canonical sparse UnionArray: Vortex's union-level nullability is a type-level summary derived from its variants, while a concrete row's validity comes from its selected child. There is no parent validity bitmap.

What is not yet settled is how a generic operation should change a union's declared nullability—for example, whether mask should make an existing variant nullable, require a Null variant, or take an explicit output type. That API question does not need to block the array layout.

This PR therefore documents the existing contract and the boundary of the deferred work rather than changing behavior.

Arrow implementation survey

The implementations agree on the portable row-level semantic rule, but expose physical and logical nulls differently:

• arrow-rs has no parent null buffer; physical is_null/null_count report false/zero, while logical_nulls() gathers validity from the selected child. Its Field::new_union helper makes the outer field non-nullable.
• Arrow C++ also has no parent bitmap; IsNull(i) follows the selected child and ComputeLogicalNullCount() counts selected-child nulls.
• Arrow Java reports no physical parent nulls, while value access follows the selected child. Its dense and sparse helpers are not consistent about default outer field nullability.

The stable cross-implementation rule is therefore: the selected child determines whether a union row is logically null, and inactive sparse-child nulls are ignored. Outer Arrow Field::nullable conventions should not define Vortex's contract.

The investigation behind this boundary is in this comment on #7882.

Changes

• document the existing variant-derived DType::Union nullability constraint
• document selected-child logical validity and the lack of a parent bitmap
• call out that DType::with_nullability currently changes only the outer summary and can create an inconsistent union DType
• require initial union operations to reject nullability-changing cases until Tracking Issue: Add DType::Union #7882 defines them

Initial implementation boundary

The canonical sparse array can now proceed with predeclared, internally consistent union DTypes and DType-preserving operations such as scalar access, slice, filter, and take with non-nullable indices. mask, nullable-index take, casts that introduce nulls, and null-appending builder behavior remain explicit follow-up work.

Verification

• cargo +nightly fmt --all --check
• cargo test -p vortex-array --doc
• pre-push Clippy checks

[codspeed-hq]

Merging this PR will not alter performance

⚡ 3 improved benchmarks
❌ 1 regressed benchmark
✅ 1626 untouched benchmarks
⏩ 42 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	compact_sliced[(4096, 90)]	779.7 ns	867.2 ns	-10.09%
⚡	Simulation	bitwise_not_vortex_buffer_mut[128]	273.6 ns	244.4 ns	+11.93%
⚡	Simulation	compact[(4096, 90)]	867.2 ns	779.7 ns	+11.22%
⚡	Simulation	compare_int_constant	298.4 µs	268.5 µs	+11.12%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing codex/union-contract-correction (728422f) with develop (42bea5d)}

Footnotes

1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[connortsui20]

Superseded by #8714, which removes the stored Union nullability and derives it from the variants at runtime.