Skip to content

fix: accept FixedSizeList dtype when deserializing ListValue scalars#8696

Merged
connortsui20 merged 2 commits into
vortex-data:developfrom
polarsignals:v0.77.0-patched
Jul 9, 2026
Merged

fix: accept FixedSizeList dtype when deserializing ListValue scalars#8696
connortsui20 merged 2 commits into
vortex-data:developfrom
polarsignals:v0.77.0-patched

Conversation

@brancz

@brancz brancz commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Scalar serialization writes ScalarValue::Tuple as a protobuf ListValue for both List and FixedSizeList dtypes, but list_from_proto only accepted List. Since #8667 made constant detection unconditional, a constant fixed-size list column (e.g. UUIDs stored as fixed_size_list(u8)[16]) is compressed to a ConstantArray whose scalar metadata then fails to deserialize on read:

expected List dtype for ListValue, got fixed_size_list(u8)[16]

so files written this way come back unreadable. Scalar validation already handles FixedSizeList + Tuple (including the size check), so deserialization was the only gap.

Includes a proto round-trip test that fails without the fix.

@codspeed-hq

codspeed-hq Bot commented Jul 9, 2026

Copy link
Copy Markdown

Merging this PR will not alter performance

⚡ 3 improved benchmarks
❌ 2 regressed benchmarks
✅ 1614 untouched benchmarks
⏩ 42 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
Simulation rebuild_naive 91.4 µs 105.8 µs -13.67%
Simulation chunked_varbinview_opt_canonical_into[(100, 100)] 305.6 µs 340.4 µs -10.23%
Simulation chunked_varbinview_into_canonical[(1000, 10)] 205.9 µs 170.1 µs +21.08%
Simulation bitwise_not_vortex_buffer_mut[128] 244.4 ns 215.3 ns +13.55%
Simulation bitwise_not_vortex_buffer_mut[1024] 304.7 ns 275.6 ns +10.58%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing polarsignals:v0.77.0-patched (e855f4c) with develop (a6ce832)

Open in CodSpeed

Footnotes

  1. 42 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@brancz brancz force-pushed the v0.77.0-patched branch from 80fd385 to 7cd3f2f Compare July 9, 2026 12:32
@robert3005 robert3005 added the changelog/fix A bug fix label Jul 9, 2026
@brancz brancz force-pushed the v0.77.0-patched branch from 7cd3f2f to ab66ac6 Compare July 9, 2026 15:30
brancz and others added 2 commits July 9, 2026 17:31
Scalar serialization writes ScalarValue::Tuple as a protobuf ListValue
regardless of whether the dtype is List or FixedSizeList, but
list_from_proto only accepted List. Since vortex-data#8667 made constant detection
unconditional in the cascading compressor, a constant FixedSizeList
column (e.g. a UUID stored as fixed_size_list(u8)[16]) is written as a
ConstantArray whose scalar metadata then fails to deserialize on read:

    expected List dtype for ListValue, got fixed_size_list(u8)[16]

Scalar validation already handles FixedSizeList + Tuple (including the
size check), so deserialization was the only gap.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
Covers the ConstantArray metadata path for fixed-size list dtypes
(e.g. a UUID column stored as fixed_size_list(u8)[16]): serialization
writes the scalar as a protobuf ListValue, and deserialization
previously rejected it with "expected List dtype for ListValue".

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: Frederic Branczyk <fbranczyk@gmail.com>
@brancz brancz force-pushed the v0.77.0-patched branch from ab66ac6 to e855f4c Compare July 9, 2026 15:31
@connortsui20 connortsui20 enabled auto-merge (squash) July 9, 2026 15:37
@connortsui20 connortsui20 merged commit 07e1b52 into vortex-data:develop Jul 9, 2026
69 of 71 checks passed
@codspeed-hq

codspeed-hq Bot commented Jul 9, 2026

Copy link
Copy Markdown

Unable to generate the performance report

There was an internal error while processing the run's data. We're working on fixing the issue. Feel free to contact us on Discord or at support@codspeed.io if the issue persists.

@brancz brancz deleted the v0.77.0-patched branch July 10, 2026 09:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

changelog/fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants