Skip to content

fix unpickle_bytes_columns crash on variable-size binary columns#273

Merged
JoOkuma merged 2 commits intoroyerlab:mainfrom
TeunHuijben:fix-unpickling-bug-plobject
Mar 26, 2026
Merged

fix unpickle_bytes_columns crash on variable-size binary columns#273
JoOkuma merged 2 commits intoroyerlab:mainfrom
TeunHuijben:fix-unpickling-bug-plobject

Conversation

@TeunHuijben
Copy link
Copy Markdown
Contributor

@TeunHuijben TeunHuijben commented Mar 26, 2026

map_elements without return_dtype infers the column type from the first deserialized element. When a binary column contains numpy arrays of different shapes (e.g. cell masks after a GEFF roundtrip), polars locks in the type from the first array and raises SchemaError on any subsequent array with a different shape.

The function already had a fallback loop to handle variable-size objects — it was simply unreachable because map_elements crashed before returning.

Fix: add return_dtype=pl.Object to skip type inference. The existing fallback loop then re-casts uniform columns to typed polars Series and silently keeps variable-size columns as pl.Object.

Regression test: test_unpickle_bytes_columns_variable_size_arrays — two differently-shaped bool arrays in a binary column, previously raised SchemaError.

@TeunHuijben TeunHuijben changed the title fix unpickling bug with pl.Object type when arrays have different sizes fix unpickle_bytes_columns crash on variable-size binary columns Mar 26, 2026
@codecov-commenter
Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.33%. Comparing base (47f450a) to head (3348d7b).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #273      +/-   ##
==========================================
+ Coverage   87.29%   87.33%   +0.04%     
==========================================
  Files          56       56              
  Lines        4713     4713              
  Branches      832      832              
==========================================
+ Hits         4114     4116       +2     
+ Misses        380      378       -2     
  Partials      219      219              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TeunHuijben TeunHuijben requested a review from JoOkuma March 26, 2026 01:04
@TeunHuijben TeunHuijben marked this pull request as ready for review March 26, 2026 16:34
@JoOkuma JoOkuma merged commit 65f3a74 into royerlab:main Mar 26, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants