fix: canonicalize sliced VarBinArray failure#5962
Conversation
| fn test_massive() { | ||
| // Attempt to convert a really large dataset to Arrow. | ||
| let strings = VarBinArray::from_iter_nonnull( | ||
| ["1234567890123"].iter().cycle().take(500_000_000), | ||
| DType::Utf8(Nullability::NonNullable), | ||
| ); | ||
|
|
||
| let sliced = strings.slice(0..5); | ||
|
|
||
| let vbv = sliced.to_varbinview(); | ||
| assert_eq!(vbv.len(), 5); | ||
| } |
There was a problem hiding this comment.
i can't check this in because it's too slow, but this was failing before and now it's not (you actually need to update VarBinArray::from_iter_nonnull to use u64 offsets to make it pass)
There was a problem hiding this comment.
can we not use the slow run on post commit?
9ab1f44 to
c5d63c5
Compare
CodSpeed Performance ReportMerging this PR will not alter performanceComparing Summary
Footnotes
|
c5d63c5 to
ca83938
Compare
fd32064 to
f42a231
Compare
Codecov Report❌ Patch coverage is
☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
f42a231 to
85b569c
Compare
Signed-off-by: Andrew Duffy <andrew@a10y.dev>
| def test_fragment_to_table(ds: vx.dataset.VortexDataset): | ||
| fragments = list(ds.get_fragments()) | ||
|
|
||
| # The first fragment contains none of the matching records |
There was a problem hiding this comment.
now that we slice before canonicalizing, we write fewer segments of the string column, which results in fewer splits getting planned, so this is no longer true.
the test is just updated to make sure the sum of the fragment row counts matches the expected row count after the filter
A user report came in with the following stacktrace:
The failure is inside of arrow-rs, in its conversion from LargeStringArray -> StringViewArray.
Arrow checks that the final offset is < i32::MAX, and then pushes the buffer. This will not work if we're trying to convert a large VarBinArray that's been sliced, so that the final offset is not the maximum size of the buffer.
Upstream arrow-rs should get a fix as well, but in the meantime, this should fix the behavior of compressing large chunks, by rezeroing offsets before we canonicalize VarBin.