test: add multi-byte UTF-8 coverage for left/right - #24219
Conversation
left and right index by Unicode scalar value rather than byte offset, but have no sqllogictest coverage with multi-byte input. left_right_byte_length takes an ASCII fast path that derives the byte offset from the character count, guarded by string.is_ascii(); non-ASCII input falls through to the char_indices()/nth_back() path. These cases pin that behaviour for both string representations, including Utf8View, whose consumer slices the underlying bytes without the &str boundary check.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #24219 +/- ##
==========================================
- Coverage 81.05% 80.98% -0.07%
==========================================
Files 1106 1106
Lines 382268 383158 +890
Branches 382268 383158 +890
==========================================
+ Hits 309864 310318 +454
- Misses 54109 54521 +412
- Partials 18295 18319 +24 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
neilconway
left a comment
There was a problem hiding this comment.
@4ktLuffy Thanks for working on this! There are some tests for left and right in datafusion/sqllogictest/test_files/string/string_query.slt.part. From a quick skim, it seems like those cover the new test scenarios added in this PR, but let me know if you can see any scenarios / code paths that would benefit from additional test cases.
|
Thanks for the pointer — you're right, and I should have found this I also checked whether literal (scalar) invocation takes a distinct |
Which issue does this PR close?
Rationale for this change
leftandrightindex by Unicode scalar value rather than byte offset, but currently have no sqllogictest coverage with multi-byte input. Searching the.sltcorpus,substrandstrposhave non-ASCII cases;leftandrighthave none.left_right_byte_lengthtakes an ASCII fast path introduced in #23762 that derives the byte offset directly from the character count. That is sound only where byte length and character count coincide, and it is guarded bystring.is_ascii(); non-ASCII input falls through to thechar_indices()/nth_back()path.The two consumers of that offset behave differently if it is ever wrong.
general_left_right_arrayslices via&strindexing, which validates char boundaries.general_left_right_viewslices the underlying bytes without the&strboundary check, so an incorrect offset can produce an invalid UTF-8 view rather than the explicit boundary panic seen onStringArray.The
unicodemodule has had several buffer-level performance changes recently — #23762 (left/right), #23586 (pad), #22171 (translate) — and none of them touched a sqllogictest file. These cases pin the character-indexing behaviour so that a future optimisation cannot alter it without a test failing.What changes are included in this PR?
Eight cases in
functions.slt, placed alongside the existingleft/righttests:héllo), 3-byte (日本語) and 4-byte (hi🌏) inputsn, covering thebyte_offset_of_charandnth_backbranchesUtf8Viewvariants sogeneral_left_right_viewis exercised in addition togeneral_left_right_arrayNo source changes.
Are these changes tested?
They are tests, and I checked that they actually protect the invariant rather than just record current output: temporarily replacing the
is_ascii()guard withtrue, so the ASCII fast path always applied, made theStringArraycases fail withand the
Utf8Viewcases return an invalid UTF-8 result. With the guard restored,cargo test -p datafusion-sqllogictest --test sqllogictests -- functionspasses.Are there any user-facing changes?
No.