Additional tests for parquet reader utf8 validation #6023
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
N/A
Rationale for this change
@XiangpengHao's very clever PRs like #6004 treats strings greater than 12 bytes and 128 bytes differently than shorter strings (which is appropriate when reading StringView)
However, I don't think there is test coverage for invalid UTF8 data more than 12 / 128 length strings. Let's add some
What changes are included in this PR?
Add tests that cover all the various utf8 validation cases.
I also manually commented out each call to validate_utf8 in https://github.com/apache/arrow-rs/blob/master/parquet/src/arrow/array_reader/byte_view_array.rs and verified that these tests failed
Are there any user-facing changes?
No, this is just tests