Fast UTF-8 validation when reading StringViewArray from Parquet

[XiangpengHao]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

Now that #5973 is merged, we have a very fast implementation for loading BinaryViewArray from Parquet. As shown in the benchmark below:

cargo bench --bench arrow_reader --features="arrow test_common experimental" "arrow_array_reader/Binary.*Array/plain encoded"

arrow_array_reader/BinaryArray/plain encoded, mandatory, no NULLs
                        time:   [712.64 µs 720.60 µs 729.31 µs]
                        change: [-1.3454% +0.0451% +1.4101%] (p = 0.95 > 0.05)


arrow_array_reader/BinaryViewArray/plain encoded, mandatory, no NULLs
                        time:   [321.74 µs 329.27 µs 336.51 µs]
                        change: [-2.7538% -0.5436% +1.5561%] (p = 0.63 > 0.05)
                        No change in performance detected.

However, reading StringViewArray is similar to (sometimes slower than) reading StringArray.

cargo bench --bench arrow_reader --features="arrow test_common experimental" "arrow_array_reader/String.*Array/plain encoded"

arrow_array_reader/StringArray/plain encoded, mandatory, no NULLs
                        time:   [789.14 µs 795.74 µs 802.45 µs]


arrow_array_reader/StringArray/plain encoded, optional, half NULLs
                        time:   [722.79 µs 729.59 µs 737.34 µs]

After many investigations, I believe the slow down comes from the fact that validating utf-8 for StringViewArray is much slower than that for StringArray.

Why

StringArray has one continuous buffer, all the strings are packed one after another. This means that if the entire underlying buffer is utf-8, then all strings in the buffer are utf-8. Validating large string chunk is much much faster than validating small strings one by one. (I believe this is because the compiler is smart enough to do auto-vectorization, which only benefits long strings)

The buffer from string view, unfortunately, is not continuous because parquet encodes string length in between the strings, string length, unfortunately, may or may not be utf-8. This forces us to validate strings one by one (need to skip the string lengths) instead of validating the entire string buffer.

Describe the solution you'd like

The goal is to make string view array utf-8 validation as fast as string array.

Describe alternatives you've considered

Additional context

[XiangpengHao]

take