[C++] parquet with invalid utf8 does not error

[asfimport]

The code below likely results in undefined behavior as the data in the parquet is not valid utf8.

from io import BytesIO
import pyarrow
import pyarrow.parquet

// parquet with 1 column marked as string with invalid utf8
data = [
        80, 65, 82, 49, 21, 6, 21, 22, 21, 22, 92, 21, 2, 21, 0, 21, 2, 21, 0, 21, 4, 21,
        0, 18, 28, 54, 0, 40, 5, 104, 101, 255, 108, 111, 24, 5, 104, 101, 255, 108, 111,
        0, 0, 0, 3, 1, 5, 0, 0, 0, 104, 101, 255, 108, 111, 38, 110, 28, 21, 12, 25, 37,
        6, 0, 25, 24, 2, 99, 49, 21, 0, 22, 2, 22, 102, 22, 102, 38, 8, 60, 54, 0, 40, 5,
        104, 101, 255, 108, 111, 24, 5, 104, 101, 255, 108, 111, 0, 0, 0, 21, 4, 25, 44,
        72, 4, 114, 111, 111, 116, 21, 2, 0, 21, 12, 37, 2, 24, 2, 99, 49, 37, 0, 76, 28,
        0, 0, 0, 22, 2, 25, 28, 25, 28, 38, 110, 28, 21, 12, 25, 37, 6, 0, 25, 24, 2, 99,
        49, 21, 0, 22, 2, 22, 102, 22, 102, 38, 8, 60, 54, 0, 40, 5, 104, 101, 255, 108,
        111, 24, 5, 104, 101, 255, 108, 111, 0, 0, 0, 22, 102, 22, 2, 0, 40, 44, 65, 114,
        114, 111, 119, 50, 32, 45, 32, 78, 97, 116, 105, 118, 101, 32, 82, 117, 115, 116,
        32, 105, 109, 112, 108, 101, 109, 101, 110, 116, 97, 116, 105, 111, 110, 32, 111,
        102, 32, 65, 114, 114, 111, 119, 0, 130, 0, 0, 0, 80, 65, 82, 49,
]
data = BytesIO(bytearray(data))

a = pyarrow.parquet.read_table(data)

print(a.column(0))

but maybe this is by design and not a concern?

Reporter: Jorge Leitão / @jorgecarleitao

_{Note: This issue was originally created as ARROW-14037. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Indeed, we don't check that the data is valid on loading.

You could call a.validate(full=True) and should get an error, though.

[asfimport]

Antoine Pitrou / @pitrou:
Oh, and by the way, the loading can probably be made a little simpler:

data = [ 80, ... ]
data = BytesIO(bytearray(data))

[asfimport]

Jorge Leitão / @jorgecarleitao:
Thanks; I have updated the example.

The context of this JIRA is that, in Rust, performing this operation must be marked as "unsafe" because it can result in UB. apache/arrow-rs#786. Rust and C++ are similar in this context (of triggering UB in non-valid utf8), which is why I raised this here also.

If I am understanding correctly, the design is that it is the user of Utf8Array's responsibility to always validate the array prior to using it, to ensure no UB is triggered? If that is the case, then I think we should mark this as won't fix / work as intended.

[asfimport]

Antoine Pitrou / @pitrou:
Well, if your operations don't care about invalid UTF8 then you don't need to validate the data. C++ itself doesn't care about UTF8 validity.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.