[Python] Conversion of numpy array of bytes (dtype='S') ignores length

[ArvidJB]

Describe the bug, including details regarding any error messages, version, and platform.

When converting a numpy array of bytes (dtype='S3' for example) pyarrow seems to make all values the same fixed-length and pad with null characters:

In [2]: pa.__version__
Out[2]: '12.0.1'

In [3]: a = np.array([b'a', b'ab', b'abc'])

In [4]: a
Out[4]: array([b'a', b'ab', b'abc'], dtype='|S3')

In [5]: b = pa.array(a, type=pa.string())

In [6]: b
Out[6]:
<pyarrow.lib.StringArray object at 0x7fb4490083a0>
[
  "a",
  "ab",
  "abc"
]

In [7]: [x.as_py() for x in b]
Out[7]: ['a\x00\x00', 'ab\x00', 'abc']

I don't think this is the intended behavior here?

Note that the stringification of StringArray hides this issue since it truncates after encountering the first null character. To be observe it you need to convert to Python strings.

This also means that the output of to_numpy is not the same as the original array:

In [11]: b.to_numpy(zero_copy_only=False)
Out[11]: array(['a\x00\x00', 'ab\x00', 'abc'], dtype=object)

If you agree that this is a bug I am happy to contribute a patch.

Component(s)

Python

[AlenkaF]

The logic of conversion to strings in an array constructor does look faulty to me. In the case you mention the bytes get handled correctly if using the default conversion data type which is binary:

>>> c = pa.array(a)
>>> c
<pyarrow.lib.BinaryArray object at 0x115dd6440>
[
  61,
  6162,
  616263
]
>>> c.to_numpy(zero_copy_only=False)
array([b'a', b'ab', b'abc'], dtype=object)

if we cast the BinaryArray to the string dtype and call to_numpy() we get the expected result (with an object dtype):

>>> c.cast(pa.string())
<pyarrow.lib.StringArray object at 0x115dd6500>
[
  "a",
  "ab",
  "abc"
]
>>> c.cast(pa.string()).to_numpy(zero_copy_only=False)
array(['a', 'ab', 'abc'], dtype=object)

But I suggest we wait for an extra opinion before you start working on a PR.