Failure of unicode sandwich for multi-dimensional columns

[mhvk]

We allow comparisons of unicode with byte-valued columns using a unicode sandwich, but this fails in the corner-case where the column is multidimensional and we take a single element from it:

from astropy.table import Column
c = Column([[b'a', b'b'], [b'c', b'd']], name='c')
print(c == ['a', 'b'])
# [[ True  True]
#  [False False]]
print(c[0] == ['a', 'b'])
# False  # expected [True, True]

EDIT (2024-01-11): now gives [False, False] so still an issue.

The reason is that for multi-D columns, access of any given Column row just returns a plain ndarray, which does not know about the unicode sandwich: https://github.com/astropy/astropy/blob/master/astropy/table/_column_mixins.pyx#L54-L55

A fix might be to convert the result to unicode or to use an ndarray subclass that still contains the unicode sandwich (but is otherwise a plain ndarray).

cc @taldcroft

p.s. Found while trying a new Masked class for creating a new MaskedColumn.

[taldcroft]

I suspect this is a bit of a rabbit hole... with new numpy dtypes is there hope for a single-byte character type anywhere on the horizon before I'm retired?

[mhvk]

Yes, hope, but before you're retired... that may depend on how long you plan to continue working....