BUG: StringArray with NA behaves different to numpy object array for elementwise comparisons

[ssche]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

This is for pandas '2.1.1', the latest version raises an error instead.

>>> import pandas as pd
>>> import numpy
>>> str_arr = pd.array(['a', 'b', 'c', 'foo', 'd', 'bar', None], dtype="string")
>>> str_arr
<StringArray>
['a', 'b', 'c', 'foo', 'd', 'bar', <NA>]
Length: 7, dtype: string
>>> b2 = np.asarray(str_arr)
>>> b2
array(['a', 'b', 'c', 'foo', 'd', 'bar', <NA>], dtype=object)
>>> b2 == a_ext
<stdin>:1: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
False

I'd like an elementwise comparison and not a scalar return value.

Issue Description

When I use a string array with NA values to perform elementwise comparison, the result is a scalar instead of an elementwise matrix.

Here's how this used to work when I used numpy object dtype:

>>> import numpy as np
>>> 
>>> a = np.asarray(['foo', 'bar', 'baz'])
>>> b = np.asarray(['a', 'b', 'c', 'foo', 'd', 'bar', None])
>>> b
array(['a', 'b', 'c', 'foo', 'd', 'bar', None], dtype=object)
>>>
>>> # Find matches of `b` in `a`
>>> a_ext = a[:, np.newaxis]
>>> b == a_ext
array([[False, False, False,  True, False, False, False],
       [False, False, False, False, False,  True, False],
       [False, False, False, False, False, False, False]])

But when I use a pandas string type as a source for b (rather than object dtype), the result is a scalar boolean (in 2.1.1) or a TypeError (in 2.3.3).

>>> import pandas as pd
>>> 
>>> str_arr = pd.array(['a', 'b', 'c', 'foo', 'd', 'bar', None], dtype="string")
>>> str_arr
<StringArray>
['a', 'b', 'c', 'foo', 'd', 'bar', <NA>]
Length: 7, dtype: string
>>> b2 = np.asarray(str_arr)
>>> b2
array(['a', 'b', 'c', 'foo', 'd', 'bar', <NA>], dtype=object)

>>> pd.__version__
'2.1.1'
>>> np.__version__
'1.24.3'
>>> b2 == a_ext
<stdin>:1: FutureWarning: elementwise comparison failed; returning scalar instead, but in the future will perform elementwise comparison
False

or

>>> pd.__version__
'2.3.3'
>>> np.__version__
'2.3.4'
>>> b2 == a_ext
Traceback (most recent call last):
  File "<python-input-8>", line 8, in <module>
    b2 == a_ext
  File "pandas/_libs/missing.pyx", line 392, in pandas._libs.missing.NAType.__bool__
TypeError: boolean value of NA is ambiguous

Workaround: replace NA with None in b2:

>>> b2[pd.isnull(b2)] = None
>>> b2 == a_ext
array([[False, False, False,  True, False, False, False],
       [False, False, False, False, False,  True, False],
       [False, False, False, False, False, False, False]])

Expected Behavior

An array is returned.

Installed Versions

pd.show_versions()

INSTALLED VERSIONS

commit : e86ed37
python : 3.11.14.final.0
python-bits : 64
OS : Linux
OS-release : 6.16.12-200.fc42.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Sun Oct 12 16:31:16 UTC 2025
machine : x86_64
processor :
byteorder : little
LC_ALL : None
LANG : en_AU.UTF-8
LOCALE : en_AU.UTF-8

pandas : 2.1.1
numpy : 1.24.3
pytz : 2020.4
dateutil : 2.8.2
setuptools : 69.2.0
pip : 24.2
Cython : 0.29.34
pytest : 7.3.1
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : 0.9.6
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.6
jinja2 : 2.11.2
IPython : None
pandas_datareader : None
bs4 : None
bottleneck : 1.3.5
dataframe-api-compat: None
fastparquet : None
fsspec : None
gcsfs : None
matplotlib : None
numba : None
numexpr : 2.8.4
odfpy : None
openpyxl : 3.1.3
pandas_gbq : None
pyarrow : 11.0.0
pyreadstat : None
pyxlsb : None
s3fs : None
scipy : 1.10.1
sqlalchemy : 1.3.23
tables : 3.8.0
tabulate : None
xarray : None
xlrd : 2.0.1
zstandard : None
tzdata : 2025.2
qtpy : None
pyqt5 : None

[jorisvandenbossche]

It is not great behaviour, but it is currently the "expected" behaviour, I think.

First, the FutureWarning you see comes from numpy (and returning this scalar False value), and as you can see this warning is no longer present when using a recent numpy versions. Then, it actually does an element-wise comparison, or at least it attempts to do so, but one of the individual comparisons is raising an error (the error you see).

This comes from the fact that pd.NA (in contrast to None or np.nan) currently 1) propagates in comparisons (instead of returning True/False), and 2) has an ambiguous "boolean" value (i.e. bool(pd.NA) is not well defined and this raises an error):

The comparisons:

>>> pd.NA == "a"
<NA>
>>> pd.NA == pd.NA
<NA>
>>> np.nan == "a"
False
>>> np.nan == np.nan
False
>>> None == "a"
False
>>> None == None
True

The value in a boolean context:

>>> bool(pd.NA)
...
TypeError: boolean value of NA is ambiguous
>>> bool(np.nan)
True
>>> bool(None)
False

So what happens if you compare element-wise two numpy object-dtype arrays with strings and pd.NA, is that one of the individual comparisons involving pd.NA returns pd.NA, and then afterwards numpy tries to cast the result of the comparisons to a bool dtype (because numpy expects to return bool dtype for comparisons), and at that point bool(pd.NA) raises the error.

The workaround is to be explicit about which value to use for pd.NA when converting to numpy. Essentially what you did by replacing the NA values with None, or you could also do str_arr.to_numpy(na_value=None) instead of np.asarray(str_arr) and manually replacing.

An older issue about the bool(pd.NA) behaviour: #38224

[ssche]

Thanks for the detailed and interesting explanation.

According to this logic

>>> pd.NA == "a"
<NA>
>>> pd.NA == pd.NA
<NA>

wouldn’t an array with a mix of boolean values and NA values be a better return value. Is this possible (in future?) or is there some hardcoded behaviour in numpy that requires the elements of the comparison of type bool (the casting you mentioned)?