BUG: Unexpected IndexingError when filtering with .loc

[bikoboshi]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

>>> a = pd.Series([True, False])
>>> b = pd.Series([np.nan, False]).astype('boolean')
>>> s = pd.Series([1,2])
>>> s.loc[a].loc[b]
...
pandas.core.indexing.IndexingError: Unalignable boolean Series provided as indexer (index of the boolean Series and of the indexed object do not match).

Issue Description

The example throws an unexpected indexing error. The error seems to trigger if the first filter is neither the identity nor empty and when combined with the second filter with nan is empty. The error does not seem to trigger otherwise.

Expected Behavior

I expect this identity to hold s.loc[a].loc[b] == s.loc[b].loc[a] == s.loc[a & b] which it does except for the exception above.

Installed Versions

INSTALLED VERSIONS ------------------ commit : 66e3805 python : 3.7.2.final.0 python-bits : 64 OS : Windows OS-release : 10 Version : 10.0.22000 machine : AMD64 processor : Intel64 Family 6 Model 158 Stepping 10, GenuineIntel byteorder : little LC_ALL : None LANG : None LOCALE : None.None

pandas : 1.3.5
numpy : 1.17.4
pytz : 2019.1
dateutil : 2.8.0
pip : 21.2.4
setuptools : 46.0.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.10
IPython : 7.8.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.3.5
tables : None
tabulate : 0.8.3
xarray : None
xlrd : 1.2.0
xlwt : None
numba : None

[mctessi]

@bikoboshi
To me this behavior makes perfect sense if one accepts that an empty Series can be indexed arbitrarily! Indexing in general is not commutative:

s[b]
# Series([], dtype: int64)  empty Series
s[b][a] #legal 

s[a]
#0    1
#dtype: int64
# not empty 
s[a][b] #raises since mismatching dimensions

to add another example for noncommutativity

s = pd.Series([1, 2])
a = pd.Series([True, False])
b = pd.Series([True])

s[a][b] #legal
#0    1
#dtype: int64 

s[b] #raises error

[bikoboshi]

@mctessi If it was just dimensions, that would be tolerable. However, the exception I'm seeing is also dependent on value. The below code runs fine. Whether it should? I'm not sure; I'm just looking for consistency.

>>> a = pd.Series([True, False])
>>> b = pd.Series([False, False]).astype('boolean')
>>> s = pd.Series([1,2])
>>> s.loc[a].loc[b]
Series([], dtype: int64)

[mctessi]

@bikoboshi
Wow, did not expect that!

s = pd.Series([1, 2])

a = pd.Series([True])
s[a] #raises

b = pd.Series([True, True, True])
s[b] #fine, but error expected 
np.array([1,2])[b] #raises 

pandas.core.indexing.IndexingError: Unalignable boolean Series provided suggests, that pandas only raises errors if the index sequences is to short! Tomorrow I will deep dive into pandas code...
Your original error is triggert by a not functioning conversion to bool:

pd.Series([np.nan, False]).astype("boolean")
#0     <NA>
#1    False

Anyway, I agree that it is inconsistent behavior and totally surprised me!

[mctessi]

take

[mctessi]

@bikoboshi
Please do not expect wonders :) I have a hunch that this behavior, although inconsistent is expected by so many lines of code that a change seems to be unlikely :(

[bikoboshi]

@mctessi
It seems almost consistent for cases where the indices of the mask is the superset of the indices of the Dataframe or Series. The only issue I've observed is when nulls come into play.