API: DataFrame.__and__ vs Series.__and__ fillna behavior mismatch

[jbrockmendel]

Based on tests.series.test_operators test_logical_ops_df_compat

# GH#1134
s1 = pd.Series([True, False, True], index=list("ABC"))
s2 = pd.Series([True, True, False], index=list("ABD"))

exp_ser = pd.Series([True, False, False, False], index=list("ABCD"))
res_ser = s1 & s2
tm.assert_series_equal(res_ser, exp_ser)

# DataFrame doesn't fill nan with False
exp_frame = pd.DataFrame({"x": [True, False, np.nan, np.nan]}, index=list("ABCD"))
res_frame = s1.to_frame() & s2.to_frame()
assert_frame_equal(res_frame, exp_frame)

Is there still a compelling reason to keep this mismatched behavior?

[TomAugspurger]

Which, if any, would you propose to accept?

Filling with NaN is consistent with other ops I suppose.

[jorisvandenbossche]

I don't think there is a good reason to keep the mismatch, but I also don't think there is a clear correct expected result right now, as we basically do not support (or have defined behaviour for) boolean data with missing values.

This relates to the discussion on missing values in #28095 and specifically what the behaviour of NA should be in logical operations.

Also | has inconsistency, and if you follow the "three-valued logic" (mentioned in the issue), both Series and DataFrame behaviour is wrong.

And if you do it on the underlying numpy values, that just raises (that might actually be a safer default for now, making it more easily to choose whatever behaviour later)

In [47]: s1 = pd.Series([True, False, True], index=list("ABC")) 
    ...: s2 = pd.Series([True, True, False], index=list("ABD")) 

In [50]: s1a, s2a = s1.align(s2)  

In [51]: s1a 
Out[51]: 
A     True
B    False
C     True
D      NaN
dtype: object

In [52]: s2a
Out[52]: 
A     True
B     True
C      NaN
D    False
dtype: object


In [53]: s1a & s2a   
Out[53]: 
A     True
B    False
C    False
D    False
dtype: bool

In [54]: s1a.values & s2a.values
...
TypeError: unsupported operand type(s) for &: 'bool' and 'float'

In [59]: s1a | s2a 
Out[59]: 
A     True
B     True
C     True
D    False
dtype: bool

In [60]: pd.DataFrame(s1a) | pd.DataFrame(s2a) 
Out[60]: 
      0
A  True
B  True
C   NaN
D   NaN

This relates

[jbrockmendel]

@jorisvandenbossche are you suggesting anything in particular for the short-term to bring these into alignment?

[jorisvandenbossche]

Nope .. except for raising an error instead, but that's probably a bit too drastic.

[jbrockmendel]

closing as duplicate of #22724