Casting bool to object during join

[kjschiroo]

import pandas as pd

core_df = pd.DataFrame([
    {'id': 1},
    {'id': 2},
]).set_index('id')

df1 = pd.DataFrame([
    {'id': 1, 'foo': True},
    {'id': 2, 'foo': False},
]).set_index('id')

df2 = pd.DataFrame([
    {'id': 1, 'bar': 3},
    {'id': 2, 'bar': 4},
    {'id': 3, 'bar': 4},  # Adding a new row in df2 is required to create the issue
]).set_index('id')

result = core_df.join([df1, df2])
print(result.dtypes)
# With pandas 0.24.2
# foo     bool  <- stays a bool
# bar    int64
# dtype: object
#
# With pandas 0.25.1
# foo    object  <- Changed to object
# bar     int64
# dtype: object

Problem description

This is a kind of subtle issue when upgrading from 0.24 to 0.25. When joining dataframes onto an initial dataframe, if one of the dataframes being joined in has an extra row compared to the other it will cause any boolean columns to be cast to object columns. This is a change in behavior from 0.24 which would leave them as bools.

Expected Output

In the example above I would expect the foo column to remain of dtype bool.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.7.3.final.0
python-bits : 64
OS : Darwin
OS-release : 18.6.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 0.25.1
numpy : 1.16.3
pytz : 2019.1
dateutil : 2.8.0
pip : 19.0.3
setuptools : 41.1.0
Cython : None
pytest : 4.4.1
hypothesis : None
sphinx : 2.0.1
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : 0.9.3
psycopg2 : None
jinja2 : 2.10.1
IPython : None
pandas_datareader: None
bs4 : 4.7.1
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : 2.6.9
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : 0.13.0
pytables : None
s3fs : None
scipy : 1.2.1
sqlalchemy : 1.3.3
tables : 3.5.2
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

[TomAugspurger]

Is there anything in the release notes about this?

If it's intentional, my first guess is that it's to ensure dtype stability with the different join methods. Pandas uses object-dtype to represent booleans with missing values, and an outer join would force object dtype

In [48]: core_df.join([df1, df2], how='outer')
Out[48]:
      foo  bar
id
1    True    3
2   False    4
3     NaN    4

[kjschiroo]

At least so far as I've been able to see it isn't mentioned in the release notes, but I might be missing how it would be called out. This is just based on looking for any mention of join.