BUG: Adding two DataFrames together with duplicate column names in one results in infinite loop

[tdpetrou]

Code Sample, a copy-pastable example

df1 = pd.DataFrame(data=[[0]], columns=['second'])
df2 = pd.DataFrame(data=[[0, 0, 0]], columns=['first', 'second', 'second'])

Adding them together never completes (infinite loop?)

df1 + df2

Problem description

When adding two DataFrame where one has duplicate column names and the other has one (or more) of the duplicated columns, the operation never completes

Expected Output

Expected to follow rules of automatic alignment of the index.

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.3.final.0
python-bits : 64
OS : Darwin
OS-release : 19.4.0
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.5
numpy : 1.18.5
pytz : 2020.1
dateutil : 2.8.1
pip : 20.1.1
setuptools : 47.1.1.post20200529
Cython : None
pytest : 5.4.2
hypothesis : None
sphinx : 3.1.2
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.5.1
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.2
IPython : 7.16.1
pandas_datareader: None
bs4 : 4.9.1
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : 4.5.1
matplotlib : 3.2.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : 5.4.2
pyxlsb : None
s3fs : None
scipy : 1.5.0
sqlalchemy : None
tables : None
tabulate : 0.8.7
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[simonjayhawkins]

Thanks @tdpetrou for the report.

I can confirm this issue still exists on master. This looks like a regression from 0.25.3

>>> import pandas as pd
>>>
>>> pd.__version__
'0.25.3'
>>>
>>> df1 = pd.DataFrame(data=[[0]], columns=["second"])
>>> df2 = pd.DataFrame(data=[[0, 0, 0]], columns=["first", "second", "second"])
>>> df1 + df2
   first  second  second
0    NaN       0       0

[simonjayhawkins]

it appears that this issue was introduced in #31679

f4dc9f9 is the first bad commit
commit f4dc9f9
Author: jbrockmendel jbrockmendel@gmail.com
Date: Tue Feb 18 16:26:27 2020 -0800

REGR: fix op(frame, frame2) with reindex (#31679)

cc @jbrockmendel

[fangchenli]

The arithmetic operation hanged on pandas/core/ops/__init__.py::_frame_arith_method_with_reindex:

cols = left.columns.intersection(right.columns)
# Index(['second', 'second'], dtype='object')

new_left = left[cols]
#    second  second
# 0       0       0

new_right = right[cols]
#    second  second  second  second
# 0       0       0       0      0

result = op(new_left, new_right)
# freezes here

The observation here is that selecting n duplicated columns in a dataframe that contains m duplicated columns will create m*n the same columns. And doing arithmetic operations between to dataframes both contain duplicated columns cause freeze.

[TomAugspurger]

Taking a look now.