New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merging two empty dataframes can incur a division by zero #17776

Closed
micklat opened this Issue Oct 4, 2017 · 4 comments

Comments

Projects
None yet
4 participants
@micklat

micklat commented Oct 4, 2017

Code Sample, a copy-pastable example if possible

import numpy
import pandas

pandas.show_versions()

a = pandas.DataFrame({'a':[],'b':[],'c':[]})
numpy.seterr(divide='raise')
pandas.merge(a,a,on=('a','b')) # no problem if we only merge on 'a'.

Problem description

The call to merge triggers a division by zero.

Traceback (most recent call last):
File "/homes/mickyl/work/bugs/pandas_merge_div_by_0.py", line 8, in
pandas.merge(a,a,on=('a','b'))
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 54, in merge
return op.get_result()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 569, in get_result
join_index, left_indexer, right_indexer = self._get_join_info()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 734, in _get_join_info
right_indexer) = self._get_join_indexers()
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 713, in _get_join_indexers
how=self.how)
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 985, in _get_join_indexers
lkey, rkey = _get_join_keys(llab, rlab, shape, sort)
File "/homes/mickyl/venvs/debian8/local/lib/python2.7/site-packages/pandas/core/reshape/merge.py", line 1457, in _get_join_keys
stride //= shape[i]
FloatingPointError: divide by zero encountered in long_scalars

The expeted behaviour is for merge to return an empty dataframe without causing division by 0.

Expected Output

just a print-out of all the version numbers with no exception.

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 2.7.9.final.0
python-bits: 64
OS: Linux
OS-release: 4.7.0-0.bpo.1-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.20.3
pytest: 2.6.3
pip: 9.0.1
setuptools: 5.5.1
Cython: 0.25.2
numpy: 1.13.3
scipy: 0.19.1
xarray: None
IPython: 5.4.1
sphinx: 1.2.3
patsy: None
dateutil: 2.6.1
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.1.1
numexpr: 2.6.4
feather: None
matplotlib: 2.0.0
openpyxl: 2.4.8
xlrd: 0.9.2
xlwt: 0.7.5
xlsxwriter: 0.5.2
lxml: 3.4.0
bs4: None
html5lib: 0.999999999
sqlalchemy: 0.9.8
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
pandas_gbq: None
pandas_datareader: None

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 4, 2017

Contributor

you could make a patch to fix this. note that we surround pandas operations with np.seterr(divide='ignore') as a matter of course; we DO want to propagate. In this case it would be checked directly though.

Contributor

jreback commented Oct 4, 2017

you could make a patch to fix this. note that we surround pandas operations with np.seterr(divide='ignore') as a matter of course; we DO want to propagate. In this case it would be checked directly though.

@jreback jreback added this to the Next Major Release milestone Oct 4, 2017

@micklat

This comment has been minimized.

Show comment
Hide comment
@micklat

micklat Oct 8, 2017

I'm sorry, I don't know what the right fix would be. Please remember that I don't understand this codebase like you do.

micklat commented Oct 8, 2017

I'm sorry, I don't know what the right fix would be. Please remember that I don't understand this codebase like you do.

@jreback

This comment has been minimized.

Show comment
Hide comment
@jreback

jreback Oct 8, 2017

Contributor

well you would have to debug this and see where the error is occurring. then surround that with a np.seterr(divid='ignore')

Contributor

jreback commented Oct 8, 2017

well you would have to debug this and see where the error is occurring. then surround that with a np.seterr(divid='ignore')

@yeemey

This comment has been minimized.

Show comment
Hide comment
@yeemey

yeemey Oct 10, 2017

Contributor

Hi, does "make a patch" mean edit the code directly? Sorry, I'm new to this and thought I'd try it, but now am not sure if I did the right thing.

Contributor

yeemey commented Oct 10, 2017

Hi, does "make a patch" mean edit the code directly? Sorry, I'm new to this and thought I'd try it, but now am not sure if I did the right thing.

yeemey added a commit to yeemey/pandas that referenced this issue Oct 13, 2017

yeemey added a commit to yeemey/pandas that referenced this issue Oct 16, 2017

@jreback jreback modified the milestones: Next Major Release, 0.21.1 Nov 26, 2017

jreback added a commit that referenced this issue Nov 27, 2017

TomAugspurger added a commit to TomAugspurger/pandas that referenced this issue Dec 8, 2017

TomAugspurger added a commit that referenced this issue Dec 11, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment