New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pandas 0.22.0 does not raise KeyError for misspelled column with .drop_duplicates() #19726

Closed
aktivkohle opened this Issue Feb 16, 2018 · 6 comments

Comments

Projects
None yet
4 participants
@aktivkohle

aktivkohle commented Feb 16, 2018

So I have tested two versions of Pandas parallel to each other with exactly the same code. 0.19.2 behaves more as expected, but 0.22.0 does what I am about to describe. Will probably switch to 0.19.2 for now. Am using Python 3.6.4

import pandas as pd
df = pd.DataFrame({"A":[34,12,78,84,26], "B":[54,87,35,81,87], "C":[56,78,0,14,13], "D":[0,87,72,87,14], "E":[78,12,31,0,34]}) 

print(df.drop_duplicates(['b','D']))
print(df.drop_duplicates(['B','D']))
print(df.drop_duplicates(['B']))
print(df.drop_duplicates(['D']))

Problem description

I became aware of the problem working with a much larger dataframe when it failed to warn me or raise a KeyError when I misspelled a column name.

Expected Output

Pandas 0.19.2 gives you the following and but Pandas 22 gives you no KeyError for the first print statement it just runs.

KeyError: 'b'

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0
4  26  87  13  14  34

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
3  84  81  14  87   0

    A   B   C   D   E
0  34  54  56   0  78
1  12  87  78  87  12
2  78  35   0  72  31
4  26  87  13  14  34

Output of pd.show_versions()

Below is the output for the Pandas version for where the problem is.

INSTALLED VERSIONS

commit: None

pandas: 0.22.0
pytest: None
pip: 9.0.1
setuptools: 38.4.0
Cython: None
numpy: 1.14.0
scipy: 1.0.0
pyarrow: None
xarray: None
IPython: 6.2.1
sphinx: None
patsy: 0.5.0
dateutil: 2.6.1
pytz: 2017.3
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.1.1
openpyxl: None
xlrd: 1.1.0
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.6.0
html5lib: 1.0.1
sqlalchemy: 1.2.1
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@TomAugspurger

This comment has been minimized.

Contributor

TomAugspurger commented Feb 16, 2018

Thanks for the report. The root cause looks like it's in DataFrame.duplicated.

Interested in submitting a fix?

@TomAugspurger TomAugspurger added this to the Next Major Release milestone Feb 16, 2018

@TomAugspurger

This comment has been minimized.

Contributor

TomAugspurger commented Feb 16, 2018

Whoever fixes this may be able to knock out #12869 at the same time.

@aktivkohle

This comment has been minimized.

aktivkohle commented Feb 16, 2018

@TomAugspurger I might have a crack, see if someone beats me to it. Had a quick look with PyCharm, have not managed to quickly locate that variable / class, it must be in there somewhere..

@TomAugspurger

This comment has been minimized.

Contributor

TomAugspurger commented Feb 16, 2018

Great! I think the offending lines are

pandas/pandas/core/frame.py

Lines 3658 to 3660 in 2fdf1e2

vals = (col.values for name, col in self.iteritems()
if name in subset)
labels, shape = map(list, zip(*map(f, vals)))

Making some bad assumptions about subset. LMK if you have any troubles.

@NoahTheDuke

This comment has been minimized.

Contributor

NoahTheDuke commented Feb 16, 2018

I've got a working fix. Mind if I submit the pull request? Or did you want to take a shot at it, @aktivkohle?

@aktivkohle

This comment has been minimized.

aktivkohle commented Feb 16, 2018

@NoahTheDuke Yes do it, thanks for asking.. I'm sure I'll get my chance one day, and would probably take hours to work it out on the weekend which is not meant for that kind of thing. Will have a look at how you did it of course..

NoahTheDuke added a commit to NoahTheDuke/pandas that referenced this issue Feb 16, 2018

BUG: drop_duplicates not raising KeyError on missing key
Fix pandas-dev#17879 introduced an error by iterating over the columns in the dataframe,
not the columns in the subset. This meant that passing in a column name missing
from the dataframe would no longer raise a `KeyError` like it had previously.

This fix checks the subset first before pulling necessary columns from the
dataframe, and raises the necessary `KeyError` when a given column doesn't
exist.

Fixes pandas-dev#19726

@jreback jreback modified the milestones: Next Major Release, 0.23.0 Feb 18, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment