Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Filtering by exclusion of duplicate rows does not preserve column list for an empty dataframe #25184

anatoly-scherbakov opened this issue Feb 6, 2019 · 3 comments


Copy link

commented Feb 6, 2019

Code Sample, a copy-pastable example if possible

import pandas as pd

x_df = pd.DataFrame(columns=['a', 'b'])
series = x_df.duplicated(subset=['a'])


# Expected output on Pandas 0.23.4: ['a', 'b']
# But, Pandas 0.24.1 returns: []

Problem description

We have been using this approach to remove duplicate rows on a dataframe, where rows are compared by one column only. Everything worked perfectly until we found out that, if the original dataframe is empty, in the result dataframe column list is lost after Pandas upgrade to latest version.

Expected Output

We would expect the column list to be preserved.

Output of pd.show_versions()


commit: None
python-bits: 64
OS: Linux
OS-release: 4.15.0-44-generic
machine: x86_64
byteorder: little
LC_ALL: None
LOCALE: None.None

pandas: 0.24.1
pytest: None
pip: 18.0
setuptools: 40.0.0
Cython: 0.22
numpy: 1.12.1
scipy: None
pyarrow: None
xarray: None
IPython: 4.2.0
sphinx: 1.4.4
patsy: None
dateutil: 2.5.0
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: None
openpyxl: 2.5.14
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: None
lxml.etree: 3.7.3
bs4: 4.5.3
html5lib: 1.


This comment has been minimized.

Copy link

commented Feb 6, 2019

Just ran into this myself today.

Looks like the root issue may be that duplicated is now returning an empty series of dtype=float64, whereas it used to return an empty series of dtype=bool.

I've never contributed to pandas before, but I think fixing this would be as simple as changing this line to return Series(dtype=bool) -


This comment has been minimized.

Copy link

commented Feb 7, 2019

@gnilrets : You are more than welcome to investigate this! For information about contributing, see here:

cc @jreback

@gfyoung gfyoung added Index Indexing Regression and removed Index labels Feb 7, 2019

@gfyoung gfyoung added this to the 0.24.2 milestone Feb 7, 2019


This comment has been minimized.

Copy link

commented Feb 7, 2019

yep looks like a regression

@gnilrets gnilrets referenced this issue Feb 9, 2019
4 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
4 participants
You can’t perform that action at this time.