Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

groupby.idxmin() returns datetime values for datatime columns #25444

Closed
tmitanitky opened this issue Feb 26, 2019 · 4 comments

Comments

@tmitanitky
Copy link

commented Feb 26, 2019

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame({'name': ['A', 'A', 'A', 'B', 'B'], 'a':[1,2,3,4,5], 'date': ['2016', '2017', '2018', '2016', '2017']}, )
df['date'] = pd.to_datetime(df['date'])

#current behaviour
df.groupby('name').idxmin()
>>       a                          date
>> name                                 
>> A     0 1970-01-01 00:00:00.000000000
>> B     3 1970-01-01 00:00:00.000000003

Problem description

The returning df's dtypes would be the same as that of index.
Thanks.

Expected Output

# expected behaviour
df.groupby('name').idxmin().astype('int')
>>       a  date
>> name         
>> A     0     0
>> B     3     3

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Darwin
OS-release: 18.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: ja_JP.UTF-8
LOCALE: ja_JP.UTF-8

pandas: 0.23.4
pytest: 4.0.1
pip: 18.1
setuptools: 40.6.2
Cython: 0.29
numpy: 1.15.4
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 7.2.0
sphinx: 1.8.2
patsy: 0.5.1
dateutil: 2.7.5
pytz: 2018.7
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 3.0.2
openpyxl: 2.5.11
xlrd: 1.1.0
xlwt: 1.2.0
xlsxwriter: 1.1.2
lxml: 4.2.5
bs4: 4.6.3
html5lib: 1.0.1
sqlalchemy: 1.2.14
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@gfyoung

This comment has been minimized.

Copy link
Member

commented Feb 27, 2019

Yep, that's a bug indeed. Investigation and PR are welcome!

@rbenes

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2019

I identified the problem. It is in pandas/core/groupby/generic.py:257 where the casting is applyed, but for idxmin, idxmax (and other?) the result should not be casted to original types ... possible solution? https://github.com/pandas-dev/pandas/compare/master...rbenes:bug_groupby_idxmin?expand=1 (but I am not sure with it - I do not want to push spaghetti code)

@gfyoung

This comment has been minimized.

Copy link
Member

commented Mar 4, 2019

@rbenes : I would first make sure that your changes don't break existing tests. Don't worry about the "spaghetti"-nature of your code. When you submit the PR, we'll be more than happy to help you figure out a way to "de-spaghetti-fy" it before merging.

How does that sound?

@rbenes

This comment has been minimized.

Copy link
Contributor

commented Mar 4, 2019

fast tests seems ok (on my PC)... ok, I will do the pull request.

@rbenes rbenes referenced this issue Mar 4, 2019

Merged

Bug groupby idxmin #25531

2 of 3 tasks complete

@jreback jreback added this to the 0.25.0 milestone Mar 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
4 participants
You can’t perform that action at this time.