Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataFrameGroupBy fillna swallows exceptions. #14955

Closed
RobertasA opened this issue Dec 22, 2016 · 3 comments · Fixed by #31161
Closed

DataFrameGroupBy fillna swallows exceptions. #14955

RobertasA opened this issue Dec 22, 2016 · 3 comments · Fixed by #31161
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Milestone

Comments

@RobertasA
Copy link

Problem description

Doing a fillna on DataFrameGroupBy fails silently - returning an empty dataframe, rather than raising what the error is.
This often masks the actual problem (e.g. errors like
ValueError: Tz-aware datetime.datetime cannot be converted to datetime64 unless utc=True when handling tz-aware data.)

Code Sample, a copy-pastable example if possible

>>> df = pd.DataFrame({'a':[1,2],'b':[1,1]})
>>> df.groupby('b').fillna()
Empty DataFrame
Columns: []
Index: []

Expected Output

ValueError: must specify a fill method or value

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.10.final.0 python-bits: 64 OS: Darwin OS-release: 15.6.0 machine: x86_64 processor: i386 byteorder: little LC_ALL: None LANG: en_GB.UTF-8 LOCALE: None.None

pandas: 0.19.1
nose: 1.3.7
pip: 1.5.6
setuptools: 25.1.1
Cython: 0.23.2
numpy: 1.11.1
scipy: 0.14.0
statsmodels: 0.6.1
xarray: None
IPython: 2.3.1
sphinx: None
patsy: 0.4.1
dateutil: 2.5.3
pytz: 2016.6.1
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.9
apiclient: 1.1
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.7.3
boto: 2.26.0
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Dec 22, 2016

xref to https://github.com/pandas-dev/pandas/pull/6338/files

we do raise when using apply like this.

In [2]: g.apply(lambda x: x.fillna())
ValueError: must specify a fill method or value

This might be quite tricky though to differentiate between a dtype incompat (e.g. ignoring a column) and a 'real' error.

e.g the same path is hit for

In [4]: df['C'] = ['foo', 'bar']
In [7]: df.groupby('a').mean()
Out[7]: 
   b
a   
1  1
2  1

@jreback jreback added Difficulty Intermediate Error Reporting Incorrect or improved errors from pandas Groupby labels Dec 22, 2016
@jreback jreback added this to the Next Major Release milestone Dec 22, 2016
@jreback
Copy link
Contributor

jreback commented Dec 22, 2016

please have a look see and see what can be done

@mroeschke
Copy link
Member

mroeschke commented Oct 26, 2019

Looks to raise correctly on master. Could use a test.

In [121]: >>> df = pd.DataFrame({'a':[1,2],'b':[1,1]})
     ...: >>> df.groupby('b').fillna()

ValueError: Must specify a fill 'value' or 'method'.

In [122]: pd.__version__
Out[122]: '0.26.0.dev0+682.g08ab156eb'

@mroeschke mroeschke added good first issue Needs Tests Unit test(s) needed to prevent regressions and removed Error Reporting Incorrect or improved errors from pandas Groupby labels Oct 26, 2019
@jreback jreback modified the milestones: Contributions Welcome, 1.1 Jan 20, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
good first issue Needs Tests Unit test(s) needed to prevent regressions
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants