Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dropna(subset=...) does not accept incomplete key #17737

Open
toobaz opened this issue Oct 2, 2017 · 1 comment
Open

dropna(subset=...) does not accept incomplete key #17737

toobaz opened this issue Oct 2, 2017 · 1 comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments

@toobaz
Copy link
Member

toobaz commented Oct 2, 2017

Code Sample, a copy-pastable example if possible

In [3]: df = pd.DataFrame(index=range(3), columns=pd.MultiIndex.from_product([['I', 'II'], ['a', 'b']]))

In [4]: df.dropna(subset=['I'])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-2ec2c1f3958a> in <module>()
----> 1 df.dropna(subset=['I'])

/home/pietro/nobackup/repo/pandas/pandas/core/frame.py in dropna(self, axis, how, thresh, subset, inplace)
   3339                 check = indices == -1
   3340                 if check.any():
-> 3341                     raise KeyError(list(np.compress(check, subset)))
   3342                 agg_obj = self.take(indices, axis=agg_axis)
   3343 

KeyError: ['I']

... which is particularly unexpected when doing:

In [5]: df['other'] = (1, 3, np.nan)

In [6]: df.dropna(subset=['other'])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-5d8972ce7a1c> in <module>()
----> 1 df.dropna(subset=['other'])

/home/pietro/nobackup/repo/pandas/pandas/core/frame.py in dropna(self, axis, how, thresh, subset, inplace)
   3339                 check = indices == -1
   3340                 if check.any():
-> 3341                     raise KeyError(list(np.compress(check, subset)))
   3342                 agg_obj = self.take(indices, axis=agg_axis)
   3343 

KeyError: ['other']

Problem description

dropna(subset=...) should mimic the behaviour of indexing with incomplete MultiIndex keys.

Expected Output

In [7]: df.dropna(subset=[('other', '')])
Out[7]: 
     I        II      other
     a    b    a    b      
0  NaN  NaN  NaN  NaN   1.0
1  NaN  NaN  NaN  NaN   3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.21.0.dev+546.gad7d051bd
pytest: 3.0.6
pip: 9.0.1
setuptools: None
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.1.0.dev
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

@sinhrks sinhrks added API Design Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate labels Oct 6, 2017
@mroeschke mroeschke added Bug and removed API Design labels Jun 12, 2021
@ashishthapa14
Copy link

can somebody explain to me, why dropna subset method doesn't work with columns?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants