Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dropna(subset=...) does not accept incomplete key #17737

Open
toobaz opened this Issue Oct 2, 2017 · 0 comments

Comments

Projects
None yet
2 participants
@toobaz
Copy link
Member

toobaz commented Oct 2, 2017

Code Sample, a copy-pastable example if possible

In [3]: df = pd.DataFrame(index=range(3), columns=pd.MultiIndex.from_product([['I', 'II'], ['a', 'b']]))

In [4]: df.dropna(subset=['I'])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-4-2ec2c1f3958a> in <module>()
----> 1 df.dropna(subset=['I'])

/home/pietro/nobackup/repo/pandas/pandas/core/frame.py in dropna(self, axis, how, thresh, subset, inplace)
   3339                 check = indices == -1
   3340                 if check.any():
-> 3341                     raise KeyError(list(np.compress(check, subset)))
   3342                 agg_obj = self.take(indices, axis=agg_axis)
   3343 

KeyError: ['I']

... which is particularly unexpected when doing:

In [5]: df['other'] = (1, 3, np.nan)

In [6]: df.dropna(subset=['other'])
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-6-5d8972ce7a1c> in <module>()
----> 1 df.dropna(subset=['other'])

/home/pietro/nobackup/repo/pandas/pandas/core/frame.py in dropna(self, axis, how, thresh, subset, inplace)
   3339                 check = indices == -1
   3340                 if check.any():
-> 3341                     raise KeyError(list(np.compress(check, subset)))
   3342                 agg_obj = self.take(indices, axis=agg_axis)
   3343 

KeyError: ['other']

Problem description

dropna(subset=...) should mimic the behaviour of indexing with incomplete MultiIndex keys.

Expected Output

In [7]: df.dropna(subset=[('other', '')])
Out[7]: 
     I        II      other
     a    b    a    b      
0  NaN  NaN  NaN  NaN   1.0
1  NaN  NaN  NaN  NaN   3.0

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.5.3.final.0
python-bits: 64
OS: Linux
OS-release: 4.9.0-3-amd64
machine: x86_64
processor:
byteorder: little
LC_ALL: None
LANG: it_IT.UTF-8
LOCALE: it_IT.UTF-8

pandas: 0.21.0.dev+546.gad7d051bd
pytest: 3.0.6
pip: 9.0.1
setuptools: None
Cython: 0.25.2
numpy: 1.12.1
scipy: 0.19.0
pyarrow: None
xarray: None
IPython: 5.1.0.dev
sphinx: 1.5.6
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2017.2
blosc: None
bottleneck: 1.2.1
tables: 3.3.0
numexpr: 2.6.1
feather: 0.3.1
matplotlib: 2.0.2
openpyxl: None
xlrd: 1.0.0
xlwt: 1.1.2
xlsxwriter: 0.9.6
lxml: None
bs4: 4.5.3
html5lib: 0.999999999
sqlalchemy: 1.0.15
pymysql: None
psycopg2: None
jinja2: 2.9.6
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.2.1

@alexander-ponomaroff alexander-ponomaroff referenced a pull request that will close this issue Feb 22, 2019

Open

dropna(subset=...) should accept incomplete key #25407

4 of 4 tasks complete
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.