Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Negate pandas SparseSeries - AssertionError #22835

lukasz0004 opened this issue Sep 26, 2018 · 1 comment


None yet
2 participants
Copy link

commented Sep 26, 2018

Code Sample, a copy-pastable example if possible

df = pd.DataFrame({'a':[0,1,0,3], 'b':[2,0,0,5]})
sparse_df = df.to_sparse(fill_value=0)
criteria = (sparse_df['a'] > 2)
sparse_df_gt_2 = sparse_df.loc[criteria, :].copy()
sparse_df_lt_2 = sparse_df.loc[~criteria, :].copy() # this doesn't work
# simple problem solution:
sparse_df_lt_2 = sparse_df.loc[(criteria == False), :].copy()

Problem description

When I want to do loc selection pandas dataframe with negated pandas SparseSeries it just doesn't work. I know how to solve this problem in another way, but I think there might be something wrong in the code.
This is an error message:

AssertionError Traceback (most recent call last)
in ()
3 criteria = (sparse_df['a'] > 2)
4 sparse_df_gt_2 = sparse_df.loc[criteria, :].copy()
----> 5 sparse_df_lt_2 = sparse_df.loc[~criteria, :].copy()
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/ in invert(self)
1142 try:
1143 arr = operator.inv(com._values_from_object(self))
-> 1144 return self.array_wrap(arr)
1145 except Exception:
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/sparse/ in array_wrap(self, result, context)
281 sparse_index=self.sp_index,
282 fill_value=fill_value,
--> 283 copy=False).finalize(self)
285 def array_finalize(self, obj):
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/sparse/ in init(self, data, index, sparse_index, kind, fill_value, name, dtype, copy, fastpath)
114 data, sparse_index, fill_value = res
115 else:
--> 116 assert (len(data) == sparse_index.npoints)
118 elif isinstance(data, SingleBlockManager):

Expected Output

Output of pd.show_versions()


commit: None
python-bits: 64
OS: Linux
OS-release: 4.15.0-34-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None

pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 39.1.0
Cython: None
numpy: 1.14.5
scipy: 1.1.0
pyarrow: 0.10.0
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 2.2.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: 2.7.5 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: 0.1.6
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@TomAugspurger TomAugspurger changed the title Negate pandas SparseSeries - AssertionError SparseArray doesn't implement __inverse__ Sep 26, 2018

@TomAugspurger TomAugspurger changed the title SparseArray doesn't implement __inverse__ Negate pandas SparseSeries - AssertionError Sep 26, 2018

@TomAugspurger TomAugspurger referenced this issue Sep 26, 2018


SparseArray is an ExtensionArray #22325

4 of 4 tasks complete

This comment has been minimized.

Copy link

commented Sep 26, 2018

Fixing this in #22835

That branch will be merged into master in the next week or two, if you're interested in trying it out.

TomAugspurger added a commit that referenced this issue Oct 13, 2018

[API/REF]: SparseArray is an ExtensionArray (#22325)
Makes SparseArray an ExtensionArray.

* Fixed DataFrame.__setitem__ for updating to sparse.

Closes #22367

* Fixed Series[sparse].to_sparse

Closes #22389

Closes #21978
Closes #19506
Closes #22835

brute4s99 added a commit to brute4s99/pandas that referenced this issue Nov 19, 2018

[API/REF]: SparseArray is an ExtensionArray (pandas-dev#22325)
Makes SparseArray an ExtensionArray.

* Fixed DataFrame.__setitem__ for updating to sparse.

Closes pandas-dev#22367

* Fixed Series[sparse].to_sparse

Closes pandas-dev#22389

Closes pandas-dev#21978
Closes pandas-dev#19506
Closes pandas-dev#22835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.