Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Negate pandas SparseSeries - AssertionError #22835

Closed
lukasz0004 opened this issue Sep 26, 2018 · 1 comment · Fixed by #22325
Closed

Negate pandas SparseSeries - AssertionError #22835

lukasz0004 opened this issue Sep 26, 2018 · 1 comment · Fixed by #22325
Labels
Sparse Sparse Data Type

Comments

@lukasz0004
Copy link

Code Sample, a copy-pastable example if possible

df = pd.DataFrame({'a':[0,1,0,3], 'b':[2,0,0,5]})
sparse_df = df.to_sparse(fill_value=0)
criteria = (sparse_df['a'] > 2)
sparse_df_gt_2 = sparse_df.loc[criteria, :].copy()
sparse_df_lt_2 = sparse_df.loc[~criteria, :].copy() # this doesn't work
# simple problem solution:
sparse_df_lt_2 = sparse_df.loc[(criteria == False), :].copy()

Problem description

When I want to do loc selection pandas dataframe with negated pandas SparseSeries it just doesn't work. I know how to solve this problem in another way, but I think there might be something wrong in the code.
This is an error message:

AssertionError Traceback (most recent call last)
in ()
3 criteria = (sparse_df['a'] > 2)
4 sparse_df_gt_2 = sparse_df.loc[criteria, :].copy()
----> 5 sparse_df_lt_2 = sparse_df.loc[~criteria, :].copy()
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/generic.py in invert(self)
1142 try:
1143 arr = operator.inv(com._values_from_object(self))
-> 1144 return self.array_wrap(arr)
1145 except Exception:
1146
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/sparse/series.py in array_wrap(self, result, context)
281 sparse_index=self.sp_index,
282 fill_value=fill_value,
--> 283 copy=False).finalize(self)
284
285 def array_finalize(self, obj):
~/Projects/ml_projects/ml_python_venv/lib/python3.6/site-packages/pandas/core/sparse/series.py in init(self, data, index, sparse_index, kind, fill_value, name, dtype, copy, fastpath)
114 data, sparse_index, fill_value = res
115 else:
--> 116 assert (len(data) == sparse_index.npoints)
117
118 elif isinstance(data, SingleBlockManager):
AssertionError:

Expected Output

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.15.0-34-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8

pandas: 0.23.4
pytest: None
pip: 18.0
setuptools: 39.1.0
Cython: None
numpy: 1.14.5
scipy: 1.1.0
pyarrow: 0.10.0
xarray: None
IPython: 6.5.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.5
blosc: None
bottleneck: None
tables: 3.4.4
numexpr: 2.6.8
feather: None
matplotlib: 2.2.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: 1.0.1
sqlalchemy: None
pymysql: None
psycopg2: 2.7.5 (dt dec pq3 ext lo64)
jinja2: 2.10
s3fs: 0.1.6
fastparquet: None
pandas_gbq: None
pandas_datareader: None

@TomAugspurger TomAugspurger added the Sparse Sparse Data Type label Sep 26, 2018
@TomAugspurger TomAugspurger changed the title Negate pandas SparseSeries - AssertionError SparseArray doesn't implement __inverse__ Sep 26, 2018
@TomAugspurger TomAugspurger changed the title SparseArray doesn't implement __inverse__ Negate pandas SparseSeries - AssertionError Sep 26, 2018
@TomAugspurger
Copy link
Contributor

Fixing this in #22835

That branch will be merged into master in the next week or two, if you're interested in trying it out.

TomAugspurger added a commit that referenced this issue Oct 13, 2018
Makes SparseArray an ExtensionArray.

* Fixed DataFrame.__setitem__ for updating to sparse.

Closes #22367

* Fixed Series[sparse].to_sparse

Closes #22389

Closes #21978
Closes #19506
Closes #22835
tm9k1 pushed a commit to tm9k1/pandas that referenced this issue Nov 19, 2018
Makes SparseArray an ExtensionArray.

* Fixed DataFrame.__setitem__ for updating to sparse.

Closes pandas-dev#22367

* Fixed Series[sparse].to_sparse

Closes pandas-dev#22389

Closes pandas-dev#21978
Closes pandas-dev#19506
Closes pandas-dev#22835
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Sparse Sparse Data Type
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants