BUG: Add SparseArray.all #17570

Licht-T · 2017-09-18T11:44:50Z

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

This is the part of #17386.
Block.where uses ndarray.all, but there is no such method in SparseArray.
This makes that all evaluates SparseArray([False, False, True]) as True.
https://github.com/pandas-dev/pandas/blob/master/pandas/core/internals.py#L1400

jreback · 2017-09-18T12:09:17Z

pandas/core/sparse/array.py

+        values = self.sp_values
+
+        if len(values) != len(self):
+            values = values.tolist()


is there a reason you are converting to a list? and not a dense array?

@jreback For the performance reason. This is sparse and self.to_dense() is not efficient.

I am asking why you are converting .tolist(), then appending, that is non-performant.

Simply I wanted to use np.all. I got an idea not to use tolist(). I'll change the solution.

jreback

pls add a whatsnew note. do we have a specific issue for this? (other that the PR)?

jreback · 2017-09-18T12:09:53Z

pandas/core/sparse/array.py

@@ -614,6 +614,24 @@ def fillna(self, value, downcast=None):
        return self._simple_new(new_values, self.sp_index,
                                fill_value=fill_value)

+    def all(self, axis=0, *args, **kwargs):
+        """


might as well add any as well.

Licht-T · 2017-09-18T13:40:44Z

@jreback Thanks for your review.

Changed the fill_value check method and now we are not using tolist().
Removed dtype parameter validation check.
There is no dtype parameter on ndarray.all.
Also added SparseArray.any.
There seems to be no specific issue for this.

jreback

minor comment. ping on green

jreback · 2017-09-21T13:47:50Z

doc/source/whatsnew/v0.21.0.txt

@@ -544,6 +544,7 @@ Sparse

 - Bug in ``SparseSeries`` raises ``AttributeError`` when a dictionary is passed in as data (:issue:`16905`)
 - Bug in :func:`SparseDataFrame.fillna` not filling all NaNs when frame was instantiated from SciPy sparse matrix (:issue:`16112`)


say that these are now implemented to handle SparseArray

jreback · 2017-09-21T13:48:34Z

pandas/core/sparse/array.py

+
+        Returns
+        -------
+        all : bool


add a See Also to point to np.all

codecov · 2017-09-21T15:06:07Z

Codecov Report

Merging #17570 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #17570      +/-   ##
==========================================
- Coverage   91.22%    91.2%   -0.02%     
==========================================
  Files         163      163              
  Lines       49625    49642      +17     
==========================================
+ Hits        45270    45277       +7     
- Misses       4355     4365      +10

Flag	Coverage Δ
#multiple	`88.99% <100%> (ø)`	⬆️
#single	`40.19% <41.17%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/compat/numpy/function.py	`93.33% <100%> (+0.2%)`	⬆️
pandas/core/sparse/array.py	`91.31% <100%> (+0.01%)`	⬆️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/core/frame.py	`97.77% <0%> (-0.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 37e23d0...5d04485. Read the comment docs.

codecov · 2017-09-21T15:06:08Z

Codecov Report

Merging #17570 into master will decrease coverage by 0.04%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #17570      +/-   ##
==========================================
- Coverage   91.24%    91.2%   -0.05%     
==========================================
  Files         163      163              
  Lines       49766    49642     -124     
==========================================
- Hits        45411    45276     -135     
- Misses       4355     4366      +11

Flag	Coverage Δ
#multiple	`88.99% <100%> (-0.04%)`	⬇️
#single	`40.19% <41.17%> (-0.21%)`	⬇️

Impacted Files	Coverage Δ
pandas/compat/numpy/function.py	`93.33% <100%> (+0.2%)`	⬆️
pandas/core/sparse/array.py	`91.31% <100%> (-0.28%)`	⬇️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/util/_decorators.py	`66% <0%> (-12%)`	⬇️
pandas/core/dtypes/missing.py	`87.19% <0%> (-3.26%)`	⬇️
pandas/core/base.py	`96.01% <0%> (-0.56%)`	⬇️
pandas/core/indexes/range.py	`92.59% <0%> (-0.25%)`	⬇️
pandas/tseries/offsets.py	`97% <0%> (-0.15%)`	⬇️
pandas/core/categorical.py	`95.57% <0%> (-0.14%)`	⬇️
... and 32 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 074b485...090f398. Read the comment docs.

Licht-T · 2017-09-21T15:07:13Z

@jreback Thank you. These are now fixed.

Licht-T · 2017-09-22T01:12:29Z

@jreback I found the bug in this solution. When fill_value = np.array([1,1,1]), np.all(fill_value) returns True. In numpy, np.array([np.array([1,1,1]), 0, np.array([1,1,1])], dtype=np.object).all() raise ValueError, but this is not. I'll fix.

UPDATE: I found SparseArray does not allow ndarray as fill_value, so this will not happen. I am sorry for the mistake.

jreback · 2017-09-28T00:30:22Z

lgtm. waiting for CI to go green again before merging (unrelated conda-ish issues)

jreback · 2017-09-28T14:40:12Z

can you rebase

Licht-T · 2017-09-28T16:58:26Z

@jreback Rebased.

jreback · 2017-09-28T23:44:34Z

thanks @Licht-T

jreback reviewed Sep 18, 2017

View reviewed changes

jreback requested changes Sep 18, 2017

View reviewed changes

jreback reviewed Sep 18, 2017

View reviewed changes

jreback added Bug Sparse Sparse Data Type labels Sep 18, 2017

Licht-T force-pushed the add-sparsearray-all branch from b5bb960 to 19e3e44 Compare September 18, 2017 13:51

jreback approved these changes Sep 21, 2017

View reviewed changes

jreback added this to the 0.21.0 milestone Sep 28, 2017

Licht-T added 5 commits September 29, 2017 00:39

BUG: Add SparseArray.all

cc4776b

TST: Add tests of SparseArray.all

6e1c915

BUG: Add SparseArray.any

9e506a4

TST: Add tests of SparseArray.any

f5b4f6b

DOC: Add SparseArray.all and SparseArray.any to whatsnew note

94973e8

Licht-T force-pushed the add-sparsearray-all branch from 2babd92 to 94973e8 Compare September 28, 2017 15:42

Merge branch 'master' into add-sparsearray-all

090f398

jreback merged commit bbf0dda into pandas-dev:master Sep 28, 2017

alanbato pushed a commit to alanbato/pandas that referenced this pull request Nov 10, 2017

BUG: Add SparseArray.all (pandas-dev#17570)

ce7f100

No-Stream pushed a commit to No-Stream/pandas that referenced this pull request Nov 28, 2017

BUG: Add SparseArray.all (pandas-dev#17570)

6dd73c1

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Add SparseArray.all #17570

BUG: Add SparseArray.all #17570

Licht-T commented Sep 18, 2017 •

edited

jreback Sep 18, 2017

Licht-T Sep 18, 2017 •

edited

jreback Sep 18, 2017

Licht-T Sep 18, 2017

jreback left a comment

jreback Sep 18, 2017

Licht-T commented Sep 18, 2017 •

edited

jreback left a comment

jreback Sep 21, 2017

jreback Sep 21, 2017

codecov bot commented Sep 21, 2017

codecov bot commented Sep 21, 2017 •

edited

Licht-T commented Sep 21, 2017

Licht-T commented Sep 22, 2017 •

edited

jreback commented Sep 28, 2017

jreback commented Sep 28, 2017

Licht-T commented Sep 28, 2017

jreback commented Sep 28, 2017

		@@ -544,6 +544,7 @@ Sparse

		- Bug in ``SparseSeries`` raises ``AttributeError`` when a dictionary is passed in as data (:issue:`16905`)
		- Bug in :func:`SparseDataFrame.fillna` not filling all NaNs when frame was instantiated from SciPy sparse matrix (:issue:`16112`)

BUG: Add SparseArray.all #17570

BUG: Add SparseArray.all #17570

Conversation

Licht-T commented Sep 18, 2017 • edited

jreback Sep 18, 2017

Choose a reason for hiding this comment

Licht-T Sep 18, 2017 • edited

Choose a reason for hiding this comment

jreback Sep 18, 2017

Choose a reason for hiding this comment

Licht-T Sep 18, 2017

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback Sep 18, 2017

Choose a reason for hiding this comment

Licht-T commented Sep 18, 2017 • edited

jreback left a comment

Choose a reason for hiding this comment

jreback Sep 21, 2017

Choose a reason for hiding this comment

jreback Sep 21, 2017

Choose a reason for hiding this comment

codecov bot commented Sep 21, 2017

Codecov Report

codecov bot commented Sep 21, 2017 • edited

Codecov Report

Licht-T commented Sep 21, 2017

Licht-T commented Sep 22, 2017 • edited

jreback commented Sep 28, 2017

jreback commented Sep 28, 2017

Licht-T commented Sep 28, 2017

jreback commented Sep 28, 2017

Licht-T commented Sep 18, 2017 •

edited

Licht-T Sep 18, 2017 •

edited

Licht-T commented Sep 18, 2017 •

edited

codecov bot commented Sep 21, 2017 •

edited

Licht-T commented Sep 22, 2017 •

edited