Size and density properties #69

nils-werner · Jan 10, 2018

NumPy has a .size property that gives you the number of elements in an array.

Additionally for sparse arrays a .density property may be handy, which is the ratio of nonzero elements to total elements in the array (a value between 0 and 1).

This PR implements both properties for COO arrays

hameerabbasi · Jan 10, 2018

Seems fine, however it would be really nice to have docstrings for these, and if possible, tests.

nils-werner · Jan 10, 2018

Added them

hameerabbasi · Jan 10, 2018

Ignore the Travis CI failure. Numpy 1.14 changed the way it converts arrays to strings, and that's causing a doctests failure. I've addressed it over in #43.

nils-werner · Jan 10, 2018

Yup, I already noticed that

hameerabbasi · Jan 10, 2018

Well, this looks good. Let me know if you're finished working on it, and I'll review and merge (don't want to be too fast this time).

hameerabbasi · Jan 10, 2018

hameerabbasi reviewed Jan 10, 2018

View changes

hameerabbasi · Jan 10, 2018

sparse/tests/test_core.py

+
+def test_density():
+    s = sparse.random((20, 30, 40), density=0.1)
+    assert 0.09 < s.density < 0.11


It might be wise to use np.isclose here.

hameerabbasi · Jan 10, 2018

hameerabbasi reviewed Jan 10, 2018

View changes

hameerabbasi · Jan 10, 2018

sparse/core.py

+        >>> x[0, :] = 1
+        >>> s = COO.from_numpy(x)
+        >>> s.density
+        0.10000000000000001


This might fluctuate in different system floating-point implementations. Perhaps np.isclose(density, 0.1)?

mrocklin · Jan 10, 2018

Or perhaps change 10 to 8 above so that the result is evenly divisible by two

…

On Wed, Jan 10, 2018 at 12:08 PM, Hameer Abbasi ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sparse/core.py <https://github.com/mrocklin/sparse/pull/69#discussion_r160755728>: > + ------- + float + The ratio of nonzero to all elements. + + See Also + -------- + COO.size : Number of elements. + COO.nnz : Number of nonzero elements. + + Examples + -------- + >>> x = np.zeros((10, 10)) + >>> x[0, :] = 1 + >>> s = COO.from_numpy(x) + >>> s.density + 0.10000000000000001 This might fluctuate in different system floating-point implementations. Perhaps np.isclose(density, 0.1)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/mrocklin/sparse/pull/69#pullrequestreview-87928365>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszEWhBJgW3Nr0LVCaGKs4YQy3xdscks5tJPyjgaJpZM4RZBRy> .

nils-werner · Jan 11, 2018

@mrocklin that's clever! Done.

hameerabbasi · Jan 11, 2018

This looks ready. If you merge, I can do a final review and merge.

nils-werner · Jan 12, 2018

Did you see my code comment?

hameerabbasi · Jan 12, 2018

Hmmm, that's strange, it isn't showing any of your comments in that link, or on any commits.

nils-werner · Jan 12, 2018

nils-werner reviewed Jan 12, 2018

View changes

nils-werner · Jan 12, 2018

sparse/core.py

@@ -1277,7 +1325,7 @@ def astype(self, dtype, out=None):

    def maybe_densify(self, allowed_nnz=1e3, allowed_fraction=0.25):
        """ Convert to a dense numpy array if not too costly.  Err othrewise """
-        if reduce(operator.mul, self.shape) <= allowed_nnz or self.nnz >= np.prod(self.shape) * allowed_fraction:
+        if self.size <= allowed_nnz or self.density >= allowed_fraction:


It is easy to see that

self.nnz >= np.prod(self.shape) * allowed_fraction

can be transformed to

self.density >= allowed_fraction

But does self.density >= allowed_fraction actually make sense? Shouldn't it be the other way round?

It actually does make sense. If the density is low, then the cost of a dense array is higher compared to that of a sparse array, and we should avoid densifying.

If, however, density is high, the overhead of a sparse matrix far outweighs the benefits and we should densify.

To put it another way, a large array with a very low density should NOT be densified but it's okay to densify a high-density or a small array.

True. Maybe I got confused that one allowed means max and the other one means min... What about renaming them to

maybe_densify(self, max_nnz=1e3, min_density=0.25)

I would make sure that no tests break either here or in Dask before doing that. We can change unit tests here... But it's best to send a PR to Dask if any tests break there.

Let's change it anyway and I'll send a PR to Dask if anything fails for them.

Actually, shouldn't max_nnz be actually named max_size? We are not comparing it to .nnz anyways...

Yes, max_size is a good idea as well.

nils-werner · Jan 12, 2018

Aha! after commenting on lines you have to also open the code review thingy.

hameerabbasi · Jan 15, 2018

Is NotImplementedError the best way to show that densfication is forbidden? What would be the right exception here?

nils-werner · Jan 15, 2018

I would say ValueError is the right one, as it's the choice of function arguments that make the operation fail.

A NotImplementedError sounds more like "whatever you pass to this function, in this version it will never succeed".

hameerabbasi · Jan 15, 2018

Best to switch that to a ValueError then.

hameerabbasi · Jan 15, 2018

hameerabbasi reviewed Jan 15, 2018

View changes

hameerabbasi · Jan 15, 2018

sparse/core.py

        an error.

        >>> x = np.zeros((5, 5), dtype=np.uint8)
        >>> x[2, 2] = 1
        >>> s = COO.from_numpy(x)
-        >>> s.maybe_densify(allowed_nnz=5, allowed_fraction=0.25)
+        >>> s.maybe_densify(max_size=5, min_fraction=0.25)
        Traceback (most recent call last):
            ...
        NotImplementedError: Operation would require converting large sparse array to dense


You need to update this line in the docstring. See here.

hameerabbasi · Jan 15, 2018

hameerabbasi reviewed Jan 15, 2018

View changes

hameerabbasi · Jan 15, 2018

sparse/core.py

@@ -2381,17 +2381,17 @@ def astype(self, dtype, out=None):
        assert out is None
        return self.elemwise(np.ndarray.astype, dtype)

-    def maybe_densify(self, allowed_nnz=1000, allowed_fraction=0.25):
+    def maybe_densify(self, max_size=1000, min_fraction=0.25):


Best keep this as min_density, it's clearer that it represents the density.

hameerabbasi · Jan 15, 2018

hameerabbasi reviewed Jan 15, 2018

View changes

hameerabbasi · Jan 15, 2018

sparse/core.py

        an error.

        >>> x = np.zeros((5, 5), dtype=np.uint8)
        >>> x[2, 2] = 1
        >>> s = COO.from_numpy(x)
-        >>> s.maybe_densify(allowed_nnz=5, allowed_fraction=0.25)
+        >>> s.maybe_densify(max_size=5, min_density=0.25)
        Traceback (most recent call last):
            ...
        NotImplementedError: Operation would require converting large sparse array to dense


Line 2425...

NotImplementedError: Operation would require converting large sparse array to dense

Need to change this to ValueError.

It's causing doctests to fail. :)

Currently there are a million tests failing on master, it is hard to spot my own mistakes...

That's strange, Travis shows everything is working on master.

I checked, they only fail until Numpy 1.13.1. Latest Numpy works fine...

That is really weird, I tested with Numpy 1.13.3 on my local system and it still works fine.

mrocklin · Jan 15, 2018

Is NotImplementedError the best way to show that densfication is forbidden? What would be the right exception here?

To me NotImplementedError means "We have not yet implemented this, but it is a reasonable request". If it is a case of a type not being supported like "we don't think that densifying scipy.sparse matrices is a good idea" then I would raise a TypeError. If it is a case of the values being inappropriate like "sometimes we do densify sparse arrays, but this particular array is very very sparse" then I would use a ValueError.

hameerabbasi · Jan 15, 2018

So, we all agree that it should be ValueError in maybe_densify then. :-)

hameerabbasi · Jan 15, 2018

Are you still making changes here? If not, this gets a +1 from me.

nils-werner · Jan 16, 2018

No, I'm done.

hameerabbasi · Jan 16, 2018

A rebase would be nice.

nils-werner force-pushed the nils-werner:density branch from 1fa7541 to 832b5ab Jan 15, 2018

nils-werner referenced this pull request Jan 15, 2018
Merged
Tox for cross Python-version testing #77

hameerabbasi added this to the 0.3 milestone Jan 15, 2018

nils-werner added some commits Jan 10, 2018

Size and density properties

b16748b

Docstrings and tests

0e3c48d

nils-werner added some commits Jan 10, 2018

Test .density using np.isclose

6d32034

Fixed typos that made tests fail

2be3a3b

hameerabbasi added the type:enhancement label Jan 16, 2018

nils-werner force-pushed the nils-werner:density branch from 647fc33 to a65e1d0 Jan 16, 2018

hameerabbasi merged commit bbb0869 into pydata:master Jan 16, 2018
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

pydata/sparse

Join GitHub today

Size and density properties #69

Conversation

nils-werner commented Jan 10, 2018 • edited

This comment has been minimized.

hameerabbasi commented Jan 10, 2018

This comment has been minimized.

nils-werner commented Jan 10, 2018

This comment has been minimized.

hameerabbasi commented Jan 10, 2018

This comment has been minimized.

nils-werner commented Jan 10, 2018

This comment has been minimized.

hameerabbasi commented Jan 10, 2018

hameerabbasi reviewed Jan 10, 2018 View changes

This comment has been minimized.

hameerabbasi reviewed Jan 10, 2018 View changes

This comment has been minimized.

This comment has been minimized.

mrocklin commented Jan 10, 2018

This comment has been minimized.

nils-werner commented Jan 11, 2018

This comment has been minimized.

hameerabbasi commented Jan 11, 2018 • edited

This comment has been minimized.

nils-werner commented Jan 12, 2018

This comment has been minimized.

hameerabbasi commented Jan 12, 2018

nils-werner reviewed Jan 12, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nils-werner commented Jan 12, 2018

This comment has been minimized.

hameerabbasi commented Jan 15, 2018

This comment has been minimized.

nils-werner commented Jan 15, 2018 • edited

This comment has been minimized.

hameerabbasi commented Jan 15, 2018

nils-werner force-pushed the nils-werner:density branch from 1fa7541 to 832b5ab Jan 15, 2018

hameerabbasi reviewed Jan 15, 2018 View changes

This comment has been minimized.

hameerabbasi reviewed Jan 15, 2018 View changes

This comment has been minimized.

hameerabbasi reviewed Jan 15, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

nils-werner Jan 15, 2018 • edited

This comment has been minimized.

This comment has been minimized.

mrocklin commented Jan 15, 2018

nils-werner referenced this pull request Jan 15, 2018

This comment has been minimized.

hameerabbasi commented Jan 15, 2018

This comment has been minimized.

hameerabbasi commented Jan 15, 2018

hameerabbasi added this to the 0.3 milestone Jan 15, 2018

This comment has been minimized.

nils-werner commented Jan 16, 2018

This comment has been minimized.

hameerabbasi commented Jan 16, 2018

nils-werner added some commits Jan 10, 2018

nils-werner added some commits Jan 10, 2018

hameerabbasi added the type:enhancement label Jan 16, 2018

nils-werner added some commits Jan 10, 2018

nils-werner force-pushed the nils-werner:density branch from 647fc33 to a65e1d0 Jan 16, 2018

Hide details View details hameerabbasi merged commit bbb0869 into pydata:master Jan 16, 2018 1 check passed

1 check passed

hameerabbasi added a commit to hameerabbasi/sparse that referenced this pull request Feb 27, 2018

nils-werner commented Jan 10, 2018 •

edited

hameerabbasi reviewed Jan 10, 2018

View changes

hameerabbasi reviewed Jan 10, 2018

View changes

hameerabbasi commented Jan 11, 2018 •

edited

nils-werner reviewed Jan 12, 2018

View changes

nils-werner commented Jan 15, 2018 •

edited

nils-werner force-pushed the nils-werner:density branch from `1fa7541` to `832b5ab` Jan 15, 2018

hameerabbasi reviewed Jan 15, 2018

View changes

hameerabbasi reviewed Jan 15, 2018

View changes

hameerabbasi reviewed Jan 15, 2018

View changes

nils-werner Jan 15, 2018 •

edited

nils-werner force-pushed the nils-werner:density branch from `647fc33` to `a65e1d0` Jan 16, 2018

hameerabbasi merged commit `bbb0869` into pydata:master Jan 16, 2018
1 check passed