Size and density properties #69

nils-werner · 2018-01-10T09:17:11Z

NumPy has a .size property that gives you the number of elements in an array.

Additionally for sparse arrays a .density property may be handy, which is the ratio of nonzero elements to total elements in the array (a value between 0 and 1).

This PR implements both properties for COO arrays

hameerabbasi · 2018-01-10T14:24:43Z

Seems fine, however it would be really nice to have docstrings for these, and if possible, tests.

nils-werner · 2018-01-10T14:46:06Z

Added them

hameerabbasi · 2018-01-10T14:48:36Z

Ignore the Travis CI failure. Numpy 1.14 changed the way it converts arrays to strings, and that's causing a doctests failure. I've addressed it over in #43.

nils-werner · 2018-01-10T14:50:09Z

Yup, I already noticed that

hameerabbasi · 2018-01-10T14:51:38Z

Well, this looks good. Let me know if you're finished working on it, and I'll review and merge (don't want to be too fast this time).

hameerabbasi · 2018-01-10T14:56:45Z

sparse/tests/test_core.py

+
+def test_density():
+    s = sparse.random((20, 30, 40), density=0.1)
+    assert 0.09 < s.density < 0.11


It might be wise to use np.isclose here.

hameerabbasi · 2018-01-10T18:08:34Z

sparse/core.py

+        >>> x[0, :] = 1
+        >>> s = COO.from_numpy(x)
+        >>> s.density
+        0.10000000000000001


This might fluctuate in different system floating-point implementations. Perhaps np.isclose(density, 0.1)?

mrocklin · 2018-01-10T21:22:59Z

Or perhaps change 10 to 8 above so that the result is evenly divisible by two

…

On Wed, Jan 10, 2018 at 12:08 PM, Hameer Abbasi ***@***.***> wrote: ***@***.**** commented on this pull request. ------------------------------ In sparse/core.py <https://github.com/mrocklin/sparse/pull/69#discussion_r160755728>: > + ------- + float + The ratio of nonzero to all elements. + + See Also + -------- + COO.size : Number of elements. + COO.nnz : Number of nonzero elements. + + Examples + -------- + >>> x = np.zeros((10, 10)) + >>> x[0, :] = 1 + >>> s = COO.from_numpy(x) + >>> s.density + 0.10000000000000001 This might fluctuate in different system floating-point implementations. Perhaps np.isclose(density, 0.1)? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/mrocklin/sparse/pull/69#pullrequestreview-87928365>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszEWhBJgW3Nr0LVCaGKs4YQy3xdscks5tJPyjgaJpZM4RZBRy> .

nils-werner · 2018-01-11T08:20:03Z

@mrocklin that's clever! Done.

hameerabbasi · 2018-01-11T10:27:54Z

This looks ready. If you merge, I can do a final review and merge.

nils-werner · 2018-01-12T09:06:46Z

Did you see my code comment?

hameerabbasi · 2018-01-12T10:52:18Z

Hmmm, that's strange, it isn't showing any of your comments in that link, or on any commits.

nils-werner · 2018-01-10T16:01:07Z

sparse/core.py

@@ -1277,7 +1325,7 @@ def astype(self, dtype, out=None):

    def maybe_densify(self, allowed_nnz=1e3, allowed_fraction=0.25):
        """ Convert to a dense numpy array if not too costly.  Err othrewise """
-        if reduce(operator.mul, self.shape) <= allowed_nnz or self.nnz >= np.prod(self.shape) * allowed_fraction:
+        if self.size <= allowed_nnz or self.density >= allowed_fraction:


It is easy to see that

self.nnz >= np.prod(self.shape) * allowed_fraction

can be transformed to

self.density >= allowed_fraction

But does self.density >= allowed_fraction actually make sense? Shouldn't it be the other way round?

It actually does make sense. If the density is low, then the cost of a dense array is higher compared to that of a sparse array, and we should avoid densifying.

If, however, density is high, the overhead of a sparse matrix far outweighs the benefits and we should densify.

To put it another way, a large array with a very low density should NOT be densified but it's okay to densify a high-density or a small array.

True. Maybe I got confused that one allowed means max and the other one means min... What about renaming them to

maybe_densify(self, max_nnz=1e3, min_density=0.25)

I would make sure that no tests break either here or in Dask before doing that. We can change unit tests here... But it's best to send a PR to Dask if any tests break there.

Let's change it anyway and I'll send a PR to Dask if anything fails for them.

Actually, shouldn't max_nnz be actually named max_size? We are not comparing it to .nnz anyways...

Yes, max_size is a good idea as well.

nils-werner · 2018-01-12T14:13:29Z

Aha! after commenting on lines you have to also open the code review thingy.

hameerabbasi · 2018-01-15T10:38:53Z

Is NotImplementedError the best way to show that densfication is forbidden? What would be the right exception here?

nils-werner · 2018-01-15T11:41:41Z

I would say ValueError is the right one, as it's the choice of function arguments that make the operation fail.

A NotImplementedError sounds more like "whatever you pass to this function, in this version it will never succeed".

hameerabbasi · 2018-01-15T11:55:01Z

Best to switch that to a ValueError then.

hameerabbasi · 2018-01-15T12:45:41Z

sparse/core.py

        an error.

        >>> x = np.zeros((5, 5), dtype=np.uint8)
        >>> x[2, 2] = 1
        >>> s = COO.from_numpy(x)
-        >>> s.maybe_densify(allowed_nnz=5, allowed_fraction=0.25)
+        >>> s.maybe_densify(max_size=5, min_fraction=0.25)
        Traceback (most recent call last):
            ...
        NotImplementedError: Operation would require converting large sparse array to dense


You need to update this line in the docstring. See here.

hameerabbasi · 2018-01-15T12:46:22Z

sparse/core.py

@@ -2381,17 +2381,17 @@ def astype(self, dtype, out=None):
        assert out is None
        return self.elemwise(np.ndarray.astype, dtype)

-    def maybe_densify(self, allowed_nnz=1000, allowed_fraction=0.25):
+    def maybe_densify(self, max_size=1000, min_fraction=0.25):


Best keep this as min_density, it's clearer that it represents the density.

hameerabbasi · 2018-01-15T13:08:26Z

sparse/core.py

        an error.

        >>> x = np.zeros((5, 5), dtype=np.uint8)
        >>> x[2, 2] = 1
        >>> s = COO.from_numpy(x)
-        >>> s.maybe_densify(allowed_nnz=5, allowed_fraction=0.25)
+        >>> s.maybe_densify(max_size=5, min_density=0.25)
        Traceback (most recent call last):
            ...
        NotImplementedError: Operation would require converting large sparse array to dense


Line 2425...

NotImplementedError: Operation would require converting large sparse array to dense

Need to change this to ValueError.

It's causing doctests to fail. :)

Currently there are a million tests failing on master, it is hard to spot my own mistakes...

That's strange, Travis shows everything is working on master.

I checked, they only fail until Numpy 1.13.1. Latest Numpy works fine...

That is really weird, I tested with Numpy 1.13.3 on my local system and it still works fine.

mrocklin · 2018-01-15T14:15:58Z

Is NotImplementedError the best way to show that densfication is forbidden? What would be the right exception here?

To me NotImplementedError means "We have not yet implemented this, but it is a reasonable request". If it is a case of a type not being supported like "we don't think that densifying scipy.sparse matrices is a good idea" then I would raise a TypeError. If it is a case of the values being inappropriate like "sometimes we do densify sparse arrays, but this particular array is very very sparse" then I would use a ValueError.

hameerabbasi · 2018-01-15T14:42:53Z

So, we all agree that it should be ValueError in maybe_densify then. :-)

hameerabbasi · 2018-01-15T18:24:43Z

Are you still making changes here? If not, this gets a +1 from me.

nils-werner · 2018-01-16T09:58:30Z

No, I'm done.

hameerabbasi · 2018-01-16T10:04:16Z

A rebase would be nice.

* Size and density properties * Docstrings and tests * Test .density using np.isclose * Fixed typos that made tests fail * Replaced other occurences of np.prod(self.shape) with self.size * Avoid float precision errors in .denstity docstring * maybe_densify raises ValueError instead of NotImplementedError * Updated docstring to match ValueError exception * changed allowed_nnz and allowed_fraction to max_size and min_density * changed min_fraction to min_density * Another docstring that didn't show ValueError * Documentation pages

hameerabbasi reviewed Jan 10, 2018

View reviewed changes

nils-werner commented Jan 12, 2018

View reviewed changes

nils-werner force-pushed the density branch from 1fa7541 to 832b5ab Compare January 15, 2018 12:18

hameerabbasi reviewed Jan 15, 2018

View reviewed changes

nils-werner mentioned this pull request Jan 15, 2018

Tox for cross Python-version testing #77

Merged

hameerabbasi added this to the 0.3 milestone Jan 15, 2018

nils-werner added 2 commits January 16, 2018 11:04

Size and density properties

b16748b

Docstrings and tests

0e3c48d

nils-werner added 2 commits January 16, 2018 11:04

Test .density using np.isclose

6d32034

Fixed typos that made tests fail

2be3a3b

hameerabbasi added the type:enhancement label Jan 16, 2018

nils-werner added 8 commits January 16, 2018 11:08

Replaced other occurences of np.prod(self.shape) with self.size

1fb947e

Avoid float precision errors in .denstity docstring

9ead182

maybe_densify raises ValueError instead of NotImplementedError

9f2012b

Updated docstring to match ValueError exception

06e8182

changed allowed_nnz and allowed_fraction to max_size and min_density

bc7613d

changed min_fraction to min_density

d297bc0

Another docstring that didn't show ValueError

f3806a4

Documentation pages

a65e1d0

nils-werner force-pushed the density branch from 647fc33 to a65e1d0 Compare January 16, 2018 10:09

hameerabbasi merged commit bbb0869 into pydata:master Jan 16, 2018

Size and density properties #69

Size and density properties #69

nils-werner commented Jan 10, 2018 •

edited

hameerabbasi commented Jan 10, 2018

nils-werner commented Jan 10, 2018

hameerabbasi commented Jan 10, 2018

nils-werner commented Jan 10, 2018

hameerabbasi commented Jan 10, 2018

hameerabbasi Jan 10, 2018

hameerabbasi Jan 10, 2018

mrocklin commented Jan 10, 2018 via email

nils-werner commented Jan 11, 2018

hameerabbasi commented Jan 11, 2018 •

edited

nils-werner commented Jan 12, 2018

hameerabbasi commented Jan 12, 2018

nils-werner Jan 10, 2018

hameerabbasi Jan 12, 2018

hameerabbasi Jan 12, 2018

nils-werner Jan 12, 2018

hameerabbasi Jan 12, 2018

hameerabbasi Jan 15, 2018

nils-werner Jan 15, 2018

hameerabbasi Jan 15, 2018

nils-werner commented Jan 12, 2018

hameerabbasi commented Jan 15, 2018

nils-werner commented Jan 15, 2018 •

edited

hameerabbasi commented Jan 15, 2018

hameerabbasi Jan 15, 2018

hameerabbasi Jan 15, 2018

hameerabbasi Jan 15, 2018

hameerabbasi Jan 15, 2018

nils-werner Jan 15, 2018

hameerabbasi Jan 15, 2018

nils-werner Jan 15, 2018 •

edited

hameerabbasi Jan 15, 2018

mrocklin commented Jan 15, 2018

hameerabbasi commented Jan 15, 2018

hameerabbasi commented Jan 15, 2018

nils-werner commented Jan 16, 2018

hameerabbasi commented Jan 16, 2018

Size and density properties #69

Size and density properties #69

Conversation

nils-werner commented Jan 10, 2018 • edited

hameerabbasi commented Jan 10, 2018

nils-werner commented Jan 10, 2018

hameerabbasi commented Jan 10, 2018

nils-werner commented Jan 10, 2018

hameerabbasi commented Jan 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrocklin commented Jan 10, 2018 via email

nils-werner commented Jan 11, 2018

hameerabbasi commented Jan 11, 2018 • edited

nils-werner commented Jan 12, 2018

hameerabbasi commented Jan 12, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nils-werner commented Jan 12, 2018

hameerabbasi commented Jan 15, 2018

nils-werner commented Jan 15, 2018 • edited

hameerabbasi commented Jan 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nils-werner Jan 15, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrocklin commented Jan 15, 2018

hameerabbasi commented Jan 15, 2018

hameerabbasi commented Jan 15, 2018

nils-werner commented Jan 16, 2018

hameerabbasi commented Jan 16, 2018

nils-werner commented Jan 10, 2018 •

edited

hameerabbasi commented Jan 11, 2018 •

edited

nils-werner commented Jan 15, 2018 •

edited

nils-werner Jan 15, 2018 •

edited