Initial implementation for arbitrary fill values. #165

hameerabbasi · 2018-07-15T16:41:14Z

Closes #143

Initial implementation for arbitrary fill values.

codecov · 2018-07-15T16:42:37Z

Codecov Report

Merging #165 into master will increase coverage by 0.15%.
The diff coverage is 98.97%.

@@            Coverage Diff             @@
##           master     #165      +/-   ##
==========================================
+ Coverage   96.96%   97.12%   +0.15%     
==========================================
  Files          11       11              
  Lines        1219     1252      +33     
==========================================
+ Hits         1182     1216      +34     
+ Misses         37       36       -1

Impacted Files	Coverage Δ
sparse/io.py	`100% <ø> (ø)`	⬆️
sparse/coo/core.py	`96.63% <100%> (+0.31%)`	⬆️
sparse/coo/umath.py	`96.82% <100%> (-0.02%)`	⬇️
sparse/utils.py	`98.4% <100%> (+1.28%)`	⬆️
sparse/dok.py	`95.57% <100%> (ø)`	⬆️
sparse/sparse_array.py	`93.1% <100%> (+1.1%)`	⬆️
sparse/coo/common.py	`97.29% <100%> (+0.03%)`	⬆️
sparse/coo/indexing.py	`98.9% <87.5%> (-1.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2906c0...3faf622. Read the comment docs.

shoyer

I have a few minor comments, but I'm excited about this! The implementation looks quite clean.

This will let us define functions all the nan-aggregations (e.g., nanmedian()) for use in xarray. These functions can simply require a fill value of NaN.

shoyer · 2018-07-15T22:38:00Z

sparse/coo/common.py

@@ -56,6 +56,7 @@ def linear_loc(coords, shape):
    return out


+@check_fill_value(2)


I find it more readable to use keyword argument with literal values, e..g, @check_zero_fill_value(nargs=2)

shoyer · 2018-07-15T22:40:19Z

sparse/utils.py

+
+
+def equivalent(x, y):
+    return (x == y) | ((x != x) & (y != y))


note: if you're calling this on arrays, you could skip the non-NA checks for dtypes that can't hold NaN/NaT.

shoyer · 2018-07-15T22:41:46Z

sparse/utils.py

+    def generator(func):
+        @functools.wraps(func)
+        def wrapped(*args, **kwargs):
+            for arg in args[:nargs]:


Watch out: This has one of the same issues we encountered in NEP 18: it means the public API for these functions will be changed, to only accepted positional arguments.

I would suggest writing helper functions to call inside func inside, e.g., check_zero_fill_value(a, b).

shoyer · 2018-07-15T22:42:25Z

sparse/utils.py

+        @functools.wraps(func)
+        def wrapped(*args, **kwargs):
+            for arg in args[:nargs]:
+                if hasattr(arg, 'fill_value') and \


Nit: per PEP 8, prefer using extra parentheses () to explicit continuation with \.

shoyer · 2018-07-15T22:43:39Z

sparse/utils.py

+    from .sparse_array import SparseArray
+
+    @functools.wraps(func)
+    def wrapped(arrays, *args, **kwargs):


As above, consider using a function instead of a decorator here.

In this case, the decorator would work OK (as long as the first argument is always called arrays) but decorators are more magical than simple function calls.

shoyer · 2018-07-16T18:06:15Z

docs/quickstart.rst

-However, operations which convert the sparse array into a dense one will raise exceptions
-For example, the following raises a :obj:`ValueError`.
+However, operations which convert the sparse array into a dense one will usually change the fill
+value instead of raising an error.


The new sentence here is a little confusing to me. These operations how change the fill value instead of converting a sparse array to a dense array, so they don't "convert the sparse array into a dense one" at all now.

Changed the sentence structure a bit.

shoyer · 2018-07-16T18:10:39Z

docs/operations.rst

@@ -265,7 +216,6 @@ All of the following will raise an :obj:`IndexError`, like in Numpy 1.13 and lat
   z[3, 6]
   z[1, 4, 8]
   z[-6]
-   z[[True, True, False, True], 3, 4]

 .. note:: Numpy advanced indexing is currently not supported.



below here: maybe note that stack and concatenate require matching fill values, and that some operations (e.g., tensordot) require a fill value of zero?

hameerabbasi · 2018-07-18T04:53:11Z

@shoyer I think this is now ready to merge.
@mrocklin If you have time, if it's going to take too long, let me know.
@ahwillia If you're willing to review. Reviewing is great for learning the codebase, if you're interested.

shoyer

Just another minor comment. I agree that this looks good to merge!

shoyer · 2018-07-18T04:57:03Z

sparse/utils.py

+    Traceback (most recent call last):
+        ...
+    ValueError: This operation requires zero fill values.
+    """
    for arg in args:
        if (hasattr(arg, 'fill_value') and
                not equivalent(arg.fill_value, _zero_of_dtype(arg.dtype))):
            raise ValueError('This operation requires zero fill values.')


It's a minor point, but it's nice to include offending values in all error messages, e.g., arg.fill_value in this case

I'm not sure what best practice would be here. We wouldn't know what exact argument would produce this fill value, so showing it may be not be useful. What do you think?

Maybe:

'This operation requires zero fill values, but argument {} has fill value {}'.format(i, arg.fill_value)

where i comes from iterating over args with enumerate().

shoyer · 2018-07-18T04:57:18Z

sparse/utils.py

@@ -276,4 +345,4 @@ def check_consistent_fill_value(arrays):
    fv = arrays[0].fill_value

    if not all(equivalent(fv, s.fill_value) for s in arrays):
-        raise ValueError('Consistent fill-values required.')
+        raise ValueError('This operation requires consistent fill-values.')


same concern as above about including fill values in the error message

mrocklin · 2018-07-18T16:16:06Z

@mrocklin If you have time, if it's going to take too long, let me know.

I appreciate getting pinged, but I probably won't be reviewing this one. That's also fine. I probably shouldn't be active on every pull request here. It looks like @shoyer seems pretty happy, which is a good sign. I trust his attention to detail :)

hameerabbasi · 2018-07-18T16:17:49Z

Well I believe there should be at least one reviewer. I'm thinking of ways to rope new contributors in... Reviewers or code-wise. 😄

Initial implementation for arbitrary fill values.

9fc23cd

hameerabbasi requested review from mrocklin and shoyer July 15, 2018 16:41

hameerabbasi added 2 commits July 15, 2018 17:04

Small fixes.

a777753

A bit of refactoring.

432dcc7

shoyer reviewed Jul 15, 2018

View reviewed changes

hameerabbasi added 4 commits July 15, 2018 17:52

Documentation.

899f0aa

Address commits by @shoyer.

ba95b12

Rename method to be more clear.

60d5dbf

More doc updates.

699b005

shoyer reviewed Jul 16, 2018

View reviewed changes

hameerabbasi added 2 commits July 18, 2018 06:21

Add docstrings with examples.

7eb7539

More docs!

654053c

shoyer reviewed Jul 18, 2018

View reviewed changes

hameerabbasi added 3 commits July 18, 2018 07:23

Make error messages more descriptive.

b4359fe

Edit docs.

b0b35dd

Rewrite notes as raises.

3faf622

hameerabbasi merged commit f767837 into pydata:master Jul 18, 2018

hameerabbasi deleted the fill-values branch July 18, 2018 16:19

hameerabbasi mentioned this pull request Jul 24, 2018

Implementing sparse + scalar scipy/scipy#9070

Closed

Initial implementation for arbitrary fill values. #165

Initial implementation for arbitrary fill values. #165

hameerabbasi commented Jul 15, 2018

codecov bot commented Jul 15, 2018 •

edited

shoyer left a comment

shoyer Jul 15, 2018

shoyer Jul 15, 2018

shoyer Jul 15, 2018

shoyer Jul 15, 2018

shoyer Jul 15, 2018

shoyer Jul 16, 2018

hameerabbasi Jul 18, 2018

shoyer Jul 16, 2018

hameerabbasi Jul 18, 2018

hameerabbasi commented Jul 18, 2018

shoyer left a comment

shoyer Jul 18, 2018

hameerabbasi Jul 18, 2018

shoyer Jul 18, 2018

hameerabbasi Jul 18, 2018

shoyer Jul 18, 2018

mrocklin commented Jul 18, 2018

hameerabbasi commented Jul 18, 2018

		@@ -56,6 +56,7 @@ def linear_loc(coords, shape):
		return out


		@check_fill_value(2)



		def equivalent(x, y):
		return (x == y) \| ((x != x) & (y != y))

Initial implementation for arbitrary fill values. #165

Initial implementation for arbitrary fill values. #165

Conversation

hameerabbasi commented Jul 15, 2018

codecov bot commented Jul 15, 2018 • edited

Codecov Report

shoyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hameerabbasi commented Jul 18, 2018

shoyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mrocklin commented Jul 18, 2018

hameerabbasi commented Jul 18, 2018

codecov bot commented Jul 15, 2018 •

edited