Initial implementation for arbitrary fill values. #165

hameerabbasi · Jul 15, 2018

Closes #143

Initial implementation for arbitrary fill values.

codecov · Jul 15, 2018

Codecov Report

Merging #165 into master will increase coverage by 0.15%.
The diff coverage is 98.97%.

@@            Coverage Diff             @@
##           master     #165      +/-   ##
==========================================
+ Coverage   96.96%   97.12%   +0.15%     
==========================================
  Files          11       11              
  Lines        1219     1252      +33     
==========================================
+ Hits         1182     1216      +34     
+ Misses         37       36       -1

Impacted Files	Coverage Δ
sparse/io.py	`100% <ø> (ø)`	⬆️
sparse/coo/core.py	`96.63% <100%> (+0.31%)`	⬆️
sparse/coo/umath.py	`96.82% <100%> (-0.02%)`	⬇️
sparse/utils.py	`98.4% <100%> (+1.28%)`	⬆️
sparse/dok.py	`95.57% <100%> (ø)`	⬆️
sparse/sparse_array.py	`93.1% <100%> (+1.1%)`	⬆️
sparse/coo/common.py	`97.29% <100%> (+0.03%)`	⬆️
sparse/coo/indexing.py	`98.9% <87.5%> (-1.1%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e2906c0...3faf622. Read the comment docs.

shoyer · Jul 15, 2018

shoyer reviewed Jul 15, 2018

View changes

I have a few minor comments, but I'm excited about this! The implementation looks quite clean.

This will let us define functions all the nan-aggregations (e.g., nanmedian()) for use in xarray. These functions can simply require a fill value of NaN.

shoyer · Jul 15, 2018

sparse/coo/common.py

@@ -56,6 +56,7 @@ def linear_loc(coords, shape):
    return out


+@check_fill_value(2)


I find it more readable to use keyword argument with literal values, e..g, @check_zero_fill_value(nargs=2)

shoyer · Jul 15, 2018

sparse/utils.py

+
+
+def equivalent(x, y):
+    return (x == y) | ((x != x) & (y != y))


note: if you're calling this on arrays, you could skip the non-NA checks for dtypes that can't hold NaN/NaT.

shoyer · Jul 15, 2018

sparse/utils.py

+    def generator(func):
+        @functools.wraps(func)
+        def wrapped(*args, **kwargs):
+            for arg in args[:nargs]:


Watch out: This has one of the same issues we encountered in NEP 18: it means the public API for these functions will be changed, to only accepted positional arguments.

I would suggest writing helper functions to call inside func inside, e.g., check_zero_fill_value(a, b).

shoyer · Jul 15, 2018

sparse/utils.py

+        @functools.wraps(func)
+        def wrapped(*args, **kwargs):
+            for arg in args[:nargs]:
+                if hasattr(arg, 'fill_value') and \


Nit: per PEP 8, prefer using extra parentheses () to explicit continuation with \.

shoyer · Jul 15, 2018

sparse/utils.py

+    from .sparse_array import SparseArray
+
+    @functools.wraps(func)
+    def wrapped(arrays, *args, **kwargs):


As above, consider using a function instead of a decorator here.

In this case, the decorator would work OK (as long as the first argument is always called arrays) but decorators are more magical than simple function calls.

shoyer · Jul 16, 2018

shoyer reviewed Jul 16, 2018

View changes

shoyer · Jul 16, 2018

docs/quickstart.rst

-However, operations which convert the sparse array into a dense one will raise exceptions
-For example, the following raises a :obj:`ValueError`.
+However, operations which convert the sparse array into a dense one will usually change the fill
+value instead of raising an error.


The new sentence here is a little confusing to me. These operations how change the fill value instead of converting a sparse array to a dense array, so they don't "convert the sparse array into a dense one" at all now.

Changed the sentence structure a bit.

shoyer · Jul 16, 2018

docs/operations.rst

@@ -265,7 +216,6 @@ All of the following will raise an :obj:`IndexError`, like in Numpy 1.13 and lat
   z[3, 6]
   z[1, 4, 8]
   z[-6]
-   z[[True, True, False, True], 3, 4]

 .. note:: Numpy advanced indexing is currently not supported.



below here: maybe note that stack and concatenate require matching fill values, and that some operations (e.g., tensordot) require a fill value of zero?

hameerabbasi · Jul 18, 2018

@shoyer I think this is now ready to merge.
@mrocklin If you have time, if it's going to take too long, let me know.
@ahwillia If you're willing to review. Reviewing is great for learning the codebase, if you're interested.

shoyer · Jul 18, 2018

shoyer reviewed Jul 18, 2018

View changes

Just another minor comment. I agree that this looks good to merge!

shoyer · Jul 18, 2018

sparse/utils.py

+    Traceback (most recent call last):
+        ...
+    ValueError: This operation requires zero fill values.
+    """
    for arg in args:
        if (hasattr(arg, 'fill_value') and
                not equivalent(arg.fill_value, _zero_of_dtype(arg.dtype))):
            raise ValueError('This operation requires zero fill values.')


It's a minor point, but it's nice to include offending values in all error messages, e.g., arg.fill_value in this case

I'm not sure what best practice would be here. We wouldn't know what exact argument would produce this fill value, so showing it may be not be useful. What do you think?

Maybe:

'This operation requires zero fill values, but argument {} has fill value {}'.format(i, arg.fill_value)

where i comes from iterating over args with enumerate().

shoyer · Jul 18, 2018

sparse/utils.py

@@ -276,4 +345,4 @@ def check_consistent_fill_value(arrays):
    fv = arrays[0].fill_value

    if not all(equivalent(fv, s.fill_value) for s in arrays):
-        raise ValueError('Consistent fill-values required.')
+        raise ValueError('This operation requires consistent fill-values.')


same concern as above about including fill values in the error message

mrocklin · Jul 18, 2018

@mrocklin If you have time, if it's going to take too long, let me know.

I appreciate getting pinged, but I probably won't be reviewing this one. That's also fine. I probably shouldn't be active on every pull request here. It looks like @shoyer seems pretty happy, which is a good sign. I trust his attention to detail :)

hameerabbasi · Jul 18, 2018

Well I believe there should be at least one reviewer. I'm thinking of ways to rope new contributors in... Reviewers or code-wise. 😄

Initial implementation for arbitrary fill values.

Loading status checks…

9fc23cd

hameerabbasi requested review from mrocklin and shoyer Jul 15, 2018

hameerabbasi added some commits Jul 15, 2018

Small fixes.

Loading status checks…

a777753

A bit of refactoring.

Loading status checks…

432dcc7

hameerabbasi added some commits Jul 15, 2018

Documentation.

899f0aa

Address commits by @shoyer.

Loading status checks…

ba95b12

Rename method to be more clear.

60d5dbf

More doc updates.

Loading status checks…

699b005

hameerabbasi added some commits Jul 18, 2018

Add docstrings with examples.

7eb7539

More docs!

Loading status checks…

654053c

hameerabbasi added some commits Jul 18, 2018

Make error messages more descriptive.

Loading status checks…

b4359fe

Edit docs.

Loading status checks…

b0b35dd

Rewrite notes as raises.

Loading status checks…

3faf622

hameerabbasi deleted the hameerabbasi:fill-values branch Jul 18, 2018

hameerabbasi referenced this pull request Jul 24, 2018
Closed
Implementing sparse + scalar #9070

pydata/sparse

Initial implementation for arbitrary fill values. #165

hameerabbasi commented Jul 15, 2018

hameerabbasi requested review from mrocklin and shoyer Jul 15, 2018

This comment has been minimized.

codecov bot commented Jul 15, 2018 •

edited

hameerabbasi added some commits Jul 15, 2018

shoyer reviewed Jul 15, 2018

View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 15, 2018

shoyer reviewed Jul 16, 2018

View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 18, 2018

This comment has been minimized.

hameerabbasi commented Jul 18, 2018

shoyer reviewed Jul 18, 2018

View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 18, 2018

This comment has been minimized.

mrocklin commented Jul 18, 2018

This comment has been minimized.

hameerabbasi commented Jul 18, 2018

hameerabbasi merged commit `f767837` into pydata:master Jul 18, 2018
4 of 5 checks passed

4 of 5 checks passed

hameerabbasi deleted the hameerabbasi:fill-values branch Jul 18, 2018

hameerabbasi referenced this pull request Jul 24, 2018

Implementing sparse + scalar #9070

pydata/sparse

Join GitHub today

Initial implementation for arbitrary fill values. #165

Conversation

hameerabbasi commented Jul 15, 2018

hameerabbasi requested review from mrocklin and shoyer Jul 15, 2018

This comment has been minimized.

codecov bot commented Jul 15, 2018 • edited

Codecov Report

hameerabbasi added some commits Jul 15, 2018

shoyer reviewed Jul 15, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 15, 2018

shoyer reviewed Jul 16, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 18, 2018

This comment has been minimized.

hameerabbasi commented Jul 18, 2018

shoyer reviewed Jul 18, 2018 View changes

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

hameerabbasi added some commits Jul 18, 2018

This comment has been minimized.

mrocklin commented Jul 18, 2018

This comment has been minimized.

hameerabbasi commented Jul 18, 2018

Hide details View details hameerabbasi merged commit f767837 into pydata:master Jul 18, 2018 4 of 5 checks passed

4 of 5 checks passed

hameerabbasi deleted the hameerabbasi:fill-values branch Jul 18, 2018

hameerabbasi referenced this pull request Jul 24, 2018

codecov bot commented Jul 15, 2018 •

edited

shoyer reviewed Jul 15, 2018

View changes

shoyer reviewed Jul 16, 2018

View changes

shoyer reviewed Jul 18, 2018

View changes

hameerabbasi merged commit `f767837` into pydata:master Jul 18, 2018
4 of 5 checks passed