Usage: confused by np.where behaviour and fill_value

[dcherian]

Please provide a description of what you'd like to do.

More dependable fill_value preservation in where

Example Code

import sparse
import numpy as np

array = sparse.COO.from_numpy(np.eye(3), fill_value=0)
assert np.where(array == 1, np.nan, array).fill_value == 0  # passes
assert np.where(array < 1, np.nan, array).fill_value == 0 # fails, result fill_value is np.nan

For mask=array==1, mask.fill_value=False and the array's fill_value is preserved in the np.where call.
For mask=array<1, mask.fill_value=True and the array's fill_value is NOT preserved in the np.where call.

This kind of behaviour is hard to rely on in a library. Is it a bug?

[dcherian]

OK I understand what's happening now. np.where is basically applied to mask.fill_value , and the .data are filled appropriately.

It would be good to call this behaviour out in the docstring

[hameerabbasi]

In general, out_fv = elemwise_fn(*in_fvs). So if array.fill_value = 0, and for a scalar, it is its own fill value.

(a.fill_value == 1) = False = (array == 1).fill_value
np.where(False, np.nan, a.fill_value) = np.nan = np.where((array == 1), np.nan, a).fill_value


(a.fill_value < 1) = True = (array < 1).fill_value
np.where(True np.nan, a.fill_value) = 0 = np.where((array < 1), np.nan, a).fill_value

The reasoning is the following: fill_values propagate through elemwise because we expect elemwise fns to produce mostly fill_values. If it was otherwise, we'd need to densify.

[dcherian]

fill_values propagate through elemwise because we expect elemwise fns to produce mostly fill_values. If it was otherwise, we'd need to densify.

Got it. thanks! Feel free to close if you don't think the docstring needs to be updated.

[prady0t]

If we still want to add this in the docstring, we can do so somewhere around here. We can also take a much simpler example to explain how fill_value is handled for element wise operations. Here's an example with + operation :

a = COO.from_numpy(np.eye(3), fill_value=1)
b = COO.from_numpy(np.eye(3), fill_value=2)

c = a + b
print(c.fill_value)  # 1.0 + 2.0 = 3.0

Mentioning "Since out_fv = elemwise_fn(*in_fvs) we only have to calculate fill_value once and hence sparse calculations saves memory".