XArray and Sparse

[mvsantosdev]

I'm trying to use the xarray and sparse combined. for this simple example how I get the values of sxr?

import numpy as np
import xarray as xr
import sparse

x = np.random.random((100, 100, 100))
x[x < 0.9] = 0  # fill most of the array with zeros

s = sparse.COO(x)

sxr = xr.DataArray(s, dims=('A', 'B', 'C'))

I tried sxr.values and the follow error message raised:

RuntimeError: Cannot convert a sparse array to dense automatically. To manually densify, use the todense method.

Them I tried sxr.todense() and another error raises:

AttributeError: 'DataArray' object has no attribute 'todense'

How can I solve this?
Cheers

[andersy005]

You will need to use the .data in order to access the underlying sparse data, and then you can call .todense()

In [3]: sxr.data
Out[3]: <COO: shape=(100, 100, 100), dtype=float64, nnz=100175, fill_value=0.0>

In [4]: sxr.data.todense()
Out[4]: 
array([[[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.95706737],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.99498561, 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.90212799],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.93170908],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.93345325, 0.        ]],
...

i hope this helps...

[mvsantosdev]

It works! Thank you, Anderson!

[andersy005]

If you want to stay in the xarray land, you may want to use sxr.as_numpy() as @dcherian and @TomNicholas suggested below.

In [12]: sxr.as_numpy()
Out[12]: 
<xarray.DataArray (A: 100, B: 100, C: 100)>
array([[[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.91933338],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.95861375, 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.93380568,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.96754762, ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.90746512, ..., 0.        ,
         0.        , 0.        ],
...
        [0.        , 0.        , 0.        , ..., 0.95806523,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]],

       [[0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.94268924, 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        ...,
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.93208322, 0.        ],
        [0.        , 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ],
        [0.95098727, 0.        , 0.        , ..., 0.        ,
         0.        , 0.        ]]])
Dimensions without coordinates: A, B, C

In [13]: xr.__version__
Out[13]: '2022.3.0'

In [14]: sparse.__version__
Out[14]: '0.13.0'

[dcherian]

The as_numpy or to_numpy methods should work but I get the same error.
Part of the traceback

File ~/work/python/xarray/xarray/core/dataarray.py:727, in DataArray.as_numpy(self)
    716 """
    717 Coerces wrapped data and coordinates into numpy arrays, returning a DataArray.
    718 
   (...)
    724 DataArray.data
    725 """
    726 coords = {k: v.as_numpy() for k, v in self._coords.items()}
--> 727 return self._replace(self.variable.as_numpy(), coords, indexes=self._indexes)

File ~/work/python/xarray/xarray/core/variable.py:1149, in Variable.as_numpy(self)
   1147 def as_numpy(self: T_Variable) -> T_Variable:
   1148     """Coerces wrapped data into a numpy array, returning a Variable."""
-> 1149     return self._replace(data=self.to_numpy())

File ~/work/python/xarray/xarray/core/variable.py:1143, in Variable.to_numpy(self)
   1141 if isinstance(data, sparse_array_type):
   1142     data = data.todense()
-> 1143 data = np.asarray(data)
   1145 return data

Suggests that it should call todense automatically but the isinstance check is failing. cc @TomNicholas

[TomNicholas]

@dcherian I just tried this example on my machine and it worked - values printed and no error raised. This is with xarray main and sparse 0.13.0

[dcherian]

oops sorry! I installed sparse after importing xarray... Can confirm as_numpy works