/ sparse Public

# Problems converting dask chunked sparse arrays greater size than 256x256#32

Closed
opened this issue Sep 1, 2017 · 3 comments
Closed

# Problems converting dask chunked sparse arrays greater size than 256x256 #32

opened this issue Sep 1, 2017 · 3 comments
Labels

### ljdursi commented Sep 1, 2017 • edited

 Presumably related to dask/dask#2586 : Computing a sparse array output from a chunked dask sparse array fails if the resulting sparse array is larger than 256x256 - coord type is uint16, but it looks like maybe intermediate calculations overflow: ```import dask.array as da import sparse def print_sparse_nonidentity(m, name): valid = True for i, j, v in zip(m.coords[0], m.coords[1], m.data): if not i == j: print name, i, j, v valid = False if valid: print "OK" def identity_da(size, chunksize): p = {(i, i): 1. for i in range(size)} # single chunk a = da.from_array(sparse.COO(p), chunks=(chunksize, chunksize), asarray=False) return a print "Should be identity matrix" size = 256 print "size = ", size a = identity_da(256, size//2) c = a.compute() print c print_sparse_nonidentity(c, "256 chunked") size = 258 print "size = ", size, " unchunked" a = identity_da(size, size) c = a.compute() print c print_sparse_nonidentity(c, "258 unchunked") print "size = ", size, " chunked" a = identity_da(size, size//2) c = a.compute() print c print_sparse_nonidentity(c, "258 chunked")``` Running this works for 256x256 matrix chunked, or 258x258 with only one chunk, but 258x258 broken into 4 chunks fails: ``````\$ python fail-conversion.py Should be identity matrix size = 256 OK size = 258 unchunked OK size = 258 chunked 258 chunked 256 0 1.0 258 chunked 257 1 1.0 `````` Versions are as follow: ``````\$ python Python 2.7.13 (default, Dec 18 2016, 07:03:39) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import dask >>> print dask.__version__ 0.15.2 >>> import sparse >>> print sparse.__version__ 0.1.1 `````` The text was updated successfully, but these errors were encountered:

### hameerabbasi commented Dec 24, 2017

 Since dask/dask#2586 was closed, it should be okay to close this as well?

### ljdursi commented Dec 27, 2017 • edited

 As far as I know, it remains a problem - I referenced dask/dask#2586 only because that was a similar problem.

### hameerabbasi commented Dec 31, 2017

 Fixed with #51.

mentioned this issue Mar 5, 2020