Dtype problem while accessing the values of a COO tensor with sparse 0.12.0

[jodemaey]

Describe the bug
I have got a problem while accessing the value of a COO tensor. This was working with sparse 0.11.2 but fails with sparse 0.12.0 . Problem seems to be related to dtype classes.

To Reproduce
I don't have the time to make a reproducible code but I can tell you that it appeared while using the qgs model : https://github.com/Climdyn/qgs .

The error I get is :

TypeError: expected dtype object, got 'numpy.dtype[int64]'
IPython CPU timings (estimated):
User : 1.68 s.
The above exception was the direct cause of the following exception:
System : 0.03 s.
Wall time: 1.70 s.
Traceback (most recent call last):
File ".../anaconda3/envs/qgs/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 2761, in safe_execfile
py3compat.execfile(
File ".../anaconda3/envs/qgs/lib/python3.8/site-packages/IPython/utils/py3compat.py", line 168, in execfile
exec(compiler(f.read(), fname, 'exec'), glob, loc)
File ".../qgs/qgs/diagnostics/streamfunctions.py", line 229, in
f, Df = create_tendencies(pars)
File ".../qgs/qgs/functions/tendencies.py", line 84, in create_tendencies
agotensor = QgsTensor(params, aip, oip, gip)
File ".../qgs/qgs/tensors/qgtensor.py", line 59, in init
self.compute_tensor()
File ".../qgs/qgs/tensors/qgtensor.py", line 179, in compute_tensor
a_inv[i, j] = aips.a(i, j)
File ".../qgs/qgs/inner_products/analytic.py", line 229, in a
return self._a[i, j]
File ".../anaconda3/envs/qgs/lib/python3.8/site-packages/sparse/_coo/indexing.py", line 77, in getitem
mask, adv_idx = _mask(x.coords, index, x.shape)
File ".../anaconda3/envs/qgs/lib/python3.8/site-packages/sparse/_coo/indexing.py", line 180, in _mask
mask, is_slice = _compute_mask(coords, _ind_ar_from_indices(indices))
SystemError: CPUDispatcher(<function _compute_mask at 0x7f1cac5dad30>) returned a result with an error set

The code that really fails is the line return self._a[i, j] which is supposed to return the tensor component.

System
Linux Ubuntu and conda. Here is my conda env (sorry I dump everything):

Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_llvm conda-forge
alabaster 0.7.12 py_0 conda-forge
attrs 19.3.0 py_0 conda-forge
babel 2.8.0 py_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
bleach 3.1.5 pyh9f0ad1d_0 conda-forge
brotlipy 0.7.0 py38h1e0a361_1000 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
ca-certificates 2021.5.30 ha878542_0 conda-forge
certifi 2021.5.30 py38h578d9bd_0 conda-forge
cffi 1.14.0 py38hd463f26_0 conda-forge
chardet 3.0.4 py38h32f6830_1006 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
cryptography 2.9.2 py38h766eaa4_0 conda-forge
cycler 0.10.0 py_2 conda-forge
dbus 1.13.6 he372182_0 conda-forge
decorator 4.4.2 py_0 conda-forge
defusedxml 0.6.0 py_0 conda-forge
diffeqpy 1.2.0 pypi_0 pypi
docutils 0.16 py38h32f6830_1 conda-forge
entrypoints 0.3 py38h32f6830_1001 conda-forge
expat 2.2.9 he1b5a44_2 conda-forge
fastcache 1.1.0 py38h7b6447c_0
ffmpeg 4.2 h167e202_0 conda-forge
fontconfig 2.13.1 h86ecdb6_1001 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
future 0.18.2 pypi_0 pypi
gettext 0.19.8.1 hc5be6a0_1002 conda-forge
glib 2.65.0 h6f030ca_0 conda-forge
gmp 6.1.2 h6c8ec71_1
gmpy2 2.0.8 py38hd5f6e3b_3
gnutls 3.6.13 h85f3911_1 conda-forge
gst-plugins-base 1.14.5 h0935bb2_2 conda-forge
gstreamer 1.14.5 h36ae1b5_2 conda-forge
icu 64.2 he1b5a44_1 conda-forge
idna 2.10 pyh9f0ad1d_0 conda-forge
imagesize 1.2.0 py_0 conda-forge
importlib-metadata 1.7.0 py38h32f6830_0 conda-forge
importlib_metadata 1.7.0 0 conda-forge
iniconfig 1.1.1 py_0
ipykernel 5.3.2 py38h23f93f0_0 conda-forge
ipython 7.25.0 py38hd0cf306_1 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
ipywidgets 7.6.3 pyhd3deb0d_0 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.17.1 py38h32f6830_0 conda-forge
jinja2 2.11.2 pyh9f0ad1d_0 conda-forge
jpeg 9d h516909a_0 conda-forge
jsonschema 3.2.0 py38h32f6830_1 conda-forge
julia 0.5.3 pypi_0 pypi
jupyter 1.0.0 py38h578d9bd_6 conda-forge
jupyter_client 6.1.5 py_0 conda-forge
jupyter_console 6.1.0 py_1 conda-forge
jupyter_core 4.6.3 py38h32f6830_1 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
kiwisolver 1.2.0 py38hbf85e49_0 conda-forge
lame 3.100 h7f98852_1001 conda-forge
latexcodec 2.0.1 pypi_0 pypi
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.34 h53a641e_5 conda-forge
lerc 2.2.1 h9c3ff4c_0 conda-forge
libblas 3.8.0 16_openblas conda-forge
libcblas 3.8.0 16_openblas conda-forge
libclang 9.0.1 default_hde54327_0 conda-forge
libdeflate 1.7 h7f98852_5 conda-forge
libffi 3.2.1 he1b5a44_1007 conda-forge
libgcc-ng 9.3.0 h2828fa1_19 conda-forge
libgfortran-ng 7.5.0 hdf63c60_6 conda-forge
libiconv 1.15 h516909a_1006 conda-forge
liblapack 3.8.0 16_openblas conda-forge
libllvm9 9.0.1 he513fc3_1 conda-forge
libopenblas 0.3.9 h5ec1e0e_0 conda-forge
libpng 1.6.37 hed695b0_1 conda-forge
libsodium 1.0.17 h516909a_0 conda-forge
libstdcxx-ng 9.3.0 h6de172a_19 conda-forge
libtiff 4.3.0 hf544144_1 conda-forge
libuuid 2.32.1 h14c3975_1000 conda-forge
libwebp-base 1.2.0 h7f98852_2 conda-forge
libxcb 1.13 h14c3975_1002 conda-forge
libxkbcommon 0.10.0 he1b5a44_0 conda-forge
libxml2 2.9.10 hee79883_0 conda-forge
llvm-openmp 9.0.1 hc9558a2_2 conda-forge
llvm-tools 9.0.1 he513fc3_1 conda-forge
llvmdev 9.0.1 he513fc3_1 conda-forge
llvmlite 0.33.0 py38h4f45e52_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_0 conda-forge
markupsafe 1.1.1 py38h1e0a361_1 conda-forge
matplotlib 3.4.2 py38h578d9bd_0 conda-forge
matplotlib-base 3.4.2 py38hcc49a3a_0 conda-forge
matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge
mistune 0.8.4 py38h1e0a361_1001 conda-forge
more-itertools 8.6.0 pyhd3eb1b0_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpmath 1.1.0 py38_0
nbconvert 5.6.1 py38h32f6830_1 conda-forge
nbformat 5.0.7 py_0 conda-forge
ncurses 6.1 hf484d3e_1002 conda-forge
nettle 3.6 he412f7d_0 conda-forge
notebook 6.0.3 py38h32f6830_1 conda-forge
nspr 4.26 he1b5a44_0 conda-forge
nss 3.47 he751ad9_0 conda-forge
numba 0.50.1 py38hcb8c335_1 conda-forge
numpy 1.21.0 py38h9894fe3_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openh264 1.8.0 hdbcaa40_1000 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openssl 1.1.1k h7f98852_0 conda-forge
oset 0.1.3 pypi_0 pypi
packaging 20.4 pyh9f0ad1d_0 conda-forge
pandoc 2.10 0 conda-forge
pandocfilters 1.4.2 py_1 conda-forge
parso 0.7.0 pyh9f0ad1d_0 conda-forge
pcre 8.44 he1b5a44_0 conda-forge
pebble 4.6.1 pyhd8ed1ab_0 conda-forge
pexpect 4.8.0 py38h32f6830_1 conda-forge
pickleshare 0.7.5 py38h32f6830_1001 conda-forge
pillow 8.2.0 py38ha0e1e83_1 conda-forge
pip 21.1.3 pyhd8ed1ab_0 conda-forge
pluggy 0.13.1 py38_0
prometheus_client 0.8.0 pyh9f0ad1d_0 conda-forge
prompt-toolkit 3.0.5 py_1 conda-forge
prompt_toolkit 3.0.5 1 conda-forge
pthread-stubs 0.4 h14c3975_1001 conda-forge
ptyprocess 0.6.0 py_1001 conda-forge
py 1.9.0 py_0
pybtex 0.22.2 pypi_0 pypi
pybtex-docutils 0.2.2 pypi_0 pypi
pycparser 2.20 pyh9f0ad1d_2 conda-forge
pydmd 0.3 pypi_0 pypi
pygments 2.6.1 py_0 conda-forge
pyopenssl 19.1.0 py_1 conda-forge
pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge
pyqt 5.12.3 py38ha8c2ead_3 conda-forge
pyqt5-sip 4.19.18 pypi_0 pypi
pyqtchart 5.12 pypi_0 pypi
pyqtwebengine 5.12.1 pypi_0 pypi
pyrsistent 0.16.0 py38h1e0a361_0 conda-forge
pysocks 1.7.1 py38h32f6830_1 conda-forge
pytest 6.2.4 py38h578d9bd_0 conda-forge
python 3.8.5 h4d41432_2_cpython conda-forge
python-dateutil 2.8.1 py_0 conda-forge
python_abi 3.8 1_cp38 conda-forge
pytz 2020.1 pyh9f0ad1d_0 conda-forge
pyyaml 5.3.1 pypi_0 pypi
pyzmq 19.0.1 py38ha71036d_0 conda-forge
qt 5.12.5 hd8c4c69_1 conda-forge
qtconsole 4.7.5 pyh9f0ad1d_0 conda-forge
qtpy 1.9.0 py_0 conda-forge
readline 8.0 hf8c457e_0 conda-forge
requests 2.24.0 pyh9f0ad1d_0 conda-forge
scipy 1.5.2 py38h8c5af15_0 conda-forge
send2trash 1.5.0 py_0 conda-forge
setuptools 49.1.0 py38h32f6830_0 conda-forge
six 1.15.0 pyh9f0ad1d_0 conda-forge
snowballstemmer 2.0.0 py_0 conda-forge
sparse 0.12.0 pyhd8ed1ab_0 conda-forge
sphinx 4.0.2 pyh6c4a22f_1 conda-forge
sphinx_rtd_theme 0.5.2 pyhd8ed1ab_1 conda-forge
sphinxcontrib-applehelp 1.0.2 py_0 conda-forge
sphinxcontrib-bibtex 1.0.0 pypi_0 pypi
sphinxcontrib-devhelp 1.0.2 py_0 conda-forge
sphinxcontrib-htmlhelp 1.0.3 py_0 conda-forge
sphinxcontrib-jsmath 1.0.1 py_0 conda-forge
sphinxcontrib-qthelp 1.0.3 py_0 conda-forge
sphinxcontrib-serializinghtml 1.1.4 py_0 conda-forge
sqlite 3.32.3 hcee41ef_0 conda-forge
sympy 1.8 py38h578d9bd_0 conda-forge
terminado 0.8.3 py38h32f6830_1 conda-forge
testpath 0.4.4 py_0 conda-forge
tk 8.6.10 hed695b0_0 conda-forge
toml 0.10.1 py_0
tornado 6.0.4 py38h1e0a361_1 conda-forge
traitlets 4.3.3 py38h32f6830_1 conda-forge
urllib3 1.25.9 py_0 conda-forge
wcwidth 0.2.5 pyh9f0ad1d_0 conda-forge
webencodings 0.5.1 py_1 conda-forge
wheel 0.34.2 py_1 conda-forge
widgetsnbextension 3.5.1 py38h32f6830_1 conda-forge
x264 1!152.20180806 h14c3975_0 conda-forge
xorg-libxau 1.0.9 h14c3975_0 conda-forge
xorg-libxdmcp 1.1.3 h516909a_0 conda-forge
xz 5.2.5 h516909a_0 conda-forge
zeromq 4.3.2 he1b5a44_2 conda-forge
zipp 3.1.0 py_0 conda-forge
zlib 1.2.11 h516909a_1006 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge

Additional context
Because I lack time right now, my solution is to downgrade to 0.11.2 .
I'm really sorry no to have the time to produce a better report, but I thought it deserved to be reported anyway.

[jodemaey]

Update

I have tried to solve the problem to be able to continue working. Funny thing is that I had to downgrade numpy and not sparse. I have downgraded numpy to 1.19.4 (the last version I knew it was working). So something happened in between numpy 1.19.4 and 1.21.0 that sparse is not ok with.

Could it be related to the comment #490 (comment) ?

[hameerabbasi]

This sounds like a Numba error, cc @stuartarchibald

[hameerabbasi]

Also possibly NumPy dtypes: @seberg.

[seberg]

A NumPy bug/change is possible, but I am not sure where to start looking, even a non-minimal reproducer may help. It would be nice to have a bit more to go on, if just which error is in-process here but not returned.

[jodemaey]

I will try to write a reproducer when I get some time next week.

[seberg]

Ah sorry, didn't notice this TypeError: expected dtype object, got 'numpy.dtype[int64]' somehow :(.

There was an issue that I had to change type(dtype) (subclass). Which is straight forward, but NumPy also had a broken macro that we should, but can't fix in old versions.
That just means: It may be that you have to update numba. Normally it should be OK to have a dependency (like numba) a bit older than NumPy, but this is a likely exception.

[hameerabbasi]

@jodemaey

numba 0.50.1

Yes, please update Numba, that should fix the issue, I believe.

[jodemaey]

Ok I will try this asap and let you know. Thanks for your super fast support !

[stuartarchibald]

Is it maybe this problem numba/numba#6078 ? I think a Numba update would be the first thing to try.

[jodemaey]

Since updating from Numba 0.50.1 to 0.54.0rc1 the problem is indeed no longer there.
Numba had not been updated in my environment while updating NumPy to 1.21 , I don't know why...

The problem is solved thus, sorry for this. Thanks for your support.