scv.tl.recover_latent_time - resolving ValueError 'could not convert integer scalar'

[mhwadsworth91]

Hello Theis lab,

Thank you for all your efforts with this package; my colleagues and I have enjoyed using it in our research.

At your earliest convenience, I would appreciate your help with solving an issue I'm running into when trying to compute the directed PAGA.

I was able to compute this back in October, however, I recently updated my conda environment and now I get the following error when I run the scVelo command 'scv.tl.recover_latent_time()':

##############################################################################################

scv.tl.recover_latent_time(adata, root_key="initial_states_probs", end_key="terminal_states_probs")
computing latent time using initial_states_probs, terminal_states_probs as prior

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
/var/folders/hl/l4gsxvws4p314gngvjm46n780000gn/T/ipykernel_32730/4232468970.py in <module>
      1 # Directed PAGA
----> 2 scv.tl.recover_latent_time(adata, root_key="initial_states_probs", end_key="terminal_states_probs")
      3 #scv.tl.recover_latent_time(adata)

~/opt/anaconda3/lib/python3.8/site-packages/scvelo/tools/dynamical_model.py in latent_time(data, vkey, min_likelihood, min_confidence, min_corr_diffusion, weight_diffusion, root_key, end_key, t_max, copy)
    883     else:
    884         conn_new = conn.copy()
--> 885         conn_new[:, idx_low_confidence] = 0
    886         conn_new.eliminate_zeros()
    887         latent_time = conn_new.dot(latent_time)

~/opt/anaconda3/lib/python3.8/site-packages/scipy/sparse/_index.py in __setitem__(self, key, x)
    123                 return
    124             x = x.reshape(i.shape)
--> 125             self._set_arrayXarray(i, j, x)
    126 
    127     def _validate_indices(self, key):

~/opt/anaconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py in _set_arrayXarray(self, row, col, x)
    810     def _set_arrayXarray(self, row, col, x):
    811         i, j = self._swap((row, col))
--> 812         self._set_many(i, j, x)
    813 
    814     def _set_arrayXarray_sparse(self, row, col, x):

~/opt/anaconda3/lib/python3.8/site-packages/scipy/sparse/compressed.py in _set_many(self, i, j, x)
    897         n_samples = x.size
    898         offsets = np.empty(n_samples, dtype=self.indices.dtype)
--> 899         ret = csr_sample_offsets(M, N, self.indptr, self.indices, n_samples,
    900                                  i, j, offsets)
    901         if ret == 1:

ValueError: could not convert integer scalar

##############################################################################################

I've tried resolving this error by deploying a few suggestions from similar issues (e.g., adata = adata.copy, updating sciPy, etc.)., however, none of these options resolve my issue. I've also tried reverting back to my original computing environment, however, I still get the same error.

My Anndata object contains ~166K total cells and 3000 genes. I was able to get this command to work with the pancreas demo dataset provided, so I think my dataset size might be causing this issue?

Any suggestions/advice you might have would be greatly appreciated! Thank you in advance.

Below are the packages/versions I'm running in my environment:

##############################################################################################

Package Version

---

alabaster 0.7.12
anaconda-client 1.9.0
anaconda-navigator 1.9.12
anaconda-project 0.10.2
aniso8601 8.0.0
anndata 0.7.8
anyio 2.2.0
appdirs 1.4.4
applaunchservices 0.2.1
appnope 0.1.2
appscript 1.1.2
argh 0.26.2
argon2-cffi 20.1.0
asn1crypto 1.4.0
astroid 2.6.6
astropy 4.3.post1
async-generator 1.10
atomicwrites 1.4.0
attrs 21.2.0
autopep8 1.5.6
Babel 2.9.1
backcall 0.2.0
backports.functools-lru-cache 1.6.4
backports.shutil-get-terminal-size 1.0.0
backports.tempfile 1.0
backports.weakref 1.0.post1
beautifulsoup4 4.10.0
bitarray 2.3.0
bkcharts 0.2
black 19.10b0
bleach 4.0.0
bokeh 2.4.1
boto 2.49.0
boto3 1.15.16
botocore 1.18.16
Bottleneck 1.3.2
Brotli 1.0.9
brotlipy 0.7.0
cached-property 1.5.2
cellrank 1.5.0
cellxgene 0.16.4
certifi 2021.10.8
cffi 1.14.6
chardet 4.0.0
charset-normalizer 2.0.4
cirrocumulus 1.1.12
click 7.1.2
cloudpickle 2.0.0
clyent 1.2.2
colorama 0.4.4
conda 4.11.0
conda-build 3.21.6
conda-pack 0.6.0
conda-package-handling 1.7.3
conda-verify 3.4.2
contextlib2 0.6.0.post1
cryptography 35.0.0
cycler 0.10.0
Cython 0.29.21
cytoolz 0.11.0
dask 2021.10.0
debugpy 1.5.1
decorator 5.1.0
defusedxml 0.7.1
diff-match-patch 20200713
distributed 2021.10.0
docrep 0.3.2
docutils 0.17.1
dunamai 1.7.0
entrypoints 0.3
et-xmlfile 1.1.0
fastcache 1.1.0
fastobo 0.9.2
filelock 3.3.1
flake8 3.9.0
Flask 2.0.2
Flask-Compress 1.7.0
Flask-Cors 3.0.9
Flask-RESTful 0.3.8
flask-server-timing 0.1.2
flask-talisman 0.7.0
flatbuffers 1.12
flatten-dict 0.3.0
fonttools 4.28.1
fsspec 2021.10.1
future 0.18.2
get_version 3.5.3
gevent 21.8.0
glob2 0.7
gmpy2 2.0.8
greenlet 1.1.1
gunicorn 20.0.4
h5py 3.3.0
HeapDict 1.0.1
html5lib 1.1
idna 3.3
imagecodecs 2021.8.26
imageio 2.9.0
imagesize 1.3.0
importlib-metadata 4.8.1
iniconfig 1.1.1
intervaltree 3.1.0
ipykernel 6.4.1
ipython 7.29.0
ipython-genutils 0.2.0
ipywidgets 7.6.5
isort 5.9.3
itsdangerous 2.0.1
jdcal 1.4.1
jedi 0.17.2
Jinja2 3.0.2
jmespath 0.10.0
joblib 1.1.0
json5 0.9.6
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 7.0.6
jupyter-console 6.4.0
jupyter-core 4.9.1
jupyter-server 1.4.1
jupyterlab 3.2.1
jupyterlab-pygments 0.1.2
jupyterlab-server 2.8.2
jupyterlab-widgets 1.0.0
keyring 23.1.0
kiwisolver 1.3.1
lazy-object-proxy 1.6.0
legacy-api-wrap 1.2
libarchive-c 2.9
llvmlite 0.37.0
locket 0.2.1
loompy 3.0.6
louvain 0.7.0
lxml 4.6.3
MarkupSafe 2.0.1
matplotlib 3.5.0
matplotlib-inline 0.1.2
mccabe 0.6.1
mistune 0.8.4
mkl-fft 1.3.0
mkl-random 1.1.1
mkl-service 2.3.0
mock 4.0.3
more-itertools 8.12.0
mpi4py 3.1.2
mpmath 1.2.1
msgpack 1.0.2
multipledispatch 0.6.0
munkres 1.1.4
mypy-extensions 0.4.3
natsort 7.1.1
navigator-updater 0.2.1
nbclassic 0.2.6
nbclient 0.5.3
nbconvert 6.1.0
nbformat 5.1.3
nest-asyncio 1.5.1
networkx 2.6.3
nltk 3.6.5
nose 1.3.7
notebook 6.4.6
numba 0.54.1
numexpr 2.7.3
numpy 1.20.3
numpy-groupies 0.9.14
numpydoc 1.1.0
olefile 0.46
openpyxl 3.0.9
packaging 21.3
panda 0.3.1
pandas 1.3.4
pandocfilters 1.4.3
parso 0.7.0
partd 1.2.0
path 16.0.0
pathlib2 2.3.6
pathspec 0.7.0
pathtools 0.1.2
patsy 0.5.2
pep8 1.7.1
petsc4py 3.14.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.4.0
pip 21.2.4
pkginfo 1.7.1
pluggy 0.13.1
ply 3.11
progressbar2 3.55.0
prometheus-client 0.12.0
prompt-toolkit 3.0.20
psutil 5.8.0
ptyprocess 0.7.0
py 1.10.0
pyarrow 3.0.0
pycodestyle 2.6.0
pycosat 0.6.3
pycparser 2.21
pycurl 7.44.1
pydocstyle 6.1.1
pyerfa 1.7.2
pyflakes 2.2.0
pygam 0.8.0
Pygments 2.10.0
pygpcca 1.0.2
pylint 2.9.6
pyls-black 0.4.6
pyls-spyder 0.3.2
pymongo 3.11.2
pynndescent 0.5.5
pyodbc 4.0.0-unsupported
pyOpenSSL 21.0.0
pyparsing 3.0.4
pyrsistent 0.18.0
pysam 0.16.0.1
PySocks 1.7.1
pytest 6.2.4
python-dateutil 2.8.2
python-igraph 0.9.0
python-jsonrpc-server 0.4.0
python-language-server 0.36.2
python-utils 2.5.6
pytz 2021.3
PyWavelets 1.1.1
PyYAML 6.0
pyzmq 22.2.1
QDarkStyle 2.8.1
QtAwesome 1.0.3
qtconsole 5.1.1
QtPy 1.10.0
regex 2021.8.3
requests 2.26.0
rope 0.21.1
Rtree 0.9.7
ruamel-yaml-conda 0.15.100
s3fs 0.4.2
s3transfer 0.3.3
scanpy 1.8.2
scikit-image 0.18.3
scikit-learn 1.0.1
scipy 1.7.3
scvelo 0.2.4
seaborn 0.11.2
Send2Trash 1.8.0
setuptools 58.0.4
setuptools-scm 4.1.2
simplegeneric 0.8.1
sinfo 0.3.1
singledispatch 3.7.0
six 1.16.0
slepc4py 3.14.0
sniffio 1.2.0
snowballstemmer 2.2.0
sortedcollections 2.1.0
sortedcontainers 2.4.0
soupsieve 2.3.1
Sphinx 4.2.0
sphinxcontrib-applehelp 1.0.2
sphinxcontrib-devhelp 1.0.2
sphinxcontrib-htmlhelp 2.0.0
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-qthelp 1.0.3
sphinxcontrib-serializinghtml 1.1.5
sphinxcontrib-websupport 1.2.4
spyder 4.2.5
spyder-kernels 1.10.2
SQLAlchemy 1.4.22
statsmodels 0.12.2
stdlib-list 0.7.0
sympy 1.9
tables 3.6.1
tblib 1.7.0
terminado 0.9.4
testpath 0.5.0
textdistance 4.2.1
texttable 1.6.3
threadpoolctl 2.2.0
three-merge 0.1.1
tifffile 2021.7.2
tiledb 0.6.6
toml 0.10.2
toolz 0.11.2
tornado 6.1
tqdm 4.62.3
traitlets 5.1.1
typed-ast 1.4.3
typing-extensions 3.10.0.2
ujson 4.0.2
umap-learn 0.5.2
unicodecsv 0.14.1
urllib3 1.26.7
watchdog 1.0.2
wcwidth 0.2.5
webencodings 0.5.1
Werkzeug 2.0.2
wheel 0.37.0
widgetsnbextension 3.5.1
wrapt 1.13.3
wurlitzer 2.1.1
xlrd 2.0.1
XlsxWriter 3.0.2
xlwings 0.24.9
xlwt 1.3.0
xmltodict 0.12.0
yapf 0.31.0
zict 2.0.0
zipp 3.6.0
zope.event 4.5.0
zope.interface 5.4.0

##############################################################################################

[Marius1311]

@WeilerP @michalk8 do you have an idea of what may be causing this?

[michalk8]

There are several SO threads I've managed to find 1, 2, though nothing conclusive (one hypothesis that int32 is used as .indices and it isn't sufficient to store the maximum index). I've tried replicating this locally with some dummy data but didn't manage.
@mhwadsworth91 would it be possible for you to share with the the connectivities and the low confidence indices in question?
After this conn_new = conn.copy() on this line, you can add:

import pickle
with open("data.pkl", "wb") as fout:
    pickle.dump({"conn": conn_new, "ixs": idx_low_confidence}, fout, protocol=4)

most likely all related issues (taken from our lab):
scverse/anndata#339
theislab/scgen#49
theislab/sfaira#250
scverse/scanpy#1183

[mhwadsworth91]

Hello All,

Thank you for following up! Will run that today and upload it when complete.

In the meantime, if helpful for the troubleshooting process, attached is the code I used to process the H5AD object. Please note that, when I do not specify the root/end keys for the directed PAGA workup (line 718), I do not get the error above.

MWadsworth_CellRank_Troubleshooting_20211203.md
.

[Marius1311]

Closing this for now, feel free to re-open of the problem persists.

[Priyankanator]

Hi -
I am getting the same error -
running PAGA using priors: ['velocity_pseudotime', 'initial_states_probs', 'terminal_states_probs']

---

ValueError Traceback (most recent call last)
in ()
7 root_key="initial_states_probs",
8 end_key="terminal_states_probs",
----> 9 use_time_prior="velocity_pseudotime",
10 )
11 #scv.tl.paga(adata_subset, groups='seurat_clusters')

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scvelo/tools/paga.py in paga(adata, groups, vkey, use_time_prior, root_key, end_key, threshold_root_end_prior, minimum_spanning_tree, copy)
283 adata.uns[f"{groups}_sizes"] = np.array(paga.ns)
284
--> 285 paga.compute_transitions()
286 adata.uns["paga"]["transitions_confidence"] = paga.transitions_confidence
287 adata.uns["paga"]["threshold"] = paga.threshold

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scvelo/tools/paga.py in compute_transitions(self)
129 set_row_csr(vgraph, rows=np.where(self._adata.obs[self.end_key] > lb)[0])
130 if isinstance(self.root_key, str) and self.root_key in self._adata.obs.keys():
--> 131 vgraph[:, np.where(self._adata.obs[self.root_key] > lb)[0]] = 0
132 vgraph.eliminate_zeros()
133

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/_index.py in setitem(self, key, x)
122 return
123 x = x.reshape(i.shape)
--> 124 self._set_arrayXarray(i, j, x)
125
126 def _validate_indices(self, key):

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/compressed.py in _set_arrayXarray(self, row, col, x)
800 def _set_arrayXarray(self, row, col, x):
801 i, j = self._swap((row, col))
--> 802 self._set_many(i, j, x)
803
804 def _set_arrayXarray_sparse(self, row, col, x):

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/compressed.py in _set_many(self, i, j, x)
884 offsets = np.empty(n_samples, dtype=self.indices.dtype)
885 ret = csr_sample_offsets(M, N, self.indptr, self.indices, n_samples,
--> 886 i, j, offsets)
887 if ret == 1:
888 # rinse and repeat

ValueError: could not convert integer scalar

I am running the cellrank for 96K cells.

Has anybody resolved these issues ?
Thanks

[sfortma2]

Hi - I am getting the same error - running PAGA using priors: ['velocity_pseudotime', 'initial_states_probs', 'terminal_states_probs']

ValueError Traceback (most recent call last) in () 7 root_key="initial_states_probs", 8 end_key="terminal_states_probs", ----> 9 use_time_prior="velocity_pseudotime", 10 ) 11 #scv.tl.paga(adata_subset, groups='seurat_clusters')

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scvelo/tools/paga.py in paga(adata, groups, vkey, use_time_prior, root_key, end_key, threshold_root_end_prior, minimum_spanning_tree, copy) 283 adata.uns[f"{groups}_sizes"] = np.array(paga.ns) 284 --> 285 paga.compute_transitions() 286 adata.uns["paga"]["transitions_confidence"] = paga.transitions_confidence 287 adata.uns["paga"]["threshold"] = paga.threshold

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scvelo/tools/paga.py in compute_transitions(self) 129 set_row_csr(vgraph, rows=np.where(self._adata.obs[self.end_key] > lb)[0]) 130 if isinstance(self.root_key, str) and self.root_key in self._adata.obs.keys(): --> 131 vgraph[:, np.where(self._adata.obs[self.root_key] > lb)[0]] = 0 132 vgraph.eliminate_zeros() 133

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/_index.py in setitem(self, key, x) 122 return 123 x = x.reshape(i.shape) --> 124 self._set_arrayXarray(i, j, x) 125 126 def _validate_indices(self, key):

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/compressed.py in _set_arrayXarray(self, row, col, x) 800 def _set_arrayXarray(self, row, col, x): 801 i, j = self._swap((row, col)) --> 802 self._set_many(i, j, x) 803 804 def _set_arrayXarray_sparse(self, row, col, x):

/home/priyanka/miniconda3/envs/ngs/lib/python3.6/site-packages/scipy/sparse/compressed.py in _set_many(self, i, j, x) 884 offsets = np.empty(n_samples, dtype=self.indices.dtype) 885 ret = csr_sample_offsets(M, N, self.indptr, self.indices, n_samples, --> 886 i, j, offsets) 887 if ret == 1: 888 # rinse and repeat

ValueError: could not convert integer scalar

I am running the cellrank for 96K cells.

Has anybody resolved these issues ? Thanks

I'm running into the same issue using the same function and priors. @Priyankanator Did you have any luck solving this one? Thanks!

[Marius1311]

Seems like this is an scvelo issue, would it make sense to move the issue to the scVelo repository?