Scipy sparse error: Cannot convert integer scalar

[le-ander]

During data store writing we hit this super strange bug again:

SCRIPT loading human_retina_2019_10xsequencing_menon_001_10.1038/s41467-019-12780-8
SCRIPT loading human_thymus_2020_10xsequencing_park_001_10.1126/science.aay3224
SCRIPT loading human_prostategland_2018_10xsequencing_henry_001_10.1016/j.celrep.2018.11.086
SCRIPT loading human_None_2020_scirnaseq_cao_001_10.1126/science.aba7721
Traceback (most recent call last):
  File "/storage/groups/ml01/code/leander.dony/sfaira/sfaira/data/utils_scripts/write_store.py", line 20, in <module>
    ds.streamline_features(
  File "/storage/groups/ml01/code/leander.dony/sfaira/sfaira/data/base/dataset.py", line 633, in streamline_features
    x_new[:, idx_feature_map[i:i + step]] = x[:, i:i + step]
  File "/opt/python/lib/python3.8/site-packages/scipy/sparse/_index.py", line 116, in __setitem__
    self._set_arrayXarray_sparse(i, j, x)
  File "/opt/python/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 807, in _set_arrayXarray_sparse
    self._zero_many(*self._swap((row, col)))
  File "/opt/python/lib/python3.8/site-packages/scipy/sparse/compressed.py", line 927, in _zero_many
    ret = csr_sample_offsets(M, N, self.indptr, self.indices, n_samples,
ValueError: could not convert integer scalar

We already copy the matrix over in chunks to circumvent this issue (was fine like this for a year now):

sfaira/sfaira/data/base/dataset.py

Lines 626 to 634 in 5b0a7b9

	# copying this over to the new matrix in chunks of size `steps` prevents a strange scipy error:
	# ... scipy/sparse/compressed.py", line 922, in _zero_many i, j, offsets)
	# ValueError: could not convert integer scalar
	step = 2000
	if step < len(idx_feature_map):
	i = 0
	for i in range(0, len(idx_feature_map), step):
	x_new[:, idx_feature_map[i:i + step]] = x[:, i:i + step]
	x_new[:, idx_feature_map[i + step:]] = x[:, i + step:]

I'll try and reduce step size and see if this goes away again (human_None_2020_scirnaseq_cao_001_10.1126/science.aba7721 has a very large number of features I think, which could cause this to break again)

[le-ander]

step size of 500 seems to do the trick. this is something we should at some point fix properly though