Type comparison issue when using read_h5ad in backed mode

[jorvis]

If I read a file with read_h5ad() and then process with

sc.pp.filter_genes(adata, min_cells=int(foo))
Things work as intended.

But if I change that read line to be read_h5ad(h5_path, backed='r') then when I attempt to filter I get this error instead:

File "/opt/Python-3.7.3/lib/python3.7/site-packages/scanpy/preprocessing/_simple.py", line 228, in filter_genes:
else X > 0, axis=0):
TypeError: '>' not supported between instances of 'SparseDataset' and 'int'

[Koncopd]

Hi, @jorvis . There is no functionality for filtering in the backed mode.

[jorvis]

Any suggestions around this? Without reading in backed mode just loading the dataset of around 200,000 cells by 30,000 genes is using over 40GB of RAM. The filtering steps help us reduce this.

[Koncopd]

Yeah, i understand the problem. There are some some things that can be done in backed mode (like pca), but not filtering for now. Adding annotation based "filtering" that will work fine with the backed mode is planned but i can't give any exact timeline.

[ivirshup]

@jorvis, while you currently can't filter (since that modifies the AnnData object) you could subset it, generating a view (i.e. adata[:, adata.var["n_cells_by_counts"] <= foo].

Would you mind telling us a little about what you'd like to be able to do?

[jorvis]

OK, that's helpful, thanks. I really just want to get through the standard clustering pipeline, but in backed mode since we have some datasets which are too expensive to load into RAM (40-50GB). We'll accept the speed trade-off of backed mode to allow for scalability here.

[ivirshup]

Hmm. If I'm understanding correctly, are you not able to load the full dataset into memory on your machine? Can you give me an idea of what your memory restrictions are?

[jorvis]

We've actually built a site which allows users to browse hundreds of datasets and use functions like those provided by scanpy in an interactive way. So while the server does have the resources to open files like this one, it can't support 10+ users doing it at the same time. It would be prohibitively expensive to run a VM with 1TB of RAM to support the existing user base, so I was trying to autodetect the input file size and if over a certain threshold use backed mode rather than reading into memory. I hope this helps.