Error in orderCells: long vectors not supported yet

[marcovito]

Hi guys,

I'm using monocle on a big dataset (>50k cells). I'm getting this error calling the orderCells function:

Error in graph.adjacency.dense(adjmatrix, mode = mode, weighted = weighted, :
long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138
Calls: orderCells ... project2MST -> graph.adjacency -> graph.adjacency.dense -> .Call

I'm sure I'm using a sparse Matrix. I suppose this could be a problem in igraph or even in R core functions. Can anybody help me fix it?
Thx, Marco.

[Xiaojieqiu]

Hi @marcovito, we are overhauling the reduceDimension and orderCells functions now. It will support analyzing millions of cells. This issue will be fixed soon!

[shanhuhaishine]

Hi,
I ran into the same problem too. Is there any good solutions to the problem?
Thx, Shine.

[liuyifang]

Hi,
I ran into the same problem too. Any good ideas?
Thx, Yifang

[MartinaHox]

hey guys! any update on this?

[Xiaojieqiu]

Hey, are you guys using monocle3-alpha?

[MartinaHox]

Yeah — just started! Looking forward to the official release... On Jan 8, 2019, at 23:44, Xiaojie Qiu <notifications@github.com<mailto:notifications@github.com>> wrote: Hey, are you guys using monocle3-alpha? — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#138 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AobZ5YCguXwq6AGNs10Z7XuOJROVRrtnks5vBXOvgaJpZM4TkznW>.

[Xiaojieqiu]

we will hopefully release it in near future. Monocle 3alpha should easily handle 50 k cells. please let me if there are any future issues

[MartinaHox]

Ehm, how about 1.5 million? :) On Jan 8, 2019, at 23:49, Xiaojie Qiu <notifications@github.com<mailto:notifications@github.com>> wrote: we will hopefully release it in near future. Monocle 3alpha should easily handle 50 k cells. please let me if there are any future issues — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#138 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AobZ5QArVxQNLM2kUKF2E4nJgHFCGxjXks5vBXTxgaJpZM4TkznW>.

[Xiaojieqiu]

it depends. If your data is sparse enough and all 1.5 million cell data can be put into a single sparse matrix, it should work just fine. Unfortunately R is limited to store 2^31-1 bit data for a single sparse matrix (or vector), so anything larger than that seems not feasible to keep in memory. If your data is not able to store in a sparse matrix, consider taking only the top 1000 variable genes or so and then do the downstream analysis.

There are plans to rewrite most monocle functions to support hdf5 so that we only connect a hdf5 file on disk and manipulate chunks of data each time in memory.

[SJW-Danny]

Hi! Has anyone solved this problem?
`> cds <- setOrderingFilter(cds_obj, ordergene)

dim(cds)
Features Samples
2000 53794
cds <- reduceDimension(cds, max_components = 2,method = 'DDRTree')
Warning messages:
1: In eval(call, parent.frame()) : NaNs produced
2: Quick-TRANSfer stage steps exceeded maximum (= 2689700)
cds <- orderCells(cds)
Error in graph.adjacency.dense(adjmatrix, mode = mode, weighted = weighted, :
long vectors not supported yet: ../../src/include/Rinlinedfuns.h:537
In addition: There were 50 or more warnings (use warnings() to see the first 50)`

[ZivTQ]

Hi guys,

I'm using monocle on a big dataset (>50k cells). I'm getting this error calling the orderCells function:

Error in graph.adjacency.dense(adjmatrix, mode = mode, weighted = weighted, : long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138 Calls: orderCells ... project2MST -> graph.adjacency -> graph.adjacency.dense -> .Call

I'm sure I'm using a sparse Matrix. I suppose this could be a problem in igraph or even in R core functions. Can anybody help me fix it? Thx, Marco.

The error occured might be the larger number cells. When I randomly sampled 5000 cells，it works.

[MiaLin31]

Hi guys,
I'm using monocle on a big dataset (>50k cells). I'm getting this error calling the orderCells function:
Error in graph.adjacency.dense(adjmatrix, mode = mode, weighted = weighted, : long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138 Calls: orderCells ... project2MST -> graph.adjacency -> graph.adjacency.dense -> .Call
I'm sure I'm using a sparse Matrix. I suppose this could be a problem in igraph or even in R core functions. Can anybody help me fix it? Thx, Marco.

The error occured might be the larger number cells. When I randomly sampled 5000 cells，it works.

@ZivTQ
Dear ZivTQ,

I hope this message finds you well. I've encountered the same error, and I was wondering if you could provide more information about the code related to "randomly sampled 5000 cells" that you mentioned. Could you please share the specific code and let me know which step it should be added to?

Thx, Lin

[huai-su]

Hi guys,
I'm using monocle on a big dataset (>50k cells). I'm getting this error calling the orderCells function:
Error in graph.adjacency.dense(adjmatrix, mode = mode, weighted = weighted, : long vectors not supported yet: ../../src/include/Rinlinedfuns.h:138 Calls: orderCells ... project2MST -> graph.adjacency -> graph.adjacency.dense -> .Call
I'm sure I'm using a sparse Matrix. I suppose this could be a problem in igraph or even in R core functions. Can anybody help me fix it? Thx, Marco.

The error occured might be the larger number cells. When I randomly sampled 5000 cells，it works.

@ZivTQ Dear ZivTQ,

I hope this message finds you well. I've encountered the same error, and I was wondering if you could provide more information about the code related to "randomly sampled 5000 cells" that you mentioned. Could you please share the specific code and let me know which step it should be added to?

Thx, Lin

I think maybe use
sce<-subset(sce, downsample = 1000),this will downsample by ident and get smaller data

[huai-su]

I met the same problem and the data dim is Features= 26465 Samples=50634
I wonder in R how many cells monocle can handle at most