program quits when the matrix is big

[fangge-Li]

Hi all,
Thank you for developing "STREAM". I have used STREAM for our data. It works very well when the gene-expression matrix is small.
Now I want to use STREAM for the new data containing 7465 cells and 8858 genes. And I have allocated 8 cores and 16G memory to Docker container. But the program would quit after printing "Selecting features...", without any error message.
Please help me to solve this problem.
Thank you so much!

[huidongchen]

Hi, Fangge,

Thanks for trying STREAM.
Would you mind sharing with us the data matrix ? We are happy to take a look.

Best,
Huidong

[fangge-Li]

Hi Huidong, My data is 6M after being compressed, which is beyond the limit of message (4M). Could you tell me your email? Then I could send the data to you. Thank you. Best, Fangge

…

________________________________ From: Huidong Chen <notifications@github.com> Sent: Thursday, July 5, 2018 1:46:03 PM To: pinellolab/STREAM Cc: Li, Fangge; Author Subject: Re: [pinellolab/STREAM] program quits when the matrix is big (#4) Hi, Fangge, Thanks for trying STREAM. Would you mind sharing with us the data matrix ? We are happy to take a look. Best, Huidong — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_pinellolab_STREAM_issues_4-23issuecomment-2D402801208&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=geCAlDrCdg0K2wtC_eUWS71IXeKxYNipZUXui0gCd-U&m=sqP4CBIoJNPPc_LwBjirRCOf9w9Kj8OIinAF2uXpOrs&s=P8t3oWnuXUxw9tiXu6J_J5SKjekyLJkh0vR2iiK6mGA&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AmymtSe3wArnBNHg4XORWe4-5FWaMMHU3kks5uDlDbgaJpZM4VBWaH&d=DwMCaQ&c=WO-RGvefibhHBZq3fL85hQ&r=geCAlDrCdg0K2wtC_eUWS71IXeKxYNipZUXui0gCd-U&m=sqP4CBIoJNPPc_LwBjirRCOf9w9Kj8OIinAF2uXpOrs&s=8GxwMf66kPzoq65Wj56ozRlUXg6w7oERvZr5viMfvl4&e=>.

[huidongchen]

@fangge-Li Sure!

[cstill1992]

Hi,

I was wondering if there was any further update to this issue. I'm running into a similar situation where st.dimension_reduction(data) keeps quiting due to memory constraints (specifically I'm getting feedback that the process has been "killed" after about 2-3 hours of working). I'm working with a large dataset as well, 13230 cells with 10622 genes present (Drop-seq dataset). I'm doing this on a cluster allocating 30 cores and 128G of memory. Any help would be appreciated.

Thanks,
Chris

[huidongchen]

Hi Chris @cstill1992 ,

Sorry about the issue. st.dimension_reduction(data) uses 'MLLE' as the default dimension reduction method, which currently is not efficiently implemented yet.

As an alternative solution, you can try

st.dimension_reduction(adata,method='umap', n_components=2)
st.seed_elastic_principal_graph(adata ,clustering='kmeans')

Of course you can also increase the n_components to get more detailed structure.

This will speed it up significantly for large dataset.

[cstill1992]

Hi Huidong,

Thanks for the quick response. I'll give it a try and let you know how it goes.

Thanks,
Chris

[cstill1992]

Hi Huidong,

Thanks for the help. The fix worked nicely.

Best,
Chris

[huidongchen]

That’s great! Feel free to let me know if you have any questions.