pbmm2 using too much memory

[zhoudreames]

my machine is down when i using pbmm2 aligning,my machine memory is 500G,and the hifi data 130G.how to solve it ?

[armintoepfer]

You can't use a full chip worth of Hifi reads as references. DeepConsensus is in the proof of concept stage, not meant for production yet. You can try this new tool, but there's no official support for it yet: https://github.com/PacificBiosciences/align-clr-to-ccs

[AndrewCarroll]

Hi @zhoudreames

As @armintoepfer indicates, DeepConsensus isn't yet scalable to run a full SMRT cell on an external machine. The v0.1 release still uses many systems that are efficient within Google's distributed systems, but are very inefficient when run on a single machine. We plan to improve this in future releases, but for now, it will be too slow for a full SMRT cell. If there is something you can use a targeted part of a sequence run, that might be better.

For the error you are encountering, I assume that this is in the pre-processing step? Thank you for pointing this out. This is one of the areas we will improve in the next release (potentially by using the tool that Armin linked). For now, to map the reads, you would need to use fewer reads for your input.

Thank you,
Andrew

[zhoudreames]

You can't use a full chip worth of Hifi reads as references. DeepConsensus is in the proof of concept stage, not meant for production yet. You can try this new tool, but there's no official support for it yet: https://github.com/PacificBiosciences/align-clr-to-ccs

thanks for your help

[zhoudreames]

Hi @zhoudreames

As @armintoepfer indicates, DeepConsensus isn't yet scalable to run a full SMRT cell on an external machine. The v0.1 release still uses many systems that are efficient within Google's distributed systems, but are very inefficient when run on a single machine. We plan to improve this in future releases, but for now, it will be too slow for a full SMRT cell. If there is something you can use a targeted part of a sequence run, that might be better.

For the error you are encountering, I assume that this is in the pre-processing step? Thank you for pointing this out. This is one of the areas we will improve in the next release (potentially by using the tool that Armin linked). For now, to map the reads, you would need to use fewer reads for your input.

Thank you,
Andrew
when will the next version releases ? How much do I need to split reads for the 500G menmory ？ thanks~

[AndrewCarroll]

Hi @zhoudreames

I cannot give you an estimate for when the next version will be ready. With a fair amount of certainty, I will say likely more than 1 month from now and less than 6 months.

The changes in the next release will probably change the memory use, and I cannot now give you an estimate of memory at that time.