Align: not enough memory for BAM sorting potentially due to BinsN too high

[cpcouturier]

In the past, changing ulimit -Hn had fixed this. Now running this on Terra and this parameter is much more difficult to change. Changing the way BinsN is chosen or allowing an optional input to set it would be great!

Thanks!

[Lioscro]

Hi, @cpcouturier,
I don't think this is related to ulimit because dynast temporarily increases the maximum number of open file descriptors to the maximum allowed by the OS. The interplay between ulimit, limitBAMsortRAM, and the amount of required memory is a bit confusing, but here is an issue in the STAR github repo that sheds some light: alexdobin/STAR#457 (comment)

Some key points are that higher outBAMsortingBinsN reduces memory requirements because the BAM sorting is done per bin and merged. The value automatically calculated (by dynast, in your case it seems it is 131062) should be pretty close to the maximum that your system can support. You can always override any STAR argument using the --STAR-overrides argument. For instance, you may want to provide --limitBAMsortRAM to the value that STAR is suggesting like so.

dynast align ... --STAR-overrides="--limitBAMsortRAM 101537264536 --otherArgument otherParameter"

Note that the overrides should be in quotes so that it doesn't get interpreted as dynast arguments. The same can be done for --outBAMsortingBinsN if you wish.

[cpcouturier]

Thanks @Lioscro!

I tried --STAR-overrides with the sortRAM and BinsN. It seems both have no impact on the outcome. In the case of BinsN, both the modified and native arguments appear. Is the override having the intended effect here?

[Lioscro]

The shell you are running dynast on is interpreting the double quotes as part of the argument. I've just pushed an update to strip quotes from the string, so I think it should work after you install the newest version.

[cpcouturier]

Awesome! Thank you @Lioscro! I reduced BinsN and increased sortRAM in separate runs - both worked.

Worth saying that locally (MacBook Pro), ulimit -Hn was set to unlimited (which came up to max 96bit) by default. It was trying to run it with hundreds of millions of bins which made it crash with a memory error. Adjusting ulimit -Hn to something reasonable (20 000 or so) had worked since I believe this value is used as input to decide the number of bins in your code.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days