in case of not enough RAM, genome indexing hangs without throwing any message

[aushev]

STAR_2.6.1d
I launched as following: STAR --runMode genomeGenerate --genomeDir $fnGindexStar --genomeFastaFiles $fnGenomeFa --sjdbGTFfile $fnAnnot --sjdbOverhang 149 --runThreadN 8
Log stopped at "sorting Suffix Array chunks and saving them to disk..." - I was waiting for more than a day (because just the previous message was "This may take a long time..."). Then I've realized that something is probably wrong and started to google. After reading some forums, found a solution with --limitGenomeGenerateRAM.
I think it would be very useful to add some informative message indicating lack of RAM, and, more importantly, stop execution - otherwise user has no idea what's going on and will just wait keeping the program running (i.e. wasting computational time on cluster).

Thanks!

[alexdobin]

Hi @aushev

thanks for this suggestion. I do need to improve memory handling, however, it's not easy as memory allocation is done mostly by the OS. For instance, if OS runs out of virtual memory, the code should crash with bad_alloc message. However, if it runs of physical but not virtual memory, it will start swapping memory to disk, which basically hangs the system. The latter case is much harder to catch.

Cheers
Alex

[nicolasrochette]

I confirm this issue. Running the same command:

STAR --runMode genomeGenerate --genomeFastaFiles ./Anolis_carolinensis.AnoCar2.0.dna.toplevel.fa --sjdbGTFfile ./Anolis_carolinensis.AnoCar2.0.95.gtf --genomeDir ./

in a grid job with respectively 16G, 32G and 48G memory leads to (1) a crash after 2 minutes on a std::bad_alloc throw, (2) STAR hanging for 10+hours at >90% CPU and 21G mem, (3) a successful completion in 1h20min with a max memory usage of 35G

[alexdobin]

Hi @nicolasrochette

thanks for reporting this. As I mentioned, this is not an easy problem to tackle - when RAM consumption is just above the available physical RAM, Linux starts swapping parts of RAM to disk which effectively hangs the process.
I used to have mlock in the code to lock the RAM from swapping, but it did not work well at the time... I need to revisit it.

Cheers
Alex

[aushev]

just noticed that even --limitGenomeGenerateRAM does not help sometimes. I have just tried to run indexing with --limitGenomeGenerateRAM 16000000000 on a cluster node with 20Gb of RAM allocated for this job - and had the same issue! Interestingly, cluster logs for memory usage say "MAX MEM: 12.5 Gbytes; AVG MEM: 10.8 Gbytes" - i.e. it did not even reach the limit! I would assume it could be some different problem - but at least the symptoms are exactly the same: process hung after "sorting Suffix Array chunks and saving them to disk"...

[alexdobin]

Hi @aushev

the final genome index has to fit into the RAM, and --limitGenomeGenerateRAM does not control that. It only controls the size of temporary files, and is helpful to increase it if you have more RAM that is needed for the genome index.

Cheers
Alex

[nicolasrochette]

Hi Alex,

I'd like to mention, in addition to what I said previous, that my three grid jobs actually ran on the same machine/node. My 48G job completed successfully and my 16G job must have hit some sort of soft limit, but it really seems like the 32G job just stopped trying to allocate more memory, as if caught in an infinite loop. Isn't there any part of the code that tries to detect the available memory prevent the job from hitting the ceiling?

[alexdobin]

Hi Nicholas,

the management of physical and virtual RAM is complicated, especially so when cluster schedulers are involved. There is no simple function that will show the available physical RAM. If a process request more RAM than available, depending on the amount of overcommitment, the system can either throw std::bad_alloc or decide to swap. The latter will often hang the process.
I am thinking of how to best deal with it, and will be happy to hear suggestions.

Cheers
Alex

[aushev]

Hello Alex,
you say "There is no simple function that will show the available physical RAM". But when I submit my job to the cluster, I indicate how much memory I'm requesting for it. Will it more or less correspond to that "available physical RAM" amount? If so, maybe it would be possible to add a command line parameter, something like --availableRAM, so that STAR will know what is the limit?

[alexdobin]

Hi @aushev

the OS/scheduler can see how much RAM (physical or virtual) a process is using and kill it when it tries to use more. The limit is more likely to be set for virtual rather than physical RAM, allowing swapping.

I need to do a better job in terms of memory management - i.e. calculate beforehand how much RAM will be needed and pre-allocate all arrays at the start. However, even this may not solve all the swap problems as Linux will swap out idle RAM pages.

Cheers
Alex