OSError: [Errno 12] Cannot allocate memory

[joweihsieh]

Hi Atul,

Thank you for developing this tool.

I have been using sPARTA to identify miRNA targets in both genic and intergenic regions using two PARE libraries from bread wheat. While I successfully processed the genic regions, I encountered memory issues when attempting to process the intergenic regions, despite allocating approximately 450GB of memory for the task.

I wonder if you have any suggestions on how to handle this issue. Would you recommend separating the genome into several smaller files and running the analysis multiple times? Are there any resource allocation parameters in the script that I can revise to better manage the memory usage?

Thanks,
Jo-Wei

============

• Here is my command for intergenic regions
  nohup python3 ~/bin/sPARTA/sPARTA.py -genomeFile iwgsc_refseqTwoPointOne_assembly.fa -annoType GFF -annoFile iwgsc_refseqTwoPointOne.gff3 -genomeFeature 1 -miRNAFile miRNA_combined_noDup.fa -libs 11787.processed.txt 11788.processed.txt -tarPred H -tarScore N --tag2FASTA --map2DD --validate -accel 60 -minTagLen 18 &

• Here is the error
  Creating PAGe Index dictionary for lib 11787.processed
  PAGe Indexing took 248.73 seconds
  Writing PAGeIndex file...
  median = 6.0
  seventyFivePercentile = 13.0
  ninetyPercentile = 49.0
  File written. Process took 2.07 seconds
  Finding the validated targets
  Exception in thread Thread-16 (_handle_workers):
  Traceback (most recent call last):
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
  self.run()
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/threading.py", line 1010, in run
  self._target(*self._args, **self._kwargs)
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/pool.py", line 516, in _handle_workers
  cls._maintain_pool(ctx, Process, processes, pool, inqueue,
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/pool.py", line 340, in _maintain_pool
  Pool._repopulate_pool_static(ctx, Process, processes, pool,
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/pool.py", line 329, in _repopulate_pool_static
  w.start()
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/process.py", line 121, in start
  self._popen = self._Popen(self)
  ^^^^^^^^^^^^^^^^^
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/context.py", line 282, in _Popen
  return Popen(process_obj)
  ^^^^^^^^^^^^^^^^^^
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/popen_fork.py", line 19, in init
  self._launch(process_obj)
  File "/work4/home/joweihsieh/miniconda3/lib/python3.12/multiprocessing/popen_fork.py", line 66, in _launch
  self.pid = os.fork()
  ^^^^^^^^^
  OSError: [Errno 12] Cannot allocate memory

[pblumenkamp]

Hi,
Thanks for this really cool software!

I'm running into the same problem with Hordeum vulgare. Trying to find intergenic targets uses more than 1.5TB of memory. Would it be possible to add an option to limit the used memory?

Thanks,
Patrick