Partitioning failed; increase redMemory

[kokevin]

Hi, I'm running into a problem with redMemory error. I'm running canu 2.2 (downloaded from github) and getting a redMemory error (please see below). The canu command looks like this:

canu -p test -d ./test genomeSize=4000 useGrid=false minReadLength=500 minOverlapLength=300 corOutCoverage=999 contigFilter="1 0 1.0 0.5 5" -nanopore sample.fastq.gz

and it is being called inside a python script with multiprocessing pool. I print the amount of free memory (using free -t -m) before each run and it's about 95% free only about 5% is used. The machine has about 100GB free disk space and 16GB of ram.
This is the error I see:

...
-- Configure RED for 15gb memory.                                                                                    
--                   Batches of at most (unlimited) reads.                                                           
--                                      500000000 bases.                                                             
--                   Expecting evidence of at most 536870912 bases per iteration.                                    
--                                                                                                                   
--           Total                                               Reads                 Olaps Evidence                
--    Job   Memory      Read Range         Reads        Bases   Memory        Olaps   Memory   Memory  (Memory in MB)
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------                
--      1  3079.44         1-61               26        60937     7.44          346     0.00  1024.00                
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------                
--                                                      60937                   346                                  
                                                                                                                     
ABORT:                                                                                                               
ABORT: canu 2.2                                                                                                      
ABORT: Don't panic, but a mostly harmless error occurred and Canu stopped.                                           
ABORT: Try restarting.  If that doesn't work, ask for help.                                                          
ABORT:                                                                                                               
ABORT:   partitioning failed; increase redMemory.                                                                    
ABORT:                                                                                                               

I have access to another machine that has the same physical setup but a different version of canu. If I run the same python script on that machine, it runs fine. The canu --version on this machine says canu branch HEAD +0 changes (r10117 5638f7d9a5379373310ab62c28aa0cdbd864722d). I believe it was compiled by someone and it doesn't have any redMemory errors when I run:

-- Configure RED for 15gb memory.                                                                                               
--                   Batches of at most (unlimited) reads.                                                                      
--                                      500000000 bases.                                                                        
--                   Expecting evidence of at most 536870912 bases per iteration.                                               
--                                                                                                                              
--           Total                                               Reads                 Olaps Evidence                           
--    Job   Memory      Read Range         Reads        Bases   Memory        Olaps   Memory   Memory  (Memory in MB)           
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------                           
--      1  3079.88         1-61               28        64541     7.88          354     0.00  1024.00                           
--   ---- -------- ------------------- --------- ------------ -------- ------------ -------- --------                           
--                                                      64541                   354                                             
-- Finished stage 'readErrorDetectionConfigure', reset canuIteration.                                                           
--                                                                                                                              
-- Running jobs.  First attempt out of 2.                                                                                       
----------------------------------------                                                                                        
-- Starting 'red' concurrent execution on Thu Sep 30 20:21:33 2021 with 105.444 GB free disk space (1 processes; 1 concurrently)
                                                                                                                                
    cd unitigging/3-overlapErrorAdjustment                                                                                      
    ./red.sh 1 > ./red.000001.out 2>&1                                                                                          
                                                                                                                                
-- Finished on Thu Sep 30 20:21:33 2021 (lickety-split) with 105.444 GB free disk space                                         
...

Can someone help me figure out how to fix the redMemory issue? It's strange that canu is only running into problems on one machine.

(edited to add)

I searched through the source code and found the message is in line 265 of https://github.com/marbl/canu/blob/master/src/pipelines/canu/OverlapErrorAdjustment.pm:

    if (($minReads < 100) ||
        (scalar(@bgn) > 9999)) {
        caExit("partitioning failed; increase redMemory", undef);
    }

Does this mean canu won't run if there are less then 100 reads? I'm not sure what the variable bgn is but I suspect my issue is with the low number of reads.

[skoren]

Yes, looks like you're right. The fix (which I just committed) is to allow less than 100 reads, as long as that's all your reads:

-    if (($minReads < 100) ||
+    if (($minReads < 100 && $minReads < $maxID) ||

[kokevin]

Great, thank you! Is there a link to the updated tar.xz that I can download?

[skoren]

No, you'd have to download/compile the tip or you could make the edit yourself in your <path to canu>/lib/site_perl/canu/OverlapErrorAdjustment.pm

[kokevin]

Thanks. I will make the edit directly in the OverlapErrorAdjustment.pm file. This seems to be the easiest way to go. Thank you very much for your help. I really appreciate it.