Mhap java.lang.ArrayIndexOutOfBoundsException during 1-overlapper stage

[roblehmann]

Hey,
I am trying to assemble my ~1Gb genome using canu commit d2cb41e. However, during the 1-overlapper stage there is always one job left which doesn't terminate until killed due to max walltime. How can I debug this issue?

The log of that job shows an exception being thrown:

Running job 358 based on SLURM_ARRAY_TASK_ID=358 and offset=0.
Running block 000097 in query 000358
Running with these settings:
--filter-threshold = 5.0E-6
--help = false
--max-shift = 0.2
--min-olap-length = 116
--min-store-length = 0
--no-rc = false
--no-self = true
--no-tf = false
--num-hashes = 768
--num-min-matches = 2
--num-threads = 16
--ordered-kmer-size = 12
--ordered-sketch-size = 1536
--repeat-idf-scale = 10.0
--repeat-weight = 0.9
--settings = 0
--store-full-id = false
--supress-noise = 0
--threshold = 0.73
--version = false
-f = ../0-mercounts/mygenome_canu.ms16.frequentMers.ignore.gz
-h = false
-k = 16
-p =
-q = queries/000358
-s = ./blocks/000097.dat

Stored 69000 sequences in the index.
Processed 69000 unique sequences (fwd and rev).
Time (s) to read and hash from file: 109.57163050700001
Opened fasta file /data/dragon/lehmanr/mygenome_pacbio_canu/mygenome/correction/1-overlapper/blocks/000166.dat.
Exception in thread "pool-4-thread-1" java.lang.ArrayIndexOutOfBoundsException: -2147483648
        at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.dtoa(FloatingDecimal.java:594)
        at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.access$100(FloatingDecimal.java:259)
        at sun.misc.FloatingDecimal.getBinaryToASCIIConverter(FloatingDecimal.java:1785)
       	at sun.misc.FormattedFloatingDecimal.valueOf(FormattedFloatingDecimal.java:37)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3298)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3238)
       	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2802)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2753)
        at java.util.Formatter.format(Formatter.java:2520)
        at java.util.Formatter.format(Formatter.java:2455)
        at java.lang.String.format(String.java:2928)
        at edu.umd.marbl.mhap.impl.MatchResult.toString(MatchResult.java:100)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch.outputResults(AbstractMatchSearch.java:327)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch$3.run(AbstractMatchSearch.java:253)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Exception in thread "pool-4-thread-16" java.lang.ArrayIndexOutOfBoundsException: 20
       	at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.dtoa(FloatingDecimal.java:750)
        at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.access$100(FloatingDecimal.java:259)
        at sun.misc.FloatingDecimal.getBinaryToASCIIConverter(FloatingDecimal.java:1785)
        at sun.misc.FormattedFloatingDecimal.valueOf(FormattedFloatingDecimal.java:37)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3298)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3238)
       	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2802)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2753)
       	at java.util.Formatter.format(Formatter.java:2520)
        at java.util.Formatter.format(Formatter.java:2455)
        at java.lang.String.format(String.java:2928)
        at edu.umd.marbl.mhap.impl.MatchResult.toString(MatchResult.java:100)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch.outputResults(AbstractMatchSearch.java:327)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch$3.run(AbstractMatchSearch.java:253)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
Current # sequences loaded and processed from file: 5000...
Current # sequences loaded and processed from file: 10000...
Current # sequences loaded and processed from file: 15000...
Current # sequences loaded and processed from file: 20000...
Current # sequences loaded and processed from file: 25000...
Current # sequences loaded and processed from file: 30000...
Exception in thread "pool-4-thread-9" java.lang.ArrayIndexOutOfBoundsException: -2147483648
       	at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.dtoa(FloatingDecimal.java:594)
        at sun.misc.FloatingDecimal$BinaryToASCIIBuffer.access$100(FloatingDecimal.java:259)
        at sun.misc.FloatingDecimal.getBinaryToASCIIConverter(FloatingDecimal.java:1785)
        at sun.misc.FormattedFloatingDecimal.valueOf(FormattedFloatingDecimal.java:37)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3298)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:3238)
       	at java.util.Formatter$FormatSpecifier.printFloat(Formatter.java:2802)
       	at java.util.Formatter$FormatSpecifier.print(Formatter.java:2753)
       	at java.util.Formatter.format(Formatter.java:2520)
        at java.util.Formatter.format(Formatter.java:2455)
        at java.lang.String.format(String.java:2928)
        at edu.umd.marbl.mhap.impl.MatchResult.toString(MatchResult.java:100)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch.outputResults(AbstractMatchSearch.java:327)
        at edu.umd.marbl.mhap.impl.AbstractMatchSearch$3.run(AbstractMatchSearch.java:253)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

[skoren]

Canu doesn't try to request wall time for its jobs so depending on your machine configuration it is up to you to add a time request to the gridOptions option.

Based on the error I think the output from the mhap precompute step is also corrupt, probably due to time limits as well. I would suggest restarting from scratch and specifying a time limit of 2 or 3 times bigger than whatever you're using now to make sure they run.

[roblehmann]

Hello Sergey,

thanks for the quick reply. Yes, I forgot to mention that all jobs on our cluster must come with a time limit set.

However, the meryl output looks ok to me, and also the precompute logs don't report any exceptions or errors. Interestingly, the error occurs only when setting the corMhapSensitivity=high. On normal setting it worked flawlessly.

cheers,
Rob

[skoren]

Increasing the sensitivity increases the runtime so your original run probably didn't hit the time limits.

You wouldn't necessarily see exceptions in the precompute logs, they may have a message from your scheduler if they were terminated early. I think it would be safest to remove the 1-overlapper folder and re-run specifying a time limit 2-3x longer than what you were using before which should hopefully let both the precompute and mhap jobs finish.

[skoren]

Closing, inactivity, open a new issue if you encounter errors.