dies with Invalid reference index -1

[dcopetti]

Hello,
I am having issues running Pilon on a genome where I could finish a run previously.
The command is
java -Xmx158G -jar /opt/pilon-1.18.jar --genome ../path/pdac_fu_124_pa_ctg.fa --frags aln_for_pilon_s.bam --output ../path/pdac_fu_124_pa_ctg_pilon.fa --diploid --threads 12 --changes
the reads were aligned with bowtie2, the error looks like:

Finished processing 001790F:1-65912
Processing 003645F:1-2522
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.simontuffs.onejar.Boot.run(Boot.java:340)
        at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.IllegalArgumentException: Invalid reference index -1
        at htsjdk.samtools.QueryInterval.<init>(QueryInterval.java:24)
        at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.query(SamReader.java:503)
        at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.queryOverlapping(SamReader.java:365)
        at org.broadinstitute.pilon.BamFile.process(BamFile.scala:106)
        at org.broadinstitute.pilon.GenomeRegion.processBam(GenomeRegion.scala:279)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5$$anonfun$apply$2.apply(GenomeFile.scala:113)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5$$anonfun$apply$2.apply(GenomeFile.scala:113)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5.apply(GenomeFile.scala:113)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5.apply(GenomeFile.scala:110)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:972)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
        at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:969)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.internal(Tasks.scala:159)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:443)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
        at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I can't understand how to fix it.
thanks!

[dcopetti]

Hi,
I rerun Pilon as above adding --debug --chunksize 1000000 and decreasing hte number of cores to two (if the issue was related to the memory required). It took more than a month to run, and it died at the same way, not producing any output or log file, just the stderr

Fix (1533,A,G)
Fix (32607,,G)
Fixing=0 1 35433
Fix (7344,,T)
Fixing=0 1 35434
Finished processing 002341F:1-35432
Processing 000237F-010-01:1-12490
Exception in thread "main" java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:497)
        at com.simontuffs.onejar.Boot.run(Boot.java:340)
        at com.simontuffs.onejar.Boot.main(Boot.java:166)
Caused by: java.lang.IllegalArgumentException: Invalid reference index -1
        at htsjdk.samtools.QueryInterval.<init>(QueryInterval.java:24)
        at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.query(SamReader.java:503)
        at htsjdk.samtools.SamReader$PrimitiveSamReaderToSamReaderAdapter.queryOverlapping(SamReader.java:365)
        at org.broadinstitute.pilon.BamFile.process(BamFile.scala:106)
        at org.broadinstitute.pilon.GenomeRegion.processBam(GenomeRegion.scala:279)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5$$anonfun$apply$2.apply(GenomeFile.scala:113)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5$$anonfun$apply$2.apply(GenomeFile.scala:113)
        at scala.collection.immutable.List.foreach(List.scala:381)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5.apply(GenomeFile.scala:113)
        at org.broadinstitute.pilon.GenomeFile$$anonfun$processRegions$5.apply(GenomeFile.scala:110)
        at scala.collection.Iterator$class.foreach(Iterator.scala:893)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
        at scala.collection.parallel.ParIterableLike$Foreach.leaf(ParIterableLike.scala:972)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply$mcV$sp(Tasks.scala:49)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$$anonfun$tryLeaf$1.apply(Tasks.scala:48)
        at scala.collection.parallel.Task$class.tryLeaf(Tasks.scala:51)
        at scala.collection.parallel.ParIterableLike$Foreach.tryLeaf(ParIterableLike.scala:969)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.internal(Tasks.scala:159)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.internal(Tasks.scala:443)
        at scala.collection.parallel.AdaptiveWorkStealingTasks$WrappedTask$class.compute(Tasks.scala:149)
        at scala.collection.parallel.AdaptiveWorkStealingForkJoinTasks$WrappedTask.compute(Tasks.scala:443)
        at scala.concurrent.forkjoin.RecursiveAction.exec(RecursiveAction.java:160)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

I need to get this genome polished as soon as possible, is there a way to fix my issue?
Thanks,
Dario

[w1bw]

Hi Dario,

The last log output line "Processing 000237F-010-01:1-12490" and the stack trace indicate it was trying to fetch reads mapping to that contig (000237F-010-01). Does that contig exist in the BAM header (@sq record)? If not, then that's the problem, though certainly Pilon should be more graceful about it (e.g., ignore that input contig).

[dcopetti]

Indeed, it was missing, together with 130 (!) other contigs. I took them out and had a look: they are still related to my species. So I started aligning the reads with bwa mem (default settings) instead of bowtie2: do you think it will help?
Thanks