Cigar issue with no M or N between two I's when using --no-end-flt flag

[gavinkurgan]

Hey Heng!

Working with Minimap2 I think I came across a weird edge case the other day that may be producing buggy behavior. This is using version 2.16. Given the following read:

AAGCTCTAATGATTGCTGAATGGAATATCATTATCTAAAATCTAGACAGAAGCCCTCTCAGAAACCACTTTGTGATATCTGCATTCAAGTCACAGAGTGAACATTCTGCTTTCTTTAGATAATGCTGAAAGACTTTTTGTAGTGTCTGGAAGTGGACATTTGGAGGCTTGATGCTCTTGGAAAAAGGGAATGTCTTCATAAAATCCAGGAGCAGAAGCATCTCTCTTGCAAAACTTGTTTGTGATGTGTGTACCAGCTAAAGGAGTTGAACATTTCTATTTGATAGAGCAGTTTTAAACACCTTGGGAAATGCAAGGGATATTTGGATAGCTTGGAGGATTTTGGAAGCGGGAATCTAAATAAAAGGTAGACAGTATTCTCAGTAGTCTTTTCTGATGTCTACTTCAACTCATGATCTCTATAGAGCAGGTTTGAAACACTCTTCTGGAGTATCTGATGATTTGAGCGATGCCACGGAAAAGTAAATATCTTCCAAAACGAGACAGAAGGATTCCTTGAGAGACAAGTTTGTGATGTGTGTAATCAGCTAACAGAAGGAACCTTTCTTTTTACAGAGCAGCTTTGAAAACTCTATTTTAGATTCTGCAGTGGATATTTAGATTGTATATTGATATCGTTGGAAAAGGGAATATCAGGTCATACAAAATCTGACAGAAGCATCTTTACAAACTTTAGATGTGTCCTAACTAATTGAGTTGAACCTTTGAAGCAGCAGTTTGAAACACTCTTTGTAGAAACTGTGTGGATATTTGGATAGTTTTGTCAGATTTCGTTGAAATGTGGAATATTAATATCTAAAATCTAGACAGAAGCCCTCTAGAAACTACTTTGTGATGATTTGCATTCAAGTCACAGAGTTGAACATTCTGCTTAGAGCACGTTGGAAACACTCTTTGTAGTGTCTGAAGTGGACATTTGAGCGTTTTGATGCCTTTAGAAAAGGAATGTCTTCCGGCAAAAAAACCTAGACAGAAGCATTCTCAGAAACTTGCTGTGGAGGTGTGTACCCAGCTAAAGGAGTTGAAACATTTCTATTGATAGAGCAGTTTTGAAACACTTTGTGGAAAATGCAAGTGGATATTTGGATAGCTTGGAGGATTTCAAGCCAGCGGAATCTAAATAAAGGTAGAGACTTCTCAGAAATTCTTTTGATAATTTGCATTCAACCCATAGAGCTGAACATTCCTTTCATAGAGCAGGTTTGAAACACTCTTCTGGGAGTATCTGAAGGGAAGCATTTGGAGTGCTTTGATTCCCTATTAGAAAGCAACATCACAAAACGAAAGTATTGTAAGGATTCTGAGAGACAAGTTTGCATGTGTACTCAGCCAACAGAGTGGAACCTTTATAGTATAGCTGAAACTTATTGTGGATTCTTGCAAATGGATATTTGTAATTGTTTTAATATTATCGTTGGAAAAGGAATATGTATCACAAAATCTAGACAGAAGCATTCTCACAAACTTCTTTCAGGAGCGTGTATCCCCAACTAAATAGAGCGAAACTTTTTCTTTTTGATGCAGCAGTTTGAAACACTGTTTTGTAGAAACAAGTGGATACTTGGATAGCTCTAACGATTTCAATGGAAATGGGAACATCATCTAAATCTAGAAGCACTATTAGAAACCATTTATATCTGCATTCAAGCTTCAGAGTTGAGCATTCCTCATTTTGAGCACGTTTAAAGTTTGACTTTTGGAAGAATCTGAAGTGATGATTTTAGCGCTTTGATGCCTTAGAAAAGGGAAACGTCTTCCAATAAAGCCAGATAGAAGATTCTCAGAAACTTGTTTGTGATGTGTGTACCTAAACTAAAAGAGTTGAACTTTCTAAGATAGAGCAGTTTTGAAACACTCTTTTGTAGATTCTGCAAGTGGATATTTGGATTGTTTGAGGATTTCAAGTTGTATGGAATTCGCATAAAAACTACAGCATTCTAGAAATTTCTTTCTGATTTCAACTCATAGAGATGAAATGATGGCCTTTCATAGAGCAGGTTTGAAACACTCTTAGTTAGAAGTGGCCATTTTGATCTCCTGATTTCATGGTGAAAAGTAGCATCTTCCTACAAAAGACAAAGCATTCTTAAACTTGTTGGGTGATATAAACCTGAACCAATAGATTGAACTTGCCATTTGATAGAGAGTAGTTTTGAAACACTTTGTGGAATCTGCAAGTGGATATTGATATCGGAGGATTTCGTTGGAAGTGGGAATTCAAATAAAAGGTGATATAGCATTCTAGAAATTTTCTGAGGTCGTATTCAACCCATAGAGTTGAACATTCCTCTCATAGGGCAGGTTTGAAGACTCTTTCTCAGTATCTGATGTGACATTTGGAGCGCTTTGATGCCTAATGGTGAAAAAGTAAATATCTTCCCATAAAATCTAGACAGAAGGATTCTGAGAAACAAGTTTGTATGTGTACTCAGCTATTGGAGGAACCTCTGTTTTGATGCAGCAGTTTGGAAACACTCTAGAAACTAGGTGATATTTGATAGCTCTTAATACTGTTGGAAACGAATATCATCATCTGGGTAAATCTAGACAGAAGTTTCCCCAGAAACTAGCTTTAGATATATCTGCATTCAAGTCACAGAGTTGAAATATTCATTTAGAGCACGTTTGTTAATTTTTGTAGTGGTCTGGTGGCATTTGGAGCGCTTTGACAGTTTTTGGTGAAAAAGGGAATGTCTTCCCATAAAACTAGACAGAAGCATTTCTCAGAAACTGGTTTAGATGTAAAATCTAGAAGAAGGAGTTAGAACATTTTACTGATAGAGCAGTTTTGAAAGACTCTTTGTGGGAAAAGTGGATATTGATATAGCTTGGAGGATTTCTGTTATGGGAATTCAAATAAAGGTAGACAGGAGACTTCTAGCTTTTCTGATGTCTGCATTCAACTCCAGAGTTGAAGATTCCTCATAGAGAAAAGCAGGTTTGAAAGACTTTCTTGAGATTCTGAGGACATTTGGAGCGCTTTGATGCCTACAGAAAAGTAAATATCTTCCATAAAAACTAGACAGAAGGGATCTTGGAGAGAAGTTTGTGATGTGTACTAGCTAACAGGAACCTTCTTTACAGAGCAGCTTGAAACTCTAGTTTTGTGGATTCTGCAAATGGATATTTTAGATTGCTTTAAATGGATATCGTTGAAAAGGGACATCGTCATACAAAATCTGACAGAAGCATTCTCACAAACTTTGTGATGTGTGTCCTCAACTAAAGAGTTGAACCTTTGATGTGCAGCAGTTTGGAAACACTTTGGTAGAAACTGTAAGTGGATATTTGATATAGCTCGAACAGATTTGTTGGAAATGGGAATATCATCATCTAAAATTCAGAAGCACCCTATTAGAAACTACTTGGTGGATATCTACTCAAGTCAAAGAGTTGAACATTCCCCATTTTGAGCACGTTTGAAACACCTTTGGAAGAATCTGGAAGTATATTTGGAGCGCTTTACGCCTTTGGTGAAAAGGAAACGGCCTCAATAAAGCCGTATAGAAGCATTTCTCAGAACTGTTTTGATGTAAAATTAAACTAAAAGAGTTGAACCTTTCTATTGATAGAGCAGTTGGAAACACTCTTCTTTGTGGATTCTGCAAGGATATTTGATTAAGTTTTGAGGATTTCGTTTGGAAGCGGGAATCTGTGATAACAACTAGACAGCAGTATCTTGGTAGCCTCTTCGGATATTTCCATTCAACCCATAGAGATTAACAGGGCTTCATAGAGCAGGTCGAAACACTCTTTTGTAGTTACAGGGTGATATTTCGATCGCTTCTTGGATGCCTATGGTGAAAAAAGGAAATATCTTCCAAAAAAAAGAGATGAAGCATTCTAGAAACTATGGTGATATGTCCTCAAATCTAACAGAGTTGAACTCATGCATTGATAGAGAGTGCTTGAACACTTTTGTGGAACCTGCAAGTGGATATTTGATATTTGGGAGGATTCTCATGCTAAAGTGGACCAAATAAAGGCAGACAGCAGCATTCTCAGAATTTCTTTTCGATTCTGTATCTAACCCCATAGAGTTGAACATTCTCTTCACATAGGCAGGCTTGAAATACTTCTGTAGTATCTGGATGAGGACATTTGAGCGCTCTTTGATGCCTACGGTGAAAAGTAAATATCTTCCCAGCAAAACAGACACAGAAGGATTTGAGAAATAAGTTTTGTGATGTGTACTCACAGAGTGGAACCCTGTTTTGATGCAGTTTGAAACACCTTTTGTGAAACTGTGGGTGGATATTTGGATAGTCTAATGATTTCTGTTGGAAACGGAATATCATCTAAAATCTAGACAGAAGCCCTCTCAGAAACTACTTTGTGAGCATCTGATCAAGCCGTTAGTGAAGTTAATTATTCAGTTTTCTCTTGTATACGTTGGAAAATAATTTTGTAGTGTCTGGAAGTGAGCATTGGAGCGCTTTGATGCCTTTTGCAGGCGGAATGTCTCCTCATAAAAACTCAGACAGAAGCATTCTCAGAAATTTGTTTGTATGTGTGTACCCAGCTAAAGGAGTTGGGAAATATTTCTATTGATAGAGCAGCTGAAACACTTTTTTGTGGAAAAGGATATTTGGATGAATGGAGGATTTTGCTGGAAGTGTATTCAAATAAATAGATAGAGCATTCAGAAATCTTCCTTTCTGATACTCATTCAACGAGTTGAAGATTCCTTTCATAGACAGGTTGGAAACACTCTTCTGGTATCTGATGTGACATTTGAGCGCTTTGATGCCTAAATGTGAAAAGTAAATATCTTCCCATAAAACTGGTGACAAGCATCTGGTGAGACAAGTTTTATGTGTACCAGCTAATTAGAGGGAACCTTCTTTACAGAGAAGAACTCATTTTTAGATCTGGCAAATGGAGATGATTGTTTAAGGATATCGTTGAAAGGAATATCGCTATACAAAATCTGGACAGTTAATTTCACAAACTTCTTGCATGTGTCCTCAACTAAATATTAGAACCTTGTTTATAGTTTGGAAACACTTTAGAAACGGGTGGATATTTTGGATAGTCAACGATTTCTGTTGGAAACGGGAATATCATCATCTAAAACGTAGACAGAAGCACTAGGCTAGGAAACTACTTGGTGATATCTCCGTATCTAAGTCAAAGAGTTGAACATTCCCTCTTTACTTTGGAGCATCGTTTAGAAACACTCTCAGTATCTGGAAGTGACATTTGGAGCGCTTTGAATGCCTTTGTGAAAAGGAAACGTCTTCCAATAAAAAGCTAGACAGAAGCATTTCGAAACTTGTTCTTGATGTGTACTGAATCAAAAGAGTTGAACTTTCTATTGATAGAGCAGTTTTGAAACACTCTTTTGTGGATTCTGTAAGTGGATATTTGATTATTGAGGATTTCTGCTAGTGGAATTGTATAACAACTAGACAGCAGCATTCCTAGAAATTTCTTTTCGATATTTTCATTCAACTCAGAGATGAACATGGCCTTTCAGAGCAGGTTTTGAAACACTTGTAGTTTGTAGCGTGGAGGTTTTGATCGCCTTAATGCTTCATGGTAGAAAAGCAAGATCTTCCATAAAAATAGACAGAAGCATTCCTAGAAACTTGTTATATATGTGTCCCAACTAACAGGAGTTGAACTTACATATAGAGAGCAGTTTGAAACACTCTTTTTCCTGAATCTGTAAGGGATATTTTGATAGCTTGGAGGATTTTCGTTAAAGTGAATCAAATAAAGGTAGATAGCATTCTCAGAAATTTGATCTCTGCATTCAACCCAGAGTTGAACATCCCTTTTCATAGGGTAGGTTGAAATACTCTTTCTGAGTATCTGATGGAAGGCTTGGAGCGGCTGTGATGCAGTGGGGGCTTATCTTCCCATAAACGAGACAGAAGGATTCTGAAACAAGTTTGTGATGTGTACCTCAGCTAGGTGGAACCGTTTTGATGGCAGTTTGGAAACACTCTTTTGTAGAAACTGTGGTGGGATATTTGGATAGCTGAATGATTTCATGGAAACGGAATATCATATCTAAAATTTTAGAGAAGCCCTCTCAGAAACTATTTTGTGATATCTGGGACTAAGTCACAGAGTTGAATATCATCTGCTTTATTAAAGACTTTTTGTAGTGTCTAAGTGGACATTTGGAGGCTTTGATGCCTTTTGTAGAAAAGGAACAGTCTTCCCAGTAAAACTAGATAGAAGCATTCTCAGAAACTGTTTGTGATGTGTACCAGTTAAAGGAGTTGAACATTTCCATTGATAGAGCAGTTTTTGAAACACTCTTTTGTGGAAAATGTAAGTGAGGTTATTTGAGTAGCTGGAGATTTCTAAGAAGGGAATTCAAATAAAGTTGTAGACAGCAGCATTCTCAGAAATTTTCGATATCTGTATTCAACCTAGACAGATTCCCTTCATAGAGTAGGTTTGAAACACTTTCTGGTATCTGAAGGGACACGAGCGCTTTATGCCACGGGTGAAAAGCAACATCTTCCTAGTTAAAACGAGGTATTGGTGAAGGATTCTGAGAGACAAGTTTTGTGAGGTACCAGCCACAGAGTGAACCTTTCTTTAATATACAGCTTTGAAACCTTCTATTTTGTGGATTCTGCAAATGATATTTAGATTGGTTTAATGATATCGTTGGAAAAGGAATATCGTCATACAAAATCTTACAGAAGCTTTCACAAACTTTTGATGTATCCTCAACTAACAGAGTTGAACCTTTCTTTGAGGCAGTTTGGAAACACTCTTTAGAAACTGGGTGATAATATTAGCTGTAAATGATTAAGTTGGAAACGGGAATACTGCCATCTAAAATTTAGACAGAAGCACGATGGAAACTACTGGTGATATCTGTATTCAAGCCAAAGAGTTGAATACCCCTTACTGAGCACGTTGAAACACTCTTTGGAAGAATTGGAAGTGGGACATTTGGAGCGCTTTGATGCCTTTTGTGAAAAGGAAACGCCCAATAAAAGCTAGGACAGAAGCATTCTCAGAAACTTGTCAGATGTGTGTACTCAACTAAAGAGTTGAACCTTTCATTGATAGAGCAGTTTTGAAACACTTTTGTAGATTCTGCAGTGGATATTTTGGATTGCTTTGAGGATTGCTGAAATGGAATCTGTATAAAAACTAGACAGCAGACATTCCCAGAAATTTTTCGATATTTCATTCAACTCAGAGATGAACATGGCCTTTCAGAGCAGGTTTGAAACACTCTTTGTAGTTTTGTAGCGGACATTTGATCGCTTCCTTTGACGCCTACGTAGAAAAGGAAATATCTACCAAGCAAAGACAGACAGAAGCATTCTGTGAAACTTGTTAAGATATGTGTCCTCAACTAATAGAGTTGAACTTGCCATTGATATAGTTTTGAAACACTCTTTTCCTGAATCTGCAAGTGACATTTGATAGCTGAGATTTGAAGGGAATTCAAATAAAGTAGACAGCAGCATTCTAAGAAATTTCTGATCTCTGCATTCAACCTAGAGCAAAGACTCTTCATAGGCAGGTTTGAACACTTTCAGGTCTGATGGGACATTTGGAGCGCTGATGCCATGGTGAAAAGTAAATATCTTCCCAAAACGAGACAGAAGGATTCTGAGAAATAAGCTTAGAGTGGTGTACTCAGCTAACAGAGTGGAACCCCTTTGATGCAGCAGTTTGAAACACTCTTTGTAGAAACTGTATGAGTCATTTGGATAGCTCTTAATGATTTCTTTTGGAAACGGAATATCATCATTCAAAATCTGGGACAGAAGCCCCCAGAAACTACTTTAGATATCTGCATTCAAGTCACAGAGTTGAACATTTTGCTTTTAGAGCACGTTGGAAACACTTTGTAGTGTCTGAAGTGTGACATTTGTTTTGATGTTTTAGAAAAGGGAATGTCTTCCCATAAAACTAGAGCGAAGTATTCTAGAAACTTGTCAGGGATAAAACGCTAAAGGAGTTGAATATTTCGGAAGATACAGAAGCGGAAAATTTCTTTTGTAGTGTAAGTGGACATGGATAGTTTGGAGGATTTCTGCTGAAGTAGGAATTCTAAATAAAAGGATAGTAGCATTCTCAGAAATTTCTTTCTGATGTCTGCATCTAAATCAGAGTTGAAGATTCCTGTTAGAGCAGGCTTAAACACCTTTCTGGTATCTGGATGTGACATTTGAAAGCTTTGATGCCAGAAAAGTAAATATCTCTCTCCATAAAAACGAGAGCAGAAGGGATTCTCAGAAACAAGTTTGTGATGTGTGTACCAAAGCAACAGAGTGAACCTCTTTGATGCAGTTTGGAAACAATTTTGTAGAAACTGTAGTGGATATTTATAGTTTTAAATAGATTTCAGGTTAAAACGGAATATCATCATCTAAAATCTAGAGAGAGTTCCTCTCAGAAACTACTTTGTAGATATCTGGGATTCAAGCCTGAGTTGAAACATCAAGTTTCTTAGAGCACAGTTTGGAAACACTTCTTTGTAGTGTCTGAAGTGGACATTTGGGAGCGCTTTGATGCCTTTGGTGAAAAAGGAATGTCTCTCCTAGGCAAAACTATAGAAGCATTCTCAGAAACCTGTTTGTGATGTGTGTACCCAGCCAAAGGAGTTGAACATTTTTGCATTATAGAGAAGCGAAAACTTTTTTTGGGAAATGGTGGAGATTTTGGGATAGCTTGGGAGCATTTCGTTGGAAGTGGAATTAAATAAAGGTGGACAGCAGCATTCTGAAATTTTCTCTGATGTCCGTATCCAACTCATA

After mapping to the human genome (hg38p12 - indexed with defaults) using the following command:

mininmap2 -a -c --MD --no-end-flt genome.mmi reads.fq | samtools -Shb> problematic.bam

An alignment is produced which fails many picard tools operations (such as SamToFastq) with the following error:

Exception in thread "main" htsjdk.samtools.SAMFormatException: SAM validation error: WARNING: Read name 72a6132c-6173-4e77-9ab9-202019f711dc, No M or N operator between pair of I operators in CIGAR
        at htsjdk.samtools.SAMUtils.processValidationErrors(SAMUtils.java:454)
        at htsjdk.samtools.BAMRecord.getCigar(BAMRecord.java:287)
        at htsjdk.samtools.SAMRecord.getAlignmentEnd(SAMRecord.java:610)
        at htsjdk.samtools.SAMRecord.computeIndexingBin(SAMRecord.java:1588)
        at htsjdk.samtools.SAMRecord.isIndexingBinNotEqualsComputedBin(SAMRecord.java:2171)
        at htsjdk.samtools.SAMRecord.isValid(SAMRecord.java:2129)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:811)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:797)
        at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:765)
        at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:576)
        at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:548)
        at picard.sam.SamToFastq.doWork(SamToFastq.java:191)
        at picard.cmdline.CommandLineProgram.instanceMain(CommandLineProgram.java:277)
        at picard.cmdline.PicardCommandLine.instanceMain(PicardCommandLine.java:103)
        at picard.cmdline.PicardCommandLine.main(PicardCommandLine.java:113)

From what I can tell it seems to be due to the following motif in the cigar:

...I1D1I3...

However, if you take away the --no-end-flt flag this is no longer an issue. Any insight you have on whether this is a bug or not would be great!

Thanks,
Gavin

[lh3]

This is a known bug. I will fix it before the next release. For now, you can reduce the stringency level of picard.

[lh3]

Fixed at the github HEAD. Let me know if it has issues. Thanks!

[gavinkurgan]

Great, thanks Heng!

[SHuang-Broad]

@lh3 ,

Sorry to report the bad news, but I am still seeing this behavior after the latest release.
I've attached the bam with the relevant SAM records.
The header contains how the alignments were produced.

The strange part of CIGARs are highlighted in the tag YG:Z:...

insdel.neighbors.zip

[mebbert]

Hello @lh3:

We're still seeing this issue with minimap2 v2.26. I doubt the reference would affect this, but we're currently getting it when aligning to CHM13.

See below for stack trace. This is coming from a tool we use to identify dark regions, but it's built on htsjdk. We'd really like to avoid lowering the validation stringency.

Happy to provide any files that would help debug. We're just aligning one of the PacBio 1KG samples (NA19384) to CHM13

htsjdk.samtools.SAMFormatException: SAM validation error: WARNING::ADJACENT_INDEL_IN_CIGAR:Read name m64119_210621_120219/71828434/ccs, No M or N operator between pair of I operators in CIGAR
    at htsjdk.samtools.SAMUtils.processValidationErrors(SAMUtils.java:458)
    at htsjdk.samtools.BAMRecord.getCigar(BAMRecord.java:284)
    at htsjdk.samtools.SAMRecord.isValid(SAMRecord.java:2234)
    at htsjdk.samtools.BAMFileReader$BAMFileIterator.advance(BAMFileReader.java:848)
    at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:834)
    at htsjdk.samtools.BAMFileReader$BAMFileIterator.next(BAMFileReader.java:802)
    at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.advance(BAMFileReader.java:1058)
    at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1048)
    at htsjdk.samtools.BAMFileReader$BAMQueryFilteringIterator.next(BAMFileReader.java:1012)
    at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:591)
    at htsjdk.samtools.SamReader$AssertingIterator.next(SamReader.java:570)
    at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
    at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
    at htsjdk.samtools.filter.FilteringSamIterator.getNextRecord(FilteringSamIterator.java:135)
    at htsjdk.samtools.filter.FilteringSamIterator.next(FilteringSamIterator.java:108)
    at htsjdk.samtools.filter.FilteringSamIterator.next(FilteringSamIterator.java:46)
    at htsjdk.samtools.util.PeekableIterator.advance(PeekableIterator.java:71)
    at htsjdk.samtools.util.PeekableIterator.next(PeekableIterator.java:57)
    at htsjdk.samtools.util.AbstractLocusIterator.next(AbstractLocusIterator.java:312)
    at htsjdk.samtools.util.AbstractLocusIterator.hasNext(AbstractLocusIterator.java:234)
    at ebbertLab.drf.DarkRegionFinder.startWalkingByLocus(DarkRegionFinder.java:170)
    at ebbertLab.drf.DarkRegionFinderEngine.findDarkRegions(DarkRegionFinderEngine.java:308)
    at ebbertLab.drf.DarkRegionFinderEngine.main(DarkRegionFinderEngine.java:43)

Can you help?

Thanks!