Picard MarkDuplicates 1 out many thousands of runs broke #2063

a-frantz · 2023-09-21T17:32:52Z

Description of bug

Very odd. I've run this version of Picard and this version of MultiQC literally thousands of times without an issue. I took a look at the offending MarkDups file, and it seems well-formed to me. Also looked over the stderr logs from my picard MarkDuplicates call, and nothing seems off (although with how verbose Picard logs are, I might've missed something).

I'm going to clear the miniwdl cache for this run and see if it happens again. Either way, thought you'd want to see this. Based on the traceback looks like it should be an easy fix.

File that triggers the error

SJMB027_D.WholeGenome.MarkDuplicates.metrics.txt

MultiQC Error log

$         multiqc -v \
            --no-ansi \
            -c /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/multiqc_config_hg38.yaml \
            --file-list file_list.txt \
            -o SJMB027_D.WholeGenome.multiqc

  /// MultiQC 🔍 | v1.14

[2023-09-21 12:02:42,107] multiqc                                            [DEBUG  ]  This is MultiQC v1.14
[2023-09-21 12:02:42,116] multiqc                                            [DEBUG  ]  Loading config settings from: /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/multiqc_config_hg38.yaml
[2023-09-21 12:02:42,116] multiqc                                            [DEBUG  ]  New config: {'run_modules': ['mosdepth', 'custom_content', 'ngsderive', 'qualimap', 'picard', 'samtools', 'kraken', 'fastqc'], 'log_filesize_limit': 1000000000, 'fastqc_config': {'fastqc_theoretical_gc': 'hg38_genome'}, 'top_modules': [{'mosdepth': {'name': 'mosdepth (whole_genome)', 'path_filters': ['*.whole_genome.mosdepth*'], 'path_filters_exclude': ['*.MarkDuplicates.whole_genome.mosdepth*']}}, {'mosdepth': {'name': 'mosdepth (exon)', 'path_filters': ['*.exon.mosdepth*'], 'path_filters_exclude': ['*.MarkDuplicates.exon.mosdepth*']}}, {'mosdepth': {'name': 'mosdepth (CDS)', 'path_filters': ['*.CDS.mosdepth.*'], 'path_filters_exclude': ['*.MarkDuplicates.CDS.mosdepth*']}}, {'mosdepth': {'name': 'mosdepth (whole_genome, markdups)', 'path_filters': ['*.MarkDuplicates.whole_genome.mosdepth*']}}, {'mosdepth': {'name': 'mosdepth (exon, markdups)', 'path_filters': ['*.MarkDuplicates.exon.mosdepth*']}}, {'mosdepth': {'name': 'mosdepth (CDS, markdups)', 'path_filters': ['*.MarkDuplicates.CDS.mosdepth.*']}}], 'custom_data': {'global_PHRED_scores': {'file_format': 'tsv', 'section_name': 'Global PHRED scores', 'plot_type': 'table'}, 'orig_read_counts': {'file_format': 'tsv', 'plot_type': 'generalstats'}}}
[2023-09-21 12:02:42,117] multiqc                                            [DEBUG  ]  Added to filename patterns: [{'global_PHRED_scores': {'fn': '*.global_PHRED_scores.tsv'}, 'orig_read_counts': {'fn': '*.orig_read_count.tsv'}, 'picard/markdups': {'fn': '*.MarkDuplicates.metrics.txt'}, 'picard/alignment_metrics': {'fn': '*.CollectAlignmentSummaryMetrics.txt'}, 'picard/basedistributionbycycle': {'skip': True}, 'picard/gcbias': {'fn': '*.CollectGcBiasMetrics.*'}, 'picard/hsmetrics': {'skip': True}, 'picard/insertsize': {'fn': '*.CollectInsertSizeMetrics.txt'}, 'picard/oxogmetrics': {'skip': True}, 'picard/pcr_metrics': {'skip': True}, 'picard/quality_by_cycle': {'skip': True}, 'picard/quality_score_distribution': {'fn': '*.QualityScoreDistribution.txt'}, 'picard/quality_yield_metrics': {'skip': True}, 'picard/rnaseqmetrics': {'skip': True}, 'picard/rrbs_metrics': {'skip': True}, 'picard/sam_file_validation': {'fn': '*.ValidateSamFile.txt'}, 'picard/variant_calling_metrics': {'skip': True}, 'picard/wgs_metrics': {'skip': True}, 'kraken': {'fn': '*.kraken2.txt'}, 'ngsderive/strandedness': {'fn': '*.strandedness.tsv'}, 'ngsderive/encoding': {'fn': '*.encoding.tsv'}, 'ngsderive/readlen': {'fn': '*.readlength.tsv'}, 'ngsderive/instrument': {'fn': '*.instrument.tsv'}, 'ngsderive/junction_annotation': {'fn': '*.junction_summary.tsv'}, 'samtools/flagstat': {'fn': '*.flagstat.txt'}}]
[2023-09-21 12:02:42,119] multiqc                                            [DEBUG  ]  Added to filename clean extensions: [['.ValidateSamFile', '.whole_genome', '.exon', '.CDS', '.MarkDuplicates', '.collated', '.qualimap_rnaseq_results']]
[2023-09-21 12:02:42,120] multiqc                                            [DEBUG  ]  Command used: /usr/local/bin/multiqc -v --no-ansi -c /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/multiqc_config_hg38.yaml --file-list file_list.txt -o SJMB027_D.WholeGenome.multiqc
[2023-09-21 12:02:42,403] multiqc                                            [WARNING]  MultiQC Version v1.15 now available!
[2023-09-21 12:02:42,404] multiqc                                            [DEBUG  ]  Working dir : /mnt/miniwdl_task_container/work
[2023-09-21 12:02:42,404] multiqc                                            [DEBUG  ]  Template    : default
[2023-09-21 12:02:42,404] multiqc                                            [DEBUG  ]  Running Python 3.11.0 | packaged by conda-forge | (main, Oct 25 2022, 06:24:40) [GCC 10.4.0]
[2023-09-21 12:02:42,406] multiqc                                            [INFO   ]  Only using modules: mosdepth, custom_content, ngsderive, qualimap, picard, samtools, kraken, fastqc
[2023-09-21 12:02:42,406] multiqc                                            [DEBUG  ]  Analysing modules: mosdepth, mosdepth, mosdepth, mosdepth, mosdepth, mosdepth, custom_content, ngsderive, qualimap, picard, samtools, kraken, fastqc
[2023-09-21 12:02:42,406] multiqc                                            [DEBUG  ]  Using temporary directory for creating report: /tmp/tmponxa5dr5
[2023-09-21 12:02:42,571] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.ValidateSamFile.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.MarkDuplicates.metrics.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.flagstat.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.instrument.tsv
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.readlength.tsv
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.encoding.tsv
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.endedness.tsv
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome_fastqc.zip
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CollectAlignmentSummaryMetrics.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CollectGcBiasMetrics.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CollectGcBiasMetrics.summary.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CollectInsertSizeMetrics.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.QualityScoreDistribution.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.kraken2.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.global_PHRED_scores.tsv
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt
[2023-09-21 12:02:42,572] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt
[2023-09-21 12:02:42,573] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt
[2023-09-21 12:02:42,573] multiqc                                            [INFO   ]  Search path : /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt
[2023-09-21 12:02:42,573] multiqc                                            [DEBUG  ]  Ignored 258 search patterns as didn't match running modules.
[2023-09-21 12:02:42,573] multiqc                                            [INFO   ]  Skipping 10 file search patterns
[2023-09-21 12:02:42,573] multiqc                                            [DEBUG  ]  Skipping search patterns: picard/basedistributionbycycle, picard/hsmetrics, picard/oxogmetrics, picard/pcr_metrics, picard/quality_by_cycle, picard/quality_yield_metrics, picard/rnaseqmetrics, picard/rrbs_metrics, picard/variant_calling_metrics, picard/wgs_metrics

[2023-09-21 12:02:42,583] multiqc                                            [DEBUG  ]  Summary of files that were skipped by the search: [skipped_no_match: 1]
[2023-09-21 12:02:43,305] multiqc.plots.linegraph                            [DEBUG  ]  Using matplotlib version 3.6.2
[2023-09-21 12:02:43,307] multiqc.plots.bargraph                             [DEBUG  ]  Using matplotlib version 3.6.2
[2023-09-21 12:02:43,307] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,307] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,308] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it matched the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,311] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,311] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,311] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,311] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,311] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it matched the path_filters for 'mosdepth (whole_genome)'
[2023-09-21 12:02:43,364] multiqc.modules.mosdepth.mosdepth                  [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,419] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  Using default coverage thresholds: 1, 5, 10, 30, 50
[2023-09-21 12:02:43,421] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,421] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,421] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,422] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it matched the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,423] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,423] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it matched the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,428] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,429] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it didn't match the path_filters for 'mosdepth (exon)'
[2023-09-21 12:02:43,429] multiqc.modules.mosdepth.mosdepth                  [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,438] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  Using default coverage thresholds: 1, 5, 10, 30, 50
[2023-09-21 12:02:43,440] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,440] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,441] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,441] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,441] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it matched the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,442] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,442] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Selecting '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it matched the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,447] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it didn't match the path_filters for 'mosdepth (CDS)'
[2023-09-21 12:02:43,447] multiqc.modules.mosdepth.mosdepth                  [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,455] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  Using default coverage thresholds: 1, 5, 10, 30, 50
[2023-09-21 12:02:43,457] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,457] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,457] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,457] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,457] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,457] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,457] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,458] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it didn't match the path_filters for 'mosdepth (whole_genome, markdups)'
[2023-09-21 12:02:43,458] multiqc                                            [DEBUG  ]  No samples found: mosdepth
[2023-09-21 12:02:43,459] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,459] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,459] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it didn't match the path_filters for 'mosdepth (exon, markdups)'
[2023-09-21 12:02:43,460] multiqc                                            [DEBUG  ]  No samples found: mosdepth
[2023-09-21 12:02:43,462] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  include_contigs: []
[2023-09-21 12:02:43,462] multiqc.modules.mosdepth.mosdepth                  [DEBUG  ]  exclude_contigs: []
[2023-09-21 12:02:43,462] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,462] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,462] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/summary - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.summary.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,462] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.exon.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,463] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/region_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.CDS.mosdepth.region.dist.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,463] multiqc.modules.base_module                        [DEBUG  ]  mosdepth/global_dist - Skipping '/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.whole_genome.mosdepth.global.dist.txt' as it didn't match the path_filters for 'mosdepth (CDS, markdups)'
[2023-09-21 12:02:43,463] multiqc                                            [DEBUG  ]  No samples found: mosdepth
[2023-09-21 12:02:43,471] multiqc.modules.custom_content.custom_content      [DEBUG  ]  No samples found: custom content (orig_read_counts)
[2023-09-21 12:02:43,481] multiqc.modules.custom_content.custom_content      [INFO   ]  global_PHRED_scores: Found 1 samples (table)
[2023-09-21 12:02:43,489] multiqc.modules.ngsderive.ngsderive                [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,513] multiqc.modules.picard.picard                      [INFO   ]  Found 1 AlignmentSummaryMetrics reports
[2023-09-21 12:02:43,522] multiqc.modules.picard.picard                      [INFO   ]  Found 1 GcBiasMetrics reports
[2023-09-21 12:02:43,530] multiqc.modules.picard.picard                      [INFO   ]  Found 1 InsertSizeMetrics reports
╭───────────────── Oops! The 'picard' MultiQC module broke... ─────────────────╮
│ Please copy this log and report it at                                        │
│ https://github.com/ewels/MultiQC/issues                                      │
│ Please attach a file that triggers the error. The last file found was:       │
│ /mnt/miniwdl_task_container/work/_miniwdl_inputs/0/SJMB027_D.WholeGenome.Mar │
│ kDuplicates.metrics.txt                                                      │
│                                                                              │
│ Traceback (most recent call last):                                           │
│   File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/MarkD │
│     parsed_data[k] += float(vals[i])                                         │
│                       ^^^^^^^^^^^^^^                                         │
│ ValueError: could not convert string to float: ''                            │
│                                                                              │
│ During handling of the above exception, another exception occurred:          │
│                                                                              │
│ Traceback (most recent call last):                                           │
│   File "/usr/local/lib/python3.11/site-packages/multiqc/multiqc.py", line 65 │
│     output = mod()                                                           │
│              ^^^^^                                                           │
│   File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/picar │
│     n["MarkDuplicates"] = MarkDuplicates.parse_reports(self)                 │
│                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                 │
│   File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/MarkD │
│     parsed_data[k] += " / " + vals[i]                                        │
│ TypeError: unsupported operand type(s) for +=: 'float' and 'str'             │
│                                                                              │
╰──────────────────────────────────────────────────────────────────────────────╯
[2023-09-21 12:02:43,571] multiqc                                            [DEBUG  ]  Oops! The 'picard' MultiQC module broke...
================================================================================
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/MarkDuplicates.py", line 137, in parse_reports
    parsed_data[k] += float(vals[i])
                      ^^^^^^^^^^^^^^
ValueError: could not convert string to float: ''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/multiqc/multiqc.py", line 654, in run
    output = mod()
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/picard.py", line 85, in __init__
    n["MarkDuplicates"] = MarkDuplicates.parse_reports(self)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/multiqc/modules/picard/MarkDuplicates.py", line 139, in parse_reports
    parsed_data[k] += " / " + vals[i]
TypeError: unsupported operand type(s) for +=: 'float' and 'str'
================================================================================
[2023-09-21 12:02:43,585] multiqc.modules.samtools.samtools                  [INFO   ]  Found 1 flagstat reports
[2023-09-21 12:02:43,698] multiqc.modules.kraken.kraken                      [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,894] multiqc.modules.fastqc.fastqc                      [INFO   ]  Found 1 reports
[2023-09-21 12:02:43,935] multiqc                                            [INFO   ]  Compressing plot data
[2023-09-21 12:02:43,997] multiqc                                            [INFO   ]  Report      : SJMB027_D.WholeGenome.multiqc/multiqc_report.html
[2023-09-21 12:02:43,997] multiqc                                            [INFO   ]  Data        : SJMB027_D.WholeGenome.multiqc/multiqc_data
[2023-09-21 12:02:43,997] multiqc                                            [DEBUG  ]  Moving data file from '/tmp/tmponxa5dr5/multiqc_data' to 'SJMB027_D.WholeGenome.multiqc/multiqc_data'
[2023-09-21 12:02:44,379] multiqc                                            [INFO   ]  MultiQC complete

Before submitting

I have read the troubleshooting documentation.
I am using the latest release of MultiQC.
I have included a full MultiQC log, not truncated.
I have attached an input file (.zip if necessary) that triggers the error.

The text was updated successfully, but these errors were encountered:

a-frantz · 2023-09-21T19:22:13Z

@adthrasher did some investigation and discovered the problem. More of a bug in Picard than MultiQC.

The BAM in question has a read group in the header that doesn't have any reads associated with it. Which would be why I've only seen this bug once out of thousands of runs. Look at lib4 in the following block.

## METRICS CLASS        picard.sam.DuplicationMetrics
LIBRARY UNPAIRED_READS_EXAMINED READ_PAIRS_EXAMINED     SECONDARY_OR_SUPPLEMENTARY_RDS  UNMAPPED_READS  UNPAIRED_READ_DUPLICATES        READ_PAIR_DUPLICATES    READ_PAIR_OPTICAL_DUPLICATES    PERCENT_DUPLICATION ESTIMATED_LIBRARY_SIZE
H_LC-SJMB027-D-lib1     928724  345290987       1385995 6419426 366675  79885735        1135555 0.231577        632291457
H_LC-SJMB027-D-lib3     2125555 276658853       1025388 13545835        1150149 55403512        4515939 0.201564        633835759
H_LC-SJMB027-D-lib4     0       0       0       0       0       0       0       0

The last row has a dangling tab literal. That leaves the empty string as the value for lib4 ESTIMATED_LIBRARY_SIZE. Picard should have filled that with a 0.

ewels · 2023-09-23T07:19:14Z

Looks like a duplicate of #2061

MultiQC should never have an unhandled exception, so something to fix here either way.

vladsavelyev · 2023-09-27T22:03:04Z

@a-frantz, would you be able to attach the bad output, or even better - create a PR in https://github.com/ewels/MultiQC_TestData?

The block you copied unfortunately has all the space characters messed up, and I can't guess what was incorrect in the original file, this can't reproduce the issue. It's generally better to attach examples as files :)

a-frantz · 2023-09-28T15:40:21Z

File that triggers the error

SJMB027_D.WholeGenome.MarkDuplicates.metrics.txt

@vladsavelyev sorry about the bad code snippet. But it comes directly from the file I attached in the original bug report. That should suit your needs.

vladsavelyev · 2023-10-01T15:57:39Z

@a-frantz, thank you for the example! Managed to reproduce the issue now, and implemented a proper fix.

* Proper fix for #2063 * [automated] Update CHANGELOG.md * Fix --------- Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

ewels added the bug: module Bug in a MultiQC module label Sep 23, 2023

ewels added this to the MultiQC v1.17 milestone Sep 23, 2023

This was referenced Sep 23, 2023

picard module - TypeError #2038

Closed

Oops! The 'picard' MultiQC module broke... #2061

Closed

vladsavelyev mentioned this issue Sep 27, 2023

Picard: fix parsing mixed strings/numbers, account for trailing tab #2083

Merged

vladsavelyev added a commit to MultiQC/test-data that referenced this issue Oct 1, 2023

Add repro for MultiQC/MultiQC#2063

1c659f4

vladsavelyev mentioned this issue Oct 1, 2023

Picard MarkDuplicates: example with missing 0 MultiQC/test-data#282

Merged

vladsavelyev added a commit to MultiQC/test-data that referenced this issue Oct 1, 2023

Add repro for MultiQC/MultiQC#2063 (#282)

1df415f

vladsavelyev added a commit that referenced this issue Oct 1, 2023

Proper fix for #2063

c080b18

vladsavelyev mentioned this issue Oct 1, 2023

Picard: MarkDuplicates: account for missing trailing 0 #2094

Merged

ewels closed this as completed in #2094 Oct 3, 2023

ewels pushed a commit that referenced this issue Oct 3, 2023

Picard: MarkDuplicates: account for missing trailing 0 (#2094)

ac93dc3

* Proper fix for #2063 * [automated] Update CHANGELOG.md * Fix --------- Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Picard MarkDuplicates 1 out many thousands of runs broke #2063

Picard MarkDuplicates 1 out many thousands of runs broke #2063

a-frantz commented Sep 21, 2023 •

edited

a-frantz commented Sep 21, 2023

ewels commented Sep 23, 2023

vladsavelyev commented Sep 27, 2023

a-frantz commented Sep 28, 2023

File that triggers the error

vladsavelyev commented Oct 1, 2023 •

edited

Picard MarkDuplicates 1 out many thousands of runs broke #2063

Picard MarkDuplicates 1 out many thousands of runs broke #2063

Comments

a-frantz commented Sep 21, 2023 • edited

Description of bug

File that triggers the error

MultiQC Error log

Before submitting

a-frantz commented Sep 21, 2023

ewels commented Sep 23, 2023

vladsavelyev commented Sep 27, 2023

a-frantz commented Sep 28, 2023

File that triggers the error

vladsavelyev commented Oct 1, 2023 • edited

a-frantz commented Sep 21, 2023 •

edited

vladsavelyev commented Oct 1, 2023 •

edited