Multiqc error in the latest development?

[naumenko-sa]

Hello!

Thanks for the excellent pipeline!

After the recent -u development upgrade (I have also upgraded tools), I run into multiqc error:

Error:

[2018-06-12T15:01Z] multiprocessing: qsignature_summary
[2018-06-12T15:01Z] multiprocessing: multiqc_summary
[2018-06-12T15:01Z] summarize metrics
[2018-06-12T15:01Z] Run multiqc
[2018-06-12T15:02Z] [INFO   ]         multiqc : This is MultiQC v1.5.dev0
[2018-06-12T15:02Z] [INFO   ]         multiqc : Template    : default
[2018-06-12T15:02Z] Searching 40 files..
[2018-06-12T15:02Z] [INFO   ]     verifybamid : Found 2 reports
[2018-06-12T15:02Z] [ERROR  ]         multiqc : Oops! The 'verifybamid' MultiQC module broke...
[2018-06-12T15:02Z]   Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
[2018-06-12T15:02Z]   If possible, please include a log file that triggers the error - the last file found was:
[2018-06-12T15:02Z]     /hpf/largeprojects/ccmbio/naumenko/validation/nist/work/qc/nist_NA12878/contamination/nist_NA12878-verifybamid.selfSM
[2018-06-12T15:02Z] ============================================================
[2018-06-12T15:02Z] Module verifybamid raised an exception: Traceback (most recent call last):
[2018-06-12T15:02Z]   File "/hpf/largeprojects/ccmbio/naumenko/tools/bcbio/anaconda/bin/multiqc", line 427, in multiqc
[2018-06-12T15:02Z]     output = mod()
[2018-06-12T15:02Z]   File "/hpf/largeprojects/ccmbio/naumenko/tools/bcbio/anaconda/lib/python2.7/site-packages/multiqc/modules/verifybamid/verifybamid.py", line 75, in __init__
[2018-06-12T15:02Z]     self.verifybamid_table()
[2018-06-12T15:02Z]   File "/hpf/largeprojects/ccmbio/naumenko/tools/bcbio/anaconda/lib/python2.7/site-packages/multiqc/modules/verifybamid/verifybamid.py", line 155, in verifybamid_table
[2018-06-12T15:02Z]     'hidden': all( [ s['RG'] == 'ALL' for s in self.verifybamid_data.values() ] )
[2018-06-12T15:02Z] KeyError: 'RG'
[2018-06-12T15:02Z] ============================================================
[2018-06-12T15:02Z] [INFO   ]           bcbio : Found 1 reports
[2018-06-12T15:02Z] [INFO   ]        samtools : Found 1 stats reports
[2018-06-12T15:02Z] [INFO   ]        samtools : Found 1 idxstats reports
[2018-06-12T15:02Z] [INFO   ]        bcftools : Found 1 stats reports
[2018-06-12T15:02Z] [INFO   ]        qualimap : Found 1 BamQC reports
[2018-06-12T15:02Z] [INFO   ]          fastqc : Found 1 reports
[2018-06-12T15:02Z] [INFO   ]         multiqc : Compressing plot data
[2018-06-12T15:02Z] [INFO   ]         multiqc : Report      : ../../bcbiotx/tmpM4IAk3/multiqc_report.html
[2018-06-12T15:02Z] [INFO   ]         multiqc : Data        : ../../bcbiotx/tmpM4IAk3/multiqc_data
[2018-06-12T15:02Z] [INFO   ]         multiqc : MultiQC complete
[2018-06-12T15:02Z] Uncaught exception occurred

yaml:

details:
- algorithm:
    aligner: bwa
    effects: vep
    effects_transcripts: all
    ensemble:
      numpass: 2
    mark_duplicates: true
    realign: false
    recalibrate: false
    save_diskspace: true
    tools_on:
    - qualimap
    - vep_splicesite_annotations
    - noalt_calling
    validate: giab-NA12878/truth_small_variants.vcf.gz
    validate_regions: giab-NA12878/truth_regions.bed
    variant_regions: /home/naumenko/work/validation/nist/input/region.bed
    variantcaller:
    - gatk-haplotype
    - samtools
    - freebayes
    - platypus
  analysis: variant2
  description: nist_NA12878
  files:
  - /hpf/largeprojects/ccmbio/naumenko/validation/nist/input/nist_NA12878.bam
  genome_build: GRCh37
  metadata:
    batch: nist
fc_name: gatk4
resources:
  default:
    cores: 1
    jvm_opts:
    - -Xms75m
    - -Xmx700m
    memory: 1G
upload:
  dir: ../final

programs (multiqc is not on the list):

bcbio-nextgen,1.1.0a0-7a82611
bcbio-variation,0.2.6

Could you please look into this?
Thanks!
Sergey

[chapmanb]

Sergey;
Sorry about the issue. The verifyBamID contamination calculation is new and we must be hitting some case I haven't seen. Would you be able to share the problem file?

/hpf/largeprojects/ccmbio/naumenko/validation/nist/work/qc/nist_NA12878/contamination/nist_NA12878-verifybamid.selfSM

This should have a 'RG' column so I'm not sure what is going on, but looking at that file might help us come up with a a workaround. Thanks much for the help debugging.

[naumenko-sa]

Sure

#SEQ_ID	RG	CHIP_ID	#SNPS	#READS	AVG_DP	FREEMIX	FREELK1	FREELK0	FREE_RH	FREE_RA	CHIPMIX	CHIPLK1	CHIPLK0	CHIP_RH	CHIP_RA	DPREF	RDPHET	RDPALT
nist_NA12878
@PG	NA	NA	100000	123	11.1818	0.5	-0	-0	NA	NA	NA	NA	NA	NA	NA	NA	NA	NA

so RG=NA,
I've tried to set up lane=1 in the yaml config, it did not help.

It looks like there is an extra line with nist_NA12878 between the header and the data.
When I remove this line, multiqc passes.

But is exactly the output produced by verifybamid2.

However, multiqc promised to work with verifymabid1.1.3 only
http://multiqc.info/docs/#verifybamid

Should I try to ask multiqc to support verifybam2 or would you prefer to downgrade to 1.1.3 in bcbio?

[chapmanb]

Sergey;
Sorry about the issue and thanks for the detailed example. I traced this back to an apparent bug in parsing the sample name when the SM tag is the last item in the RG line. I posted a report for VerifyBamID2 to fix and also pushed a workaround to bcbio to avoid the issue for now. This appears to be an oversight, rather than any fundamental incompatibility between verifybamid2 and 1, so hopefully should work cleanly going forward. Please let us know if you run into any other problems.