Read in multi-sample bcftools stats file

[WimSpee]

Is your feature request related to a problem? Please describe.
For large variant callings files (e.g. 500GB+, 500M variants, 150+ samples) it takes very long to create a bcftools stats file for each sample. For each sample it is then needed to parse the whole 500GB file.

bcbio/bcbio-nextgen#2336

An alternative is to create a single multi-sample bcftools stats file, for which the 500GB VCF is only read once.

-s, --samples <list> list of samples for sample stats, "-" to include all samples

The resulting file has a PSC and PSI blocks with per sample information.

# PSC, Per-sample counts. Note that the ref/het/hom counts include only SNPs, for indels see PSI. The rest include both SNPs and indels.
# PSC   [2]id   [3]sample       [4]nRefHom      [5]nNonRefHom   [6]nHets        [7]nTransitions [8]nTransversions       [9]nIndels      [10]average depth       [11]nSingletons [12]nHapRef     [13]nHapAlt     [14]nMissing

# PSI, Per-Sample Indels
# PSI   [2]id   [3]sample       [4]in-frame     [5]out-frame    [6]not applicable       [7]out/(in+out) ratio   [8]nHets        [9]nAA

The rest of the information in the stats file is not per sample QC information, but aggregated QC information for the whole variant calling.

The information outside of the PSC and PSI blocks can be discarded, because there is not place to put QC information in the multiQC reports that is not on the level of a single sample?

MultiQC currently does not recognize (the samples in) the multi-sample VCF.
It takes the name of the file as the sample name, and I think it also does not use the PSC or PSI information, but reads in the analysis level information.

Describe the solution you'd like
Read in the PSC and PSI information from multi-sample BCFTools stats files. Don't use the information outside of the PSC and PSI blocks, or show it as analysis level information.

Describe alternatives you've considered
Keep creating (in bcbio) and reading in (in MultiQC) single sample bcftools stats files.
But make it more efficient to do this, by using a multi-sample BCF file, or first splitting in 1 pass the multi-sample VCF file to single sample VCF files. See the linked bcbio ticket.

Thank you.

[ewels]

Hi @WimSpee,

Sounds sensible. I only have one example output file from bcftools currently and it doesn't have PSC or PSI blocks. Please could you submit a PR to the test data repo with one or more example output files?

In terms of implementation, data in MultiQC is typically sample-level but we can plot whatever we want really. I guess there could be a flag where if PSC or PSI is found in any of the files parsed a warning box is shown above the plots alerting the user that (at least some) of the data shown is at an aggregate level. If those sections are found then we ditch everything that is currently put into the General Statistics table and instead use per-sample numbers from those sections.

Do we have data corresponding to the current table columns? Vars / SNP / Indel / Ts/Tv / MNP

Phil

[ewels]

Example data available in duplicate issue #1272

[ewels]

Done in #1271 - if you get a chance please give it a test!