Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

bclconvert: incorrect stats for single-read data #1697

Closed
ajeffs opened this issue May 31, 2022 · 3 comments · Fixed by #1701
Closed

bclconvert: incorrect stats for single-read data #1697

ajeffs opened this issue May 31, 2022 · 3 comments · Fixed by #1701
Labels
bug: module Bug in a MultiQC module

Comments

@ajeffs
Copy link

ajeffs commented May 31, 2022

Description of bug

bcl-convert module in v1.13.dev0 reports incorrect Lane and Sample Statistics for single-read data:

  1. "Yield (Mb)" is shown as twice the actual yield.
  2. The actual "Yield (Mb)" is used as "Bases (Mb) >=Q30 (PF)".
  3. "% Bases >= Q30 (PF)" is therefore half the correct value.

It appears the bcl-convert module is incorrectly calculating the above instead of pulling the correct data directly from the bcl-convert report file (Quality_Metrics.csv etc.)?

The bcl-convert module shows correct values for paired-end data.

Example for Lane Stats below (1x51 bp; Sample Stats behaves the same):

bcl-convert:
bcl-convert_multiqc_sr

bcl2fastq (correct values):
bcl2fastq_multiqc_sr

File that triggers the error

No response

MultiQC Error log

$ multiqc . ../../Stats -e snippy -i "OG6761 QC report" -n "OG6761-multiqc-report.html"
 /// MultiQC 🔍 | v1.13.dev0

|           multiqc | Report title: OG6761 QC report
|           multiqc | Excluding modules 'snippy'
|           multiqc | Search path : /mnt/HCS/aozan/fastq/220321_VH00684_22_AAAM322HV/bcl2fastq/OG6761/qc
|           multiqc | Search path : /mnt/HCS/aozan/fastq/220321_VH00684_22_AAAM322HV/bcl2fastq/Stats
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 152/152  
|            fastqc | Found 72 reports
|           multiqc | Compressing plot data
|           multiqc | Report      : OG6761-multiqc-report.html
|           multiqc | Data        : OG6761-multiqc-report_data
|           multiqc | MultiQC complete
@ewels
Copy link
Member

ewels commented Jun 1, 2022

Huh, that sounds odd and potentially serious. Is it just v1.13dev, not v1.12? There's been a few changes in the past months on this module - #1595 and (issue: #1563)

@andrei-seleznev / @yanick - any ideas what could be going on here?

@ajeffs
Copy link
Author

ajeffs commented Jun 1, 2022 via email

@andrei-seleznev
Copy link
Contributor

Huh, that sounds odd and potentially serious. Is it just v1.13dev, not v1.12? There's been a few changes in the past months on this module - #1595 and (issue: #1563)

@andrei-seleznev / @yanick - any ideas what could be going on here?

bclconvert module simply assumes paired-end reads - line 244 bclconvert.py
return {"run_id": run_id, "read_length": int(read_length), "cluster_length": int(read_length) * 2}

I'll add some code to do a check and adjustment for single-end there and put in a PR.

@ewels ewels added the bug: module Bug in a MultiQC module label Jun 3, 2022
@ewels ewels changed the title Incorrect lane and sample statistics for single-read data (BCL Convert module, v1.13.dev0) bclconvert: incorrect stats for single-read data Jun 3, 2022
andrei-seleznev added a commit to andrei-seleznev/MultiQC that referenced this issue Jun 7, 2022
@andrei-seleznev
bclconvert singleeand reads now have cluster length equal to read len…
2b902b2 
…gth, this resolves MultiQC#1697
@andrei-seleznev andrei-seleznev mentioned this issue Jun 7, 2022
Merged
1 task
@ewels ewels closed this as completed in a6a283f Sep 9, 2022
@ewels ewels mentioned this issue Oct 4, 2022
Closed
bnbowman pushed a commit to bnbowman/MultiQC that referenced this issue Dec 20, 2022
@alexiswl @ErikDanielsson
@ewels @idot @seb-mueller @J-81 @NiemannJ @fgvieira @mattloose @jchorl @vladsavelyev @jethror1 @gartician @andrei-seleznev @phue @IllustratedMan-code @y9c @beausoleilmo @github-actions @JeanMainguy @aidaanva @SusiJo @TNalpat @Redmar-van-den-Berg @jfy133 @Maarten-vd-Sande @olehpratsko
Merge/bnbowman master upstream master (#35)
ae557c4 
* Try prefixing analysis dirs

* Update CHANGELOG.md

* tried to add conditional execution to actions

* Update CHANGELOG.md

* Add dependency statement

* only overwrite id when not set

* changelog

* always set c_id

* Revert setting force_interactive flag for rich with --no-ansi

* Don't force terminal escape codes for the progress bar

* Extend kallisto module regex to recognize newer output

I've noticed multiqc (v 1.12) didn't recognise some kallisto output (I'm using kb_python 0.26.4).
Having digged a bit, it seems to not work with more recent kallisto output, i.e. this snippet taken from here:
https://github.com/pachterlab/GRNP_2020/blob/daed9c2f204f1c3f6ee0e864c3db93b0baadfc8a/notebooks/FASTQ_processing/ProcessPBMC_NG.ipynb
```
[index] k-mer length: 31
[index] number of targets: 187,626
[index] number of k-mers: 108,619,921
tcmalloc: large alloc 3221225472 bytes == 0x556459b7e000 @  0x7feac4ab5887 0x556458814ad2 0x55645880d061 0x5564587e1372 0x7feac3935bf7 0x5564587e60da
[index] number of equivalence classes: 752,021
[quant] will process sample 1: A_R1.gz
                               A_R2.gz
[quant] will process sample 2: B_R1.gz
                               B_R2.gz
[quant] finding pseudoalignments for the reads ... done
[quant] processed 170,526,037 reads, 98,632,205 reads pseudoaligned
```
Turns out multiqc only looks for `pair|file` but not sample. Replacing sample for file did do the trick, hence I suggest to add `sample` in the regex pattern.
I haven't tested this, but it should work now.

Here is the code which generates this output:
https://github.com/pachterlab/kallisto/blob/83bde908c403ea4014b5092a243e5c7240f48dd5/src/ProcessReads.cpp#L235

This is the commit which introduced it (already in 2018, so not sure why this hasn't been caught yet)
pachterlab/kallisto@62e9464

* Replace logger.hasHandlers() with logger.handlers

There are cases where configuring logging results in logger.handlers being empty but logger.hasHandlers() returns True: MultiQC#1643

Since the block modified removes based on logger.handlers, the condition to enter the block should check logger.handlers rather than logger.hasHandlers()

* Added description of changes for pull request

* Document 'no_version_check' config option

* Docs tweak

* Fix kwargs for MultiQC plugins

* New config option 'custom_table_header_config'

* Run black

* Update adapterRemoval.py

Returns actual proportion of reads that were collapsed and discarded

* Black format

* Black format

* BlackPython

* Fix chart labels and titles

* Fix chart labels and titles

* Add columns to stats table

Add columns with proportion of collapsed/discarded reads to the general stats table

* Add Columns - Fix format

* Changelog

* Fixed bug when other fields also have a "-" instead of an integer.

* Updated CHANGELOG

* Fixed typos

* Fixed format typo

* Fixed format typo

* Nanostat: Remove HTML escaping

Jinja2 escape() function removed in jinja2 v3.10

I don't think that this escaping should be required. I can't see any effect in the report when I remove it anyway.

* Changelog

* Changing 0 to None

* Skip fields with `-`

* Pangolin 4.0 compatability

Recently pangolin has been updated to version 4.0 and this changes the output CSV file - see: https://github.com/cov-lineages/pangolin/releases/tag/v4.0

This causes the module to fail in its current state as row['qc_status'] already exists and the current replacement triggers a key error by searching for row['status'] which no longer exists. Thanks to @alexomics for tracking down the issue.

* Don't duplicate custom-content section descriptions.

Fixed edge-case bug in custom content where a `description` that doesn't terminate in `.` gave duplicate section descriptions.

* Changelog

* Tidied the verbose log, added summaries for skipped search files to debug log

* Allow sorting of table columns with text contents

* update changelog

* optimize linegraph category comparison

* Somalier: division by zero in sex ploidy plot

* Changelog

* Add time zone

* Update changelog

* Fix typo in bcl2fastq.py

* Handle too long and low complexity

* update changelog

* fix zero division error in sambamba markdup module

* black formatting

* update CHANGELOG.md to address MultiQC#1654

* bclconvert checks RunInfo xml if reads are singleend or pairedend and sets clusterlength appropriately. resolves MultiQC#1697

* Added CITATION.cff file for standardized citations

* fixed formatting of url

* fixed citation formatting

* Run prettier

* Fix module crashing due to missing field in report

* Fix bug where module wouldn't run if all content was within a MultiQC config file

Fixes MultiQC#1686

* nanostat: add check for quality scores

* update CHANGELOG.md

* update CHANGELOG.md

* Custom content: Fix crash when 'info' isn't set

Closes MultiQC#1688

* Added nix flake support

* Update docs/installation.md

Co-authored-by: Phil Ewels <phil@seqera.io>

* Fix zero division error

* Update fastqc.py

* Update fastqc.py

* fix format

* add change log

* fix doc ref

* Don't need Prettier _and_ markdown/yamllint CI

* Just capture the ValueError

* Rich-codex screenshot in the readme

* Corrected 'outdir' flag

Missing a dash for the flag to work.

* Clean up clean_img_paths

* Generate new screengrabs with rich-codex

* Add samtools flagstat column '% Read Mapped'

* update samtools flagstat changelog

* Added try,except for divisions to avoid division by 0 errors

* added the fixing of malt in the change log

* report median read length for fastqc

* add after filtering total reads to general stats table

* GitHub Actions: Tweet about new releases

* Bump to v1.13 for release

* rich-codex screenshots: Manual only, skip git checks

* Generate new screengrabs with rich-codex

* Fix changelog date

* Bump to v1.14dev

* Custom content: Render report even if there's only general stats there

See MultiQC#1756

* Bugfix: Make `config.data_format` work again

* Bump minimum version of Jinja2 to `>=3.0.0`

Closes MultiQC#1642

* Disable search progress bar if running with `--quiet` or `--no-ansi`

Closes MultiQC#1638

* Attempt to cooerce line / scatter x-axes into floats so as not to lose labels

See MultiQC#1242

* Use row 1 as x-axis labels if no sample name.

Closes MultiQC#1242

* Malt: Move changelog up to new version

* Merge changelog up

* Use OrderedDict instead of 'placement'

* Add code comment

* Add CI testing for Python 3.10 and 3.11

* Fix typo

* Quotes so it's 3.10 and not 3.1

* 3.11-dev

* Remove 3.11-dev for Windows

* Move merge markers GHA into lint workflow file

* Shorter job name

* Be more selective about when slow MultiQC test runs fire

- Master only for push event
- Don't run if only docs / markdown

* Run isort

* Remove py2 'from __future__ import print_function'

* Add GitHub actions CI for isort

* Changelog

* Remove all py2 'from __future__ imports'

* Tweak some imports

* Changelog

* added setuptools to flake

* rm emtpy bcftools stats variant depths plot

* moved changelog comment

* adjusted PR num

* fix duplicate heatmap for kraken

* changelog

* use None instead

* First commit of pre-commit

* Comment out all the tests that don't yet work

* Update gene_body_coverage.py

Using a normalized coverage to make genebody coverage plot ( similar to the method used by RSeQC). Us the formula 'norm_cov = ( cov - min(cov ) / ( max(cov) - min(cov) )' to compute normalized coverage

* Update gene_body_coverage.py

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update CHANGELOG.md

* Move changelog entry

* Test for Python 3.11 now that the official release is out

* CI: Use new version of actions/checkout to avoid Node.js depreciation warning

* Remove sample and chromosome before converting to int

This fixes issue-1793

* Remove filtered samples from general stats table

This fixes MultiQC#1780

* Update changelog

* Add additional entries for qualimap when region stats present

* fastp: use passed filter reads instead of after filter total reads

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* bclconvert now handles different r1 and r2 lengths instead of assuming they are the same

* updated CHANGELOG.md

* update bustools

* Update CHANGELOG.md

* Remove changelog entry

* Move changelog to entry to correct place

* Fix changelog

* Kraken: Improve heatmap config

* Apply suggestions from code review

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* handle singleindex data

* cleanup

* CHANGELOG.md bclconvert fix issue link typo and note single-index paired-end data handled

* Qualimap BamQC: Refactor to parse regexes per section

Also: Fix randomly aggressive Snippy module parsing bug

* HsMetrics: Allow custom columns in General Stats too

* Replace nested loop with list comprehension when parsing output file headers

* CHANGELOG

* Output headers order preserved and Sample is first value

* Fix ubuntu version in GitHub CI to preserve Py3.6 testing.

Python 3.6, I think your days are numbered..

* add back original avg field

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* fixes

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* update busco colors

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* fix: frontmatter yaml formatting issue

* Update docs to use --cl-config instead of --cl_config

Closes MultiQC#1825

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* suggestion

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>
Co-authored-by: Erik Danielsson <danielsson.erik.0@gmail.com>
Co-authored-by: Phil Ewels <phil.ewels@scilifelab.se>
Co-authored-by: Ido Tamir <ido.tamir@vbcf.ac.at>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Jonathan Oribello <Jonathan.d.oribello@gmail.com>
Co-authored-by: NiemannJ <69033839+NiemannJ@users.noreply.github.com>
Co-authored-by: fgvieira <fgarrettvieira@gmail.com>
Co-authored-by: mattloose <matt.loose@nottingham.ac.uk>
Co-authored-by: Josh Chorlton <jchorlton@gmail.com>
Co-authored-by: vladsaveliev <vladislav.savelyev@populationgenomics.org.au>
Co-authored-by: Sam Chorlton <>
Co-authored-by: jethror1 <45037268+jethror1@users.noreply.github.com>
Co-authored-by: Garth Kong <kongga2017@gmail.com>
Co-authored-by: Andrei Seleznev <aseleznev@illumina.com>
Co-authored-by: lew2mz <david.lewis@cchmc.org>
Co-authored-by: Phil Ewels <phil@seqera.io>
Co-authored-by: phue <patrick.huether@imp.ac.at>
Co-authored-by: David Lewis <60514384+IllustratedMan-code@users.noreply.github.com>
Co-authored-by: Chang Y <yech1990@gmail.com>
Co-authored-by: beausoleilmo <beausoleilmo@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Mainguy <jean.mainguy@outlook.fr>
Co-authored-by: aidaanva <aida.andrades@gmail.com>
Co-authored-by: SusiJo <susanne.jodoin@gmx.de>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Co-authored-by: TNalpat <thomas.nalpathamkalam@gmail.com>
Co-authored-by: Redmar van den Berg <RedmarvandenBerg@lumc.nl>
Co-authored-by: James Fellows Yates <jfy133@gmail.com>
Co-authored-by: Maarten-vd-Sande <maartenvandersande@hotmail.com>
Co-authored-by: Adam Talbot <adam.talbot@nonacus.com>
Co-authored-by: Oleh Pratsko <olehpratsko@gmail.com>
Co-authored-by: Josh Chorlton <jchorl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug: module Bug in a MultiQC module
Projects
None yet
Milestone
No milestone
3 participants
@ewels @ajeffs @andrei-seleznev