Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

custom content MultiQC module broke #1688

Closed
carmelinacharalambous opened this issue May 18, 2022 · 7 comments
Closed

custom content MultiQC module broke #1688

carmelinacharalambous opened this issue May 18, 2022 · 7 comments
Labels
bug: module Bug in a MultiQC module custom-content

Comments

@carmelinacharalambous
Copy link

Description of bug

Config file that worked for multiqc v1.10.1.7 partially works for multiqc v1.12

Report is generated but some modules are missing.
I'm attaching the config file (nucleo_CH_config_v1.0.yaml changed to .txt as .yml is not supported).

File that triggers the error

nucleo_CH_config_v1.0.txt

MultiQC Error log

$  /// MultiQC 🔍 | v1.13.dev0

|           multiqc | Report title: MSK CMO MultiQC Report
|           multiqc | Search path : /wMJDfE
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 68/68  
|             table | Not able to apply table conditional formatting to '' ({'lt': 2000})
|             table | Not able to apply table conditional formatting to '' ({'eq': 2000})
|             table | Not able to apply table conditional formatting to '' ({'gt': 500})
|             table | Not able to apply table conditional formatting to '' ({'eq': 500})
|             table | Not able to apply table conditional formatting to '' ({'gt': 2000})
|             table | Not able to apply table conditional formatting to '' ({'lt': 500})
|             table | Not able to apply table conditional formatting to '' ({'lt': 100})
|             table | Not able to apply table conditional formatting to '' ({'lt': 170})
|             table | Not able to apply table conditional formatting to '' ({'eq': 170})
|             table | Not able to apply table conditional formatting to '' ({'gt': 163})
|             table | Not able to apply table conditional formatting to '' ({'eq': 163})
|             table | Not able to apply table conditional formatting to '' ({'gt': 170})
|             table | Not able to apply table conditional formatting to '' ({'lt': 163})
|             table | Not able to apply table conditional formatting to '' ({'gt': 180})
|             table | Not able to apply table conditional formatting to '' ({'lt': 150})
|             table | Not able to apply table conditional formatting to '' ({'lt': 100000000})
|             table | Not able to apply table conditional formatting to '' ({'eq': 100000000})
|             table | Not able to apply table conditional formatting to '' ({'gt': 50000000})
|             table | Not able to apply table conditional formatting to '' ({'eq': 50000000})
|             table | Not able to apply table conditional formatting to '' ({'gt': 100000000})
|             table | Not able to apply table conditional formatting to '' ({'lt': 50000000})
|             table | Not able to apply table conditional formatting to '' ({'lt': 1000000})
|             table | Not able to apply table conditional formatting to '' ({'gt': 0.97})
|             table | Not able to apply table conditional formatting to '' ({'eq': 0.97})
|             table | Not able to apply table conditional formatting to '' ({'lt': 0.97})
|             table | Not able to apply table conditional formatting to '' ({'lt': 0.95})
|    custom_content | genstats_qcstatus_plasma: Found 1 samples (table)
|    custom_content | biometrics_major_collapsed: Found 1 samples (table)
╭───────────────────────────────────────── Oops! The 'custom_content' MultiQC module broke... ─────────────────────────────────────────╮
│ Please copy this log and report it at https://github.com/ewels/MultiQC/issues                                                        │
│ Please attach a file that triggers the error. The last file found was: ./aggregate_qc_stats/qc_criteria.csv                          │
│                                                                                                                                      │
│ Traceback (most recent call last):                                                                                                   │
│   File "/usr/local/lib/python3.6/site-packages/multiqc/multiqc.py", line 651, in run                                                 │
│     output = mod()                                                                                                                   │
│   File "/usr/local/lib/python3.6/site-packages/multiqc/modules/custom_content/custom_content.py", line 277, in custom_module_classes │
│     parsed_modules[mod_id].add_cc_section(c_id, mod)                                                                                 │
│   File "/usr/local/lib/python3.6/site-packages/multiqc/modules/custom_content/custom_content.py", line 418, in add_cc_section        │
│     if section_description.strip(".") == self.info.strip("."):                                                                       │
│ AttributeError: 'NoneType' object has no attribute 'strip'                                                                           │
│                                                                                                                                      │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
|          mosdepth | Found 1 reports
|            picard | Found 1 AlignmentSummaryMetrics reports
/usr/local/lib/python3.6/site-packages/multiqc/modules/picard/HsMetrics.py:369: RuntimeWarning: Mean of empty slice.
  vals_avg = vals[gc_binned == bin_index].mean()
/usr/local/lib/python3.6/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
@ewels
Copy link
Member

ewels commented Jun 20, 2022

Hi @carmelinacharalambous,

Any reason you closed this? Looks like a genuine bug with that traceback 🤔 Even if stuff is weird / wrong in the custom content file you shouldn't get unhandled tracebacks like that.

Phil

@ewels ewels reopened this Jun 20, 2022
@ewels ewels added custom-content bug: module Bug in a MultiQC module labels Jun 20, 2022
@phue
Copy link
Contributor

phue commented Jul 1, 2022

This seems to have introduced this: ea18493

@ewels
Copy link
Member

ewels commented Jul 1, 2022

Ah good spot, thanks @phue 👍🏻 Should be an easy fix.

@ewels
Copy link
Member

ewels commented Jul 1, 2022

I'm struggling to reproduce - the log says that the last file was ./aggregate_qc_stats/qc_criteria.csv but the attached file is a huge MultiQC config (possibly the biggest I've ever seen 😅) with no data.

@ewels ewels closed this as completed in 3db021d Jul 1, 2022
@ewels
Copy link
Member

ewels commented Jul 1, 2022

Ok, just pushed a "blind" fix - wasn't able to reproduce the error but going on what you said @phue I think this should have hopefully fixed it..

@phue
Copy link
Contributor

phue commented Jul 1, 2022

🥷 that was quick @ewels 👀

I guess this would have been the alternative:
phue@a38ffc6

@ewels
Copy link
Member

ewels commented Jul 1, 2022

Ah sorry 😅 Yup, that works as well 👍🏻 Is actually more consistent..

bnbowman pushed a commit to bnbowman/MultiQC that referenced this issue Dec 20, 2022
* Try prefixing analysis dirs

* Update CHANGELOG.md

* tried to add conditional execution to actions

* Update CHANGELOG.md

* Add dependency statement

* only overwrite id when not set

* changelog

* always set c_id

* Revert setting force_interactive flag for rich with --no-ansi

* Don't force terminal escape codes for the progress bar

* Extend kallisto module regex to recognize newer output

I've noticed multiqc (v 1.12) didn't recognise some kallisto output (I'm using kb_python 0.26.4).
Having digged a bit, it seems to not work with more recent kallisto output, i.e. this snippet taken from here:
https://github.com/pachterlab/GRNP_2020/blob/daed9c2f204f1c3f6ee0e864c3db93b0baadfc8a/notebooks/FASTQ_processing/ProcessPBMC_NG.ipynb
```
[index] k-mer length: 31
[index] number of targets: 187,626
[index] number of k-mers: 108,619,921
tcmalloc: large alloc 3221225472 bytes == 0x556459b7e000 @  0x7feac4ab5887 0x556458814ad2 0x55645880d061 0x5564587e1372 0x7feac3935bf7 0x5564587e60da
[index] number of equivalence classes: 752,021
[quant] will process sample 1: A_R1.gz
                               A_R2.gz
[quant] will process sample 2: B_R1.gz
                               B_R2.gz
[quant] finding pseudoalignments for the reads ... done
[quant] processed 170,526,037 reads, 98,632,205 reads pseudoaligned
```
Turns out multiqc only looks for `pair|file` but not sample. Replacing sample for file did do the trick, hence I suggest to add `sample` in the regex pattern.
I haven't tested this, but it should work now.

Here is the code which generates this output:
https://github.com/pachterlab/kallisto/blob/83bde908c403ea4014b5092a243e5c7240f48dd5/src/ProcessReads.cpp#L235

This is the commit which introduced it (already in 2018, so not sure why this hasn't been caught yet)
pachterlab/kallisto@62e9464

* Replace logger.hasHandlers() with logger.handlers

There are cases where configuring logging results in logger.handlers being empty but logger.hasHandlers() returns True: MultiQC#1643

Since the block modified removes based on logger.handlers, the condition to enter the block should check logger.handlers rather than logger.hasHandlers()

* Added description of changes for pull request

* Document 'no_version_check' config option

* Docs tweak

* Fix kwargs for MultiQC plugins

* New config option 'custom_table_header_config'

* Run black

* Update adapterRemoval.py

Returns actual proportion of reads that were collapsed and discarded

* Black format

* Black format

* BlackPython

* Fix chart labels and titles

* Fix chart labels and titles

* Add columns to stats table

Add columns with proportion of collapsed/discarded reads to the general stats table

* Add Columns - Fix format

* Changelog

* Fixed bug when other fields also have a "-" instead of an integer.

* Updated CHANGELOG

* Fixed typos

* Fixed format typo

* Fixed format typo

* Nanostat: Remove HTML escaping

Jinja2 escape() function removed in jinja2 v3.10

I don't think that this escaping should be required. I can't see any effect in the report when I remove it anyway.

* Changelog

* Changing 0 to None

* Skip fields with `-`

* Pangolin 4.0 compatability

Recently pangolin has been updated to version 4.0 and this changes the output CSV file - see: https://github.com/cov-lineages/pangolin/releases/tag/v4.0

This causes the module to fail in its current state as row['qc_status'] already exists and the current replacement triggers a key error by searching for row['status'] which no longer exists. Thanks to @alexomics for tracking down the issue.

* Don't duplicate custom-content section descriptions.

Fixed edge-case bug in custom content where a `description` that doesn't terminate in `.` gave duplicate section descriptions.

* Changelog

* Tidied the verbose log, added summaries for skipped search files to debug log

* Allow sorting of table columns with text contents

* update changelog

* optimize linegraph category comparison

* Somalier: division by zero in sex ploidy plot

* Changelog

* Add time zone

* Update changelog

* Fix typo in bcl2fastq.py

* Handle too long and low complexity

* update changelog

* fix zero division error in sambamba markdup module

* black formatting

* update CHANGELOG.md to address MultiQC#1654

* bclconvert checks RunInfo xml if reads are singleend or pairedend and sets clusterlength appropriately. resolves MultiQC#1697

* Added CITATION.cff file for standardized citations

* fixed formatting of url

* fixed citation formatting

* Run prettier

* Fix module crashing due to missing field in report

* Fix bug where module wouldn't run if all content was within a MultiQC config file

Fixes MultiQC#1686

* nanostat: add check for quality scores

* update CHANGELOG.md

* update CHANGELOG.md

* Custom content: Fix crash when 'info' isn't set

Closes MultiQC#1688

* Added nix flake support

* Update docs/installation.md

Co-authored-by: Phil Ewels <phil@seqera.io>

* Fix zero division error

* Update fastqc.py

* Update fastqc.py

* fix format

* add change log

* fix doc ref

* Don't need Prettier _and_ markdown/yamllint CI

* Just capture the ValueError

* Rich-codex screenshot in the readme

* Corrected 'outdir' flag

Missing a dash for the flag to work.

* Clean up clean_img_paths

* Generate new screengrabs with rich-codex

* Add samtools flagstat column '% Read Mapped'

* update samtools flagstat changelog

* Added try,except for divisions to avoid division by 0 errors

* added the fixing of malt in the change log

* report median read length for fastqc

* add after filtering total reads to general stats table

* GitHub Actions: Tweet about new releases

* Bump to v1.13 for release

* rich-codex screenshots: Manual only, skip git checks

* Generate new screengrabs with rich-codex

* Fix changelog date

* Bump to v1.14dev

* Custom content: Render report even if there's only general stats there

See MultiQC#1756

* Bugfix: Make `config.data_format` work again

* Bump minimum version of Jinja2 to `>=3.0.0`

Closes MultiQC#1642

* Disable search progress bar if running with `--quiet` or `--no-ansi`

Closes MultiQC#1638

* Attempt to cooerce line / scatter x-axes into floats so as not to lose labels

See MultiQC#1242

* Use row 1 as x-axis labels if no sample name.

Closes MultiQC#1242

* Malt: Move changelog up to new version

* Merge changelog up

* Use OrderedDict instead of 'placement'

* Add code comment

* Add CI testing for Python 3.10 and 3.11

* Fix typo

* Quotes so it's 3.10 and not 3.1

* 3.11-dev

* Remove 3.11-dev for Windows

* Move merge markers GHA into lint workflow file

* Shorter job name

* Be more selective about when slow MultiQC test runs fire

- Master only for push event
- Don't run if only docs / markdown

* Run isort

* Remove py2 'from __future__ import print_function'

* Add GitHub actions CI for isort

* Changelog

* Remove all py2 'from __future__ imports'

* Tweak some imports

* Changelog

* added setuptools to flake

* rm emtpy bcftools stats variant depths plot

* moved changelog comment

* adjusted PR num

* fix duplicate heatmap for kraken

* changelog

* use None instead

* First commit of pre-commit

* Comment out all the tests that don't yet work

* Update gene_body_coverage.py

Using a normalized coverage to make genebody coverage plot ( similar to the method used by RSeQC). Us the formula 'norm_cov = ( cov - min(cov ) / ( max(cov) - min(cov) )' to compute normalized coverage

* Update gene_body_coverage.py

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update CHANGELOG.md

* Move changelog entry

* Test for Python 3.11 now that the official release is out

* CI: Use new version of actions/checkout to avoid Node.js depreciation warning

* Remove sample and chromosome before converting to int

This fixes issue-1793

* Remove filtered samples from general stats table

This fixes MultiQC#1780

* Update changelog

* Add additional entries for qualimap when region stats present

* fastp: use passed filter reads instead of after filter total reads

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* bclconvert now handles different r1 and r2 lengths instead of assuming they are the same

* updated CHANGELOG.md

* update bustools

* Update CHANGELOG.md

* Remove changelog entry

* Move changelog to entry to correct place

* Fix changelog

* Kraken: Improve heatmap config

* Apply suggestions from code review

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* handle singleindex data

* cleanup

* CHANGELOG.md bclconvert fix issue link typo and note single-index paired-end data handled

* Qualimap BamQC: Refactor to parse regexes per section

Also: Fix randomly aggressive Snippy module parsing bug

* HsMetrics: Allow custom columns in General Stats too

* Replace nested loop with list comprehension when parsing output file headers

* CHANGELOG

* Output headers order preserved and Sample is first value

* Fix ubuntu version in GitHub CI to preserve Py3.6 testing.

Python 3.6, I think your days are numbered..

* add back original avg field

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* fixes

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* update busco colors

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>

* fix: frontmatter yaml formatting issue

* Update docs to use --cl-config instead of --cl_config

Closes MultiQC#1825

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Update multiqc/modules/fastqc/fastqc.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* suggestion

Signed-off-by: Josh Chorlton <jchorlton@gmail.com>
Co-authored-by: Erik Danielsson <danielsson.erik.0@gmail.com>
Co-authored-by: Phil Ewels <phil.ewels@scilifelab.se>
Co-authored-by: Ido Tamir <ido.tamir@vbcf.ac.at>
Co-authored-by: seb-mueller <sebm@posteo.de>
Co-authored-by: Jonathan Oribello <Jonathan.d.oribello@gmail.com>
Co-authored-by: NiemannJ <69033839+NiemannJ@users.noreply.github.com>
Co-authored-by: fgvieira <fgarrettvieira@gmail.com>
Co-authored-by: mattloose <matt.loose@nottingham.ac.uk>
Co-authored-by: Josh Chorlton <jchorlton@gmail.com>
Co-authored-by: vladsaveliev <vladislav.savelyev@populationgenomics.org.au>
Co-authored-by: Sam Chorlton <>
Co-authored-by: jethror1 <45037268+jethror1@users.noreply.github.com>
Co-authored-by: Garth Kong <kongga2017@gmail.com>
Co-authored-by: Andrei Seleznev <aseleznev@illumina.com>
Co-authored-by: lew2mz <david.lewis@cchmc.org>
Co-authored-by: Phil Ewels <phil@seqera.io>
Co-authored-by: phue <patrick.huether@imp.ac.at>
Co-authored-by: David Lewis <60514384+IllustratedMan-code@users.noreply.github.com>
Co-authored-by: Chang Y <yech1990@gmail.com>
Co-authored-by: beausoleilmo <beausoleilmo@users.noreply.github.com>
Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Jean Mainguy <jean.mainguy@outlook.fr>
Co-authored-by: aidaanva <aida.andrades@gmail.com>
Co-authored-by: SusiJo <susanne.jodoin@gmx.de>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>
Co-authored-by: TNalpat <thomas.nalpathamkalam@gmail.com>
Co-authored-by: Redmar van den Berg <RedmarvandenBerg@lumc.nl>
Co-authored-by: James Fellows Yates <jfy133@gmail.com>
Co-authored-by: Maarten-vd-Sande <maartenvandersande@hotmail.com>
Co-authored-by: Adam Talbot <adam.talbot@nonacus.com>
Co-authored-by: Oleh Pratsko <olehpratsko@gmail.com>
Co-authored-by: Josh Chorlton <jchorl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug: module Bug in a MultiQC module custom-content
Projects
None yet
Development

No branches or pull requests

3 participants