Add new module to support DRAGEN-FastQC

[bnbowman]

• This comment contains a description of changes (within reason)
• CHANGELOG.md has been updated

Illumina is in the process of updating it's DRAGEN Bio-It platform that, among other things, adds a new feature to generate hardware-accelerated primary analysis statistics similar to the widely used FastQC tool. This PR creates a new module that parses the outputs from that new feature and uses it to replicate the plots that a user would expect to see from FastQC, or from MultiQC's existing FastQC module.

NOTE #1: This module doesn't yet properly support sample-quality flags, but it needs to before it's merged. But I wanted to get this PR up to start the discussion before the weekend, while I'm still working on that.

NOTE #2: As part of supporting full graphical parity with FastQC, this PR also includes new plotting functionality to create box-and-whisker plots. I imagine these need a fair amount of work still, particularly for things like pconfig support, but I think some code-review would help differentiate the must-haves from nice-to-haves

• There is example tool output for tools in the https://github.com/ewels/MultiQC_TestData repository or attached to this PR
• Code is tested and works locally (including with --lint flag)
• docs/README.md is updated with link to below
• docs/modulename.md is created
• Everything that can be represented with a plot instead of a table is a plot
• Report sections have a description and help text (with self.add_section)
• There aren't any huge tables with > 6 columns (explain reasoning if so)
• Each table column has a different colour scale to its neighbour, which relates to the data (eg. if high numbers are bad, they're red)
• Module does not do any significant computational work

[ewels]

This is huge, thank you for your work on this @bnbowman! Lots to look through and test so I need to carve out a proper slot of time to work on this, but a couple of very fast speed thoughts:

• Box plots code looks nice, but I see that you kind of cheated and it's only doing flat matplotlib images 😉
  • This plot type is well overdue, if I update HighCharts then it should be possible to have interactive as well as static boxplots now. It will require quite a bit of new javascript.
  • I'm happy to take this on, and would prefer to keep it in a separate PR if possible
  • I'm very happy that you did the matplotlib stuff already, hopefully I can take your work there and extend it - in the past it's been the matplotlib stuff that has given me the most grief 👍
• Shared FastQC plot code
  • This really needs to be ported / rewritten as core functionality. I think this is about the third or maybe fourth module that has tried to reuse the code so it's definitely time.
  • More than just moving the files around, I think it could do with some refactoring. Much of this code comes from the very early days of MultiQC and there is probably a better approach with some of it now.
  • Specifically for the base content heatmap, I would like to look into rewriting this using the core MultiQC heatmap functions if possible.

So I think we basically have three separate PRs in one here - boxplots, core FastQC functionality, and DRAGEN. My preferred approach would be for someone (probably me) to tackle the first two in separate PRs. Especially the second point, as I'd like to update other modules such as FastQC as part of the same process. Once those are done we can come back to the DRAGEN part to make use of this new functionality.

This is going to take a fair amount of work but I think that these updates will really improve MultiQC and are often requested. I'll try to spend some time on this once I've got my current batch of @nf_core stuff done.

So this isn't going to be a quick approve + merge PR I'm afraid @bnbowman 😞 I need to finish other stuff and once I start it'll take me a while to implement the above. But the PR is 2k lines of new code which touch at the heart of MultiQC, so maybe that's not surprising 😅

I guess that time will be a factor for you and you'll want this merged ASAP. If so then maybe we can discuss a strategy to break this up a little and start bringing in parts of the new DRAGEN functionality that don't rely on the core MultiQC changes mentioned above. What do you think?

Phil

[ewels]

ps. I'm not entirely sure what you mean about sample-quality flags?

[bnbowman]

Sorry for the delayed response - I've been on vacation for the past week and a half (Not to mention dealing with a brand new puppy...)

So I think we basically have three separate PRs in one here - boxplots, core FastQC functionality, and DRAGEN.
My preferred approach would be for someone (probably me) to tackle the first two in separate PRs.

I broadly agree with this. I can help somewhat on the Python-side of the boxplots, and I agree the DRAGEN code is my responsibility, but HighCharts, core infra, and JS in general are not my forte.

So this isn't going to be a quick approve + merge PR I'm afraid

Totally understand. I tried to get people rallied around this course of action early b/c I knew it was going to be a lift, but there was a lot of internal dithering over MultiQC vs. a bespoke solution. So here we are. We're using my personal branch for now for business purposes, so it's not a show-stopper, but it is definitely something we want to get done sooner rather than later.

ps. I'm not entirely sure what you mean about sample-quality flags?

The thing you do where you color some lines green/yellow/red depending on how the sample stacks up against various criteria? And have a little colored box of the number of samples at the top of each section? That thing.

[bnbowman]

On a side note - I've got a major deadline for another project due at the end of the month, so I won't have much time to work on this until then. But afterwards I should have more flexibility - will reach out to check in and see if there is anything I can help with when my schedule opens up a little.

[ewels]

@andrei-seleznev / @alexiswl / @bnbowman / anyone else who might knows - this PR includes a new kiwisolver dependency, but I can't see it mentioned anywhere in the codebase except setup.py.

Is this new dependency definitely still used? It's one of the main blockers for me from merging currently. I'm basically happy to merge anything within the dragen folders at this point as-is, plus the box plot. The only other thing outside of that scope is this extra dependency.

[alexiswl]

@ewels I'm not sure what the kiwisolver is used for but it was first seen in this commit from @bnbowman.

I remember having the same issue you are seeing when trying to create a MultiQC docker image from this branch and attributed it this kiwi solver issue. Although this issue was resolved in kiwisolver version 1.4.4, trying to build the Dockerfile with this later version of kiwisolver resulted in the same error:

...
  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -Os -fomit-frame-pointer -g -DTHREAD_STACK_SIZE=0x100000 -fPIC -I/tmp/pip-build-env-iwfb688u/overlay/lib/python3.8/site-packages/cppy/include -I. -I/usr/include/python3.8 -c py/src/constraint.cpp -o build/temp.linux-x86_64-cpython-38/py/src/constraint.o -std=c++11
  /tmp/pip-build-env-iwfb688u/overlay/lib/python3.8/site-packages/setuptools/config/pyprojecttoml.py:108: _BetaConfiguration: Support for `[tool.setuptools]` in `pyproject.toml` is still *beta*.
    warnings.warn(msg, _BetaConfiguration)
  error: command 'gcc' failed: No such file or directory
  ----------------------------------------
  ERROR: Failed building wheel for kiwisolver
  Running setup.py clean for kiwisolver
  Building wheel for pyyaml (PEP 517): started
  Building wheel for pyyaml (PEP 517): finished with status 'done'
  Created wheel for pyyaml: filename=PyYAML-6.0-cp38-cp38-linux_x86_64.whl size=45331 sha256=480051c2b1516b42060a514652cb283c4dcabb6cac9d829a8bd24409bbe9a21e
  Stored in directory: /root/.cache/pip/wheels/95/84/67/ebeac632c63797cfbeb90128ca41073117721540dad526d213
Successfully built multiqc pyyaml
Failed to build kiwisolver
ERROR: Could not build wheels for kiwisolver which use PEP 517 and cannot be installed directly

A workaround for me was using #1541 which installs (and then removes) the build essentials for compiling pip packages that don't have the available wheels to install.

[ewels]

That commit was 2 years ago though and presumably to also solve some kind of installation error. I'm hoping that it's no longer needed and we can remove it.

[alexiswl]

That commit was 2 years ago though and presumably to also solve some kind of installation error. I'm hoping that it's no longer needed and we can remove it.

Didn't even think of that! Rebuilt this branch without kiwisolver and was still able to produce dragen and dragen-fastqc module outputs.

[andrei-seleznev]

I got in touch with Brett - indeed it was an old trick to solve local environment dependency issues. Good to be removed.

[ewels]

Great news! Ok then it's just fixing up these merge conflicts, a few test runs to ease my conscience, and hopefully we're good to go..!

[alexiswl]

@ewels I've put in a PR to @bnbowman to resolve the merge conflicts with bnbowman#35.

[ewels]

Thanks @alexiswl! The module still needs quite a bit of work, I've made a start over on #1829 where I can push to the branch. Nearly there now I think, see comment #1829 (comment)

[ewels]

Ok, I think I've got to a point where I'm happy to merge 🎉

Caveats:

• I'm pretty sure that the Dragen (and maybe the dragen_fastqc? or maybe not?) test data is pretty incomplete.
  • If anyone is able to contribute more test data there to cover all module outputs, that would be great.
  • I would be surprised if CI tests continue to pass when new test data is added! But that's kind of the point, we can fix that
• Some stuff in this PR is still not done, eg. heatmap / hover bars showing sample status.
  • This can be added at a later date if needed
• I haven't properly read / tested all code, as there's just too much of it for me to do in the time I have
  • I'm hoping that the code changes in Fix merge conflicts for #1232 #1829 are isolated enough that anything that makes it through will not affect other MultiQC users
  • If anyone hits bugs in the Dragen code, please report - or better still, fix with a PR! 🙏🏻

Let's do this.. Now or never 😅 Hopefully this means we can bring you all back onto the main MultiQC distribution / development branch!

Many thanks to everyone who has contributed to this PR. Especially @bnbowman for the bulk of the new code but also to those who have come in to help with the cleanup at such a late stage. Much appreciated.

Phil