bcl2fastq module breaks when number of demultiplexed Reads >2

[sachalau]

Description of bug:
I updated to 1.7 recently from 1.3. The bcl2fastq module added specific statistics regarding R1 and R2. However if more reads are demultiplexed the module breaks because no variable is created in the module for those reads.

MultiQC Error log:

[ERROR  ]         multiqc : Oops! The 'bcl2fastq' MultiQC module broke...
  Please copy the following traceback and report it at https://github.com/ewels/MultiQC/issues
  If possible, please include a log file that triggers the error - the last file found was:
    NB500904_0187/fastq/stats/Stats.json
============================================================
Module bcl2fastq raised an exception: Traceback (most recent call last):
  File "/software/UHTS/Analysis/MultiQC/1.7/bin/multiqc", line 440, in multiqc
    output = mod()
  File "/software/UHTS/Analysis/MultiQC/1.7/lib/python2.7/site-packages/multiqc/modules/bcl2fastq/bcl2fastq.py", line 29, in __init__
    self.parse_file_as_json(myfile)
  File "/software/UHTS/Analysis/MultiQC/1.7/lib/python2.7/site-packages/multiqc/modules/bcl2fastq/bcl2fastq.py", line 215, in parse_file_as_json
    lsample["R{}_yield".format(r)] += readMetric["Yield"]
KeyError: 'R3_yield'

File that triggers the error:
A Stats.json file from bcl2fastq in which R1, R2 and R3 were generated (for instance when R2 is a UMI)

An easy fix would be to add the variable for R3 in the module or to create the number of variable depending on the number of reads demultiplexed during the run. (One can also imagine runs for which only R1 was demultiplexed)

I can open a PR if the first fix is chosen, implementing the second fix would take more time I imagine...

[ewels]

Thanks for reporting this @sachalau! Is it possible to give an example Stats.json file which triggers this error? It would be good to have one in the MultiQC test data repo.

So, relevant code is here: https://github.com/ewels/MultiQC/blob/12f519cabd5cda344535b97186c46684722e555c/multiqc/modules/bcl2fastq/bcl2fastq.py#L209-L217

I guess just ignoring R3 / R4 etc is not an option, as in your case it's really R1 and R3 that you're interested in?

As I can't see us ever having more than 1-4 for this, I think your suggestion of adding R3 and R4 is a good idea. Then perhaps we can remove these batches of attributes if the ReadNumber isn't found?

It'll be quite a lot of copy & pasting to do this, but I think easier and clearer than doing it dynamically.

A PR would be great! I have quite the backlog of MultiQC work and will be going on parental leave soon, so probably won't have time to work on this for a while yet sorry.

Phil

[sachalau]

No worries I'll work on it then. I'll proceed to adding R3/4 and removing them if they are not found after parsing. I'll add the file for testing purposes.

[ewels]

Great, thanks!

[ewels]

Fixed in #912 and a0fa0d7