New module: Freyja #1903

Joon-Klaps · 2023-04-13T09:48:44Z

Adress Issue #1902
The nf-core/viralrecon pipeline is being extended with the Freyja module to analyse mixed SARS-CoV-2 samples (i.e wastewater samples)
PR: nf-core/viralrecon#375

This comment contains a description of changes (with reason)
CHANGELOG.md has been updated

There is example tool output for tools in the https://github.com/ewels/MultiQC_TestData repository or attached to this PR
- freyja_out.zip
Code is tested and works locally (including with --lint flag)
docs/README.md is updated with link to below
docs/modulename.md is created
Everything that can be represented with a plot instead of a table is a plot
Report sections have a description and help text (with self.add_section)
There aren't any huge tables with > 6 columns (explain reasoning if so)
Each table column has a different colour scale to its neighbour, which relates to the data (eg. if high numbers are bad, they're red)
Module does not do any significant computational work

vladsavelyev · 2023-08-30T12:44:40Z

multiqc/utils/search_patterns.yaml

@@ -345,6 +345,10 @@ flash/hist:
 flexbar:
  contents: "Flexbar - flexible barcode and adapter removal"
  shared: true
+freyja:
+  fn: "*.tsv"
+  contents: "summarized"


Feels not very specific - do you think more conditions can be added? Is it fair to expect that files are named *demix_outs.tsv?

I guess not...
Perhaps we could check more contents lines? Like require lineages and abundances?

Yeah definitely if its better to add those checks. I thought it would be enough because of the number of lines we specify.

multiqc/modules/freyja/freyja.py

vladsavelyev · 2023-08-30T12:59:57Z

multiqc/modules/freyja/freyja.py

+
+            # Percentages don't always add up to 1, show a warning if this is the case
+            if sum(d.values()) != 1:
+                log.warning(f"Freyja {s_name}: percentages don't sum to 1")


How often does this happen? In all the 3 test examples, the percentages do not sum to 1. If it happens frequently, we probably shouldn't be polluting the logs with these warnings.

I'm guessing it will always be the case unless amplicon or any other targeted sequencing approach is used.

Sorry, merged before I remembered that we didn't fully address this issue. I think I will remove this warning in a separate PR, or make it a debug message.

vladsavelyev · 2023-08-30T13:00:50Z

multiqc/modules/freyja/freyja.py

+        headers["Top_lineage_freyja"] = {
+            "title": "Top lineage (Freyja)",
+            "description": "The most abundant lineage in the sample",
+            "scale": "RdYlGn-rev",  # Not sure if this is the best scale


I think one of the categorical scales should be good here (like Set2, etc, see: https://multiqc.info/docs/development/plots/#table-colour-scales), though I can't make it work 🤔

Ah, it's not working because the color is applied to the bar, which has a width of zero here.

Okay, I pushed a change to use categorical colours instead, let me know if it looks good:
looks good
And would love @ewels opinion on this.

vladsavelyev · 2023-08-30T13:44:20Z

Thanks a lot for contributing this module! I added a few changes to the branch.

I don't have a strong opinion on this, but one of a bit unusual things for MultiQC is the use of a non-numerical column in the general stats (i.e., the most abundant lineage here). I'm interested in @ewels's take on it, whether we want it, and what would be the best color scale. The barplot pretty much shows the same information, but we want something to be in the general stats anyway, right?

I added the use of a categorial scale into this PR, but I don't like that the generated colors in the result don't match the colors used in the barplot; I'm going to look into whether I can make them match.

vladsavelyev · 2023-08-30T14:24:05Z

multiqc/modules/freyja/freyja.py

+
+                > **Note**: Lineage designation is based on the used WHO nomenclature, which could vary over time. 
+                """,
+            plot=bargraph.plot(self.freyja_data, None, pconfig),


It would be good to make sure the bars have the same sorting order, e.g. on bar 2 Omicron goes at the tail, and on the bar 3 it is in the head.

Sorting order was simple (pushed a change), however, making sure the barchart colors are the same as the general stats column colors is not obvious: for the column, more pastel colors look better, whereas on the barchart, the bright default highcharts colors look the best.

…add sorting

vladsavelyev · 2023-08-31T13:47:46Z

Stacked on top #2017 - do no merge until that PR is merged.

… than 1 (as it is almost always going to be the case), see #1903

… than 1 (as it is almost always going to be the case), see #1903 (#2022)

Joon-Klaps added 10 commits April 10, 2023 17:58

Initial setup, pre step 1

9746e5f

finishing step 1

55ec321

Step 2 part done, data parse for freyja needed

46f96e2

Step 2 done

834b4c9

step 3 & 4 done

4f11491

finishing step 6

1f167e1

Fixing bugs, missing called functions

eaa18f4

fixing linting

1981e5c

Keeping all linter's precomit checkers happy

26f36d6

Think they are prettier is happy now?

bea96b4

Joon-Klaps mentioned this pull request Apr 13, 2023

Adding new nf-core subworkflow freyja for weighted variant analysis nf-core/viralrecon#375

Merged

9 tasks

ewels and others added 8 commits May 21, 2023 23:20

Merge branch 'master' into freyja

b3d0478

Merge branch 'master' into freyja

3cf1346

Merge branch 'master' into freyja

a04b0bd

Update freyja.py, output form Freyja

a0bcb65

match collection schema

02f4b53

Merge branch 'master' into freyja

aaa68c5

Remove capital D descirption

90f7354

remove all capitals in md

cb4e568

vladsavelyev reviewed Aug 30, 2023

View reviewed changes

vladsavelyev mentioned this pull request Aug 30, 2023

Add Freyja demix test data MultiQC/test-data#272

Merged

vladsavelyev reviewed Aug 30, 2023

View reviewed changes

multiqc/modules/freyja/freyja.py Outdated Show resolved Hide resolved

vladsavelyev reviewed Aug 30, 2023

View reviewed changes

multiqc/modules/freyja/freyja.py Outdated Show resolved Hide resolved

vladsavelyev reviewed Aug 30, 2023

View reviewed changes

vladsavelyev added 2 commits August 30, 2023 15:24

Use categorical colors for the most abundant lineage

9614649

Bit more specific contents line

c4b1752

Merge branch 'master' into freyja

56a4e83

vladsavelyev requested a review from ewels August 30, 2023 13:45

vladsavelyev added the awaits-review Awaiting final review and merge. label Aug 30, 2023

vladsavelyev reviewed Aug 30, 2023

View reviewed changes

vladsavelyev and others added 5 commits August 30, 2023 17:00

Experiment with color, make the gen stat col same scale as bargraph, …

0420cd2

…add sorting

Add scale for highcharts colors and use it instead of hardcoding

4f11311

Fix

97d2e5c

Make Highcharts categorical. Allow passing color indices to get_color

d9ec893

Merge branch 'master' into freyja

dc9e7df

vladsavelyev changed the base branch from master to colors August 31, 2023 13:46

Merge branch 'colors' into freyja

1ad049e

vladsavelyev changed the base branch from colors to master August 31, 2023 15:57

vladsavelyev and others added 2 commits August 31, 2023 18:00

Merge branch 'master' into freyja

b850714

Refactor using color scale. Use plot_defaults

849f432

vladsavelyev self-requested a review August 31, 2023 16:35

vladsavelyev changed the title ~~Add new module Freyja~~ New module: Freyja Aug 31, 2023

vladsavelyev merged commit 33db316 into MultiQC:master Aug 31, 2023
11 checks passed

vladsavelyev added a commit that referenced this pull request Sep 1, 2023

refactor(freyja): remove warning about percentages summing up to less…

19f2064

… than 1 (as it is almost always going to be the case), see #1903

vladsavelyev mentioned this pull request Sep 1, 2023

refactor(freyja): remove warning about percentages summing up to <1 #2022

Merged

vladsavelyev added a commit that referenced this pull request Sep 1, 2023

refactor(freyja): remove warning about percentages summing up to less…

8e471b3

… than 1 (as it is almost always going to be the case), see #1903 (#2022)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New module: Freyja #1903

New module: Freyja #1903

Joon-Klaps commented Apr 13, 2023 •

edited

vladsavelyev Aug 30, 2023

vladsavelyev Aug 30, 2023

Joon-Klaps Aug 30, 2023

vladsavelyev Aug 30, 2023

Joon-Klaps Aug 30, 2023

vladsavelyev Sep 1, 2023

vladsavelyev Aug 30, 2023

vladsavelyev Aug 30, 2023

vladsavelyev Aug 30, 2023

Joon-Klaps Aug 30, 2023

vladsavelyev commented Aug 30, 2023

vladsavelyev Aug 30, 2023

vladsavelyev Aug 30, 2023

vladsavelyev commented Aug 31, 2023

New module: Freyja #1903

New module: Freyja #1903

Conversation

Joon-Klaps commented Apr 13, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladsavelyev commented Aug 30, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vladsavelyev commented Aug 31, 2023

Joon-Klaps commented Apr 13, 2023 •

edited