Merge branch 'main' into ngsbits-2

.github/RELEASE_CHECKLIST.md

@@ -2,10 +2,10 @@
 
 This checklist is for my own reference, as I forget the steps every time.
 
-1. Check that everything is up to date and ready to go
+1. Check that everything is up-to-date and ready to go
 2. Figure out what this release should be called
-3. Update version numbers in code: `setup.py`, `CHANGELOG.md`
-4. Link the changelog subheading to the as yet non-existant release URL. Add date.
+3. Update version numbers in code: `pyproject.toml`, `CHANGELOG.md`
+4. Link the changelog subheading to the as yet non-existent release URL. Add date.
 5. Install the package again in `install` mode:
 
    ```bash
@@ -59,11 +59,12 @@ This checklist is for my own reference, as I forget the steps every time.
 12. Check that [PyPI listing page](https://pypi.python.org/pypi/multiqc/) looks sane
 13. Update version numbers to new dev version in `setup.py` + a new section in the changelog for the development version
 14. Commit and push version bump
-15. Look for the automated release PR on `bioconda` and approve / merge
+15. Run Seqera `#product-releases` [Slack workflow](https://slack.com/shortcuts/Ft06GYSX4UUB/c3733786a0ad2fc1794d1959aed5df19)
+16. Look for the automated release PR on `bioconda` and approve / merge
     - IMPORTANT: If any new dependencies added, need a manual PR to add them.
     - Do this quickly, as other people merge the automated PRs really quickly
-16. Tweet that new version is released
-17. Continue making more awesome :metal:
+17. Tweet that new version is released
+18. Continue making more awesome :metal:
 
 ## Appendix
 

.github/workflows/changelog.py

@@ -44,7 +44,7 @@
 
 changelog_path = workspace_path / "CHANGELOG.md"
 
-if any(
+no_changelog = any(
     line in pr_title.lower()
     for line in [
         "skip changelog",
@@ -53,9 +53,7 @@
         "no change log",
         "bump version",
     ]
-):
-    print("Skipping changelog update")
-    sys.exit(0)
+)
 
 
 def _run_cmd(cmd):
@@ -258,30 +256,35 @@ def _determine_change_type(pr_title, pr_number) -> tuple[str, dict]:
     return section, {}
 
 
-# Determine the type of the PR: new module, module update, or core update.
-section, mod = _determine_change_type(pr_title, pr_number)
+pr_link = f"([#{pr_number}]({REPO_URL}/pull/{pr_number}))"
 
 # Prepare the change log entry.
 new_lines = []
-pr_link = f"([#{pr_number}]({REPO_URL}/pull/{pr_number}))"
-if comment := comment.removeprefix("@multiqc-bot changelog").strip():
-    pr_title = comment
-else:
-    if section == "### New modules":
-        new_lines = [
-            f"- [**{mod['name']}**]({mod['url']}) {pr_link}\n",
-            f"  - {mod['name']} {mod['info']}\n",
-        ]
-    elif section == "### Module updates":
-        assert mod is not None
-        descr = pr_title.split(":", maxsplit=1)[1].strip()
+section = None
+if not no_changelog:
+    # Determine the type of the PR: new module, module update, or core update.
+    section, mod = _determine_change_type(pr_title, pr_number)
+
+    if comment := comment.removeprefix("@multiqc-bot changelog").strip():
+        pr_title = comment
+    else:
+        if section == "### New modules":
+            new_lines = [
+                f"- [**{mod['name']}**]({mod['url']}) {pr_link}\n",
+                f"  - {mod['name']} {mod['info']}\n",
+            ]
+        elif section == "### Module updates":
+            assert mod is not None
+            descr = pr_title
+            if ":" in descr:
+                descr = descr.split(":", maxsplit=1)[1].strip()
+            new_lines = [
+                f"- **{mod['name']}**: {descr} {pr_link}\n",
+            ]
+    if not new_lines:
         new_lines = [
-            f"- **{mod['name']}**: {descr} {pr_link}\n",
+            f"- {pr_title} {pr_link}\n",
         ]
-if not new_lines:
-    new_lines = [
-        f"- {pr_title} {pr_link}\n",
-    ]
 
 # Finally, updating the changelog.
 # Read the current changelog lines. We will print them back as is, except for one new
@@ -330,6 +333,10 @@ def _skip_existing_entry_for_this_pr(line, same_section=True):
     # If the line already contains a link to the PR, don't add it again.
     line = _skip_existing_entry_for_this_pr(line, same_section=False)
 
+    if no_changelog:  # do not add anything, and remove if already added
+        updated_lines.append(line)
+        continue
+
     if line.startswith("## "):  # Version header, e.g. "## MultiQC v1.10dev"
         updated_lines.append(line)
 

.github/workflows/screenshots.yml

@@ -11,6 +11,8 @@ jobs:
 
       - name: Set up Python
         uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
 
       - name: Install MultiQC
         run: pip install .

.gitignore

@@ -37,7 +37,8 @@ dist/
 ehthumbs.db
 Thumbs.db
 
-.idea/
+.idea
+.vscode
 runTest.sh
 
 # Prevent direnv from cluttering the repo

CHANGELOG.md

@@ -1,33 +1,136 @@
 # MultiQC Version History
 
-## MultiQC v1.21dev
+## MultiQC v1.22dev
 
 ### MultiQC updates
 
-- Warn if `run_modules` contains a non-existent module ([#2322](https://github.com/MultiQC/MultiQC/pull/2322))
-- Group-mode bar plot: fix inner gap ([#2321](https://github.com/MultiQC/MultiQC/pull/2321))
-- Pin Plotly package and add a runtime version check ([#2325](https://github.com/MultiQC/MultiQC/pull/2325))
-- Always create JSON even when MegaQC upload disabled ([#2330](https://github.com/MultiQC/MultiQC/pull/2330))
-- Fix getting id from optional pconfig=None ([#2337](https://github.com/MultiQC/MultiQC/pull/2337))
-- Barplot: keep sample order ([#2339](https://github.com/MultiQC/MultiQC/pull/2339))
-- MegaQC: dump `pconfig` ([#2344](https://github.com/MultiQC/MultiQC/pull/2344))
-- Catch non-hashable values in table data ([#2348](https://github.com/MultiQC/MultiQC/pull/2348))
-- Prevent parsing numerical sample names in heatmap ([#2349](https://github.com/MultiQC/MultiQC/pull/2349))
-
 ### New modules
 
 - [**ngs-bits**](https://github.com/imgag/ngs-bits) ([#2231](https://github.com/ewels/MultiQC/pull/2231))
   - ngs-bits is a collection of short-read sequencing tools.
 
 ### Module updates
 
+## [MultiQC v1.21](https://github.com/ewels/MultiQC/releases/tag/v1.21) - 2024-02-28
+
+### Highlights
+
+#### Box plot
+
+Added a new plot type: box plot. It's useful to visualise a distribution when you have a set of values for each sample.
+
+```py
+from multiqc.plots import box
+self.add_section(
+    ...,
+    plot=box.plot(
+        {
+            "sample 1": [4506, 4326, 3137, 1563, 1730, 3254, 2259, 3670, 2719, ...],
+            "sample 2": [2145, 2011, 3368, 2132, 1673, 1993, 6635, 1635, 4984, ...],
+            "sample 3": [1560, 1845, 3247, 1701, 2829, 2775, 3179, 1724, 1828, ...],
+        },
+        pconfig={
+            "title": "Iso-Seq: Insert Length",
+        },
+    )
+)
+```
+
+<img width="400" src="https://raw.githubusercontent.com/MultiQC/MultiQC/1e2ad5529d547ab9dc04c99274da49ad6a2e556f/docs/images/changelog/v1.21-boxplot.png">
+
+Note the difference with the violin plot: the box plot visualises the distributions of many values within one sample, whereas the violin plot shows the distribution of one metric across many samples.
+
+#### `pyproject.toml`
+
+The `setup.py` file has been superseded by `pyproject.toml` for the build configuration.
+Note that now for new modules, an entry point should be added to `pyproject.toml` instead of `setup.py`, e.g.:
+
+```toml
+[project.entry-points."multiqc.modules.v1"]
+afterqc = "multiqc.modules.afterqc:MultiqcModule"
+```
+
+#### Heatmap
+
+The heatmap plot now supports passing a dict as input data, and also supports a `zlab`
+parameter to set the label for the z-axis:
+
+```py
+from multiqc.plots import heatmap
+self.add_section(
+    ...,
+    plot=heatmap.plot(
+        {
+            "sample 1": {"sample 2": 0, "sample 3": 1},
+            "sample 2": {"sample 1": 0, "sample 3": 0},
+            "sample 3": {"sample 1": 1, "sample 2": 0, "sample 3": 1},
+        },
+        pconfig={
+            "title": "Sample comparison",
+            "zlab": "Match",
+        },
+    )
+)
+```
+
+### MultiQC updates
+
+- New plot type: box plot ([#2358](https://github.com/MultiQC/MultiQC/pull/2358))
+- Add "Export to CSV" button for tables ([#2394](https://github.com/MultiQC/MultiQC/pull/2394))
+- Replace `setup.py` with `pyproject.toml` ([#2353](https://github.com/MultiQC/MultiQC/pull/2353))
+- Heatmap: allow a dict dicts of data ([#2386](https://github.com/MultiQC/MultiQC/pull/2386))
+- Heatmap: add `zlab` config parameter. Show `xlab`, `ylab`, `zlab` in tooltip ([#2387](https://github.com/MultiQC/MultiQC/pull/2387))
+- Warn if `run_modules` contains a non-existent module ([#2322](https://github.com/MultiQC/MultiQC/pull/2322))
+- Catch non-hashable values (dicts, lists) passed as a table cell value ([#2348](https://github.com/MultiQC/MultiQC/pull/2348))
+- Always create JSON even when MegaQC upload is disabled ([#2330](https://github.com/MultiQC/MultiQC/pull/2330))
+- Use generic font family for Plotly ([#2368](https://github.com/MultiQC/MultiQC/pull/2368))
+- Use a padded span with `nowrap` instead of `&nbsp;` before suffixes in table cells ([#2395](https://github.com/MultiQC/MultiQC/pull/2395))
+- Refactor: fix unescaped regex strings ([#2384](https://github.com/MultiQC/MultiQC/pull/2384))
+
+Fixes:
+
+- Pin the required Plotly version and add a runtime version check ([#2325](https://github.com/MultiQC/MultiQC/pull/2325))
+- Bar plot: preserve the sample order ([#2339](https://github.com/MultiQC/MultiQC/pull/2339))
+- Bar plot: fix inner gap in group mode ([#2321](https://github.com/MultiQC/MultiQC/pull/2321))
+- Violin: filter `Inf` values ([#2380](https://github.com/MultiQC/MultiQC/pull/2380))
+- Table: Fix use of the `no_violin` (ex-`no_beeswarm`) table config flag ([#2376](https://github.com/MultiQC/MultiQC/pull/2376))
+- Heatmap: prevent from parsing numerical sample names ([#2349](https://github.com/MultiQC/MultiQC/pull/2349))
+- Work around call of `full_figure_for_development` to avoid Kaleido errors ([#2359](https://github.com/MultiQC/MultiQC/pull/2359))
+- Auto-generate plot `id` when `pconfig=None` ([#2337](https://github.com/MultiQC/MultiQC/pull/2337))
+- Fix: infinite `dmax` or `dmin` fail JSON dump load in JavaScript ([#2354](https://github.com/MultiQC/MultiQC/pull/2354))
+- Fix: dump `pconfig` for MegaQC ([#2344](https://github.com/MultiQC/MultiQC/pull/2344))
+
+### New modules
+
+- [**IsoSeq**](https://github.com/PacificBiosciences/IsoSeq)
+  - Iso-Seq contains the newest tools to identify transcripts in PacBio single-molecule sequencing data (HiFi reads). `cluster` and `refine` commands are supported.
+- [**Space Ranger**](https://support.10xgenomics.com/spatial-gene-expression/software/pipelines/latest/what-is-space-ranger)
+  - Works with data from 10X Genomics Visium. Processes sequencing reads and images created using
+    the 10x Visium platform to generate count matrices with spatial information.
+  - New MultiQC module parses Space Ranger quality reports.
+
+### Module updates
+
 - **bcl2fastq**: fix the top undetermined barcodes plot ([#2340](https://github.com/MultiQC/MultiQC/pull/2340))
 - **DRAGEN**: add few coverage metrics in general stats ([#2341](https://github.com/MultiQC/MultiQC/pull/2341))
 - **DRAGEN**: fix showing the number of found samples ([#2347](https://github.com/MultiQC/MultiQC/pull/2347))
 - **DRAGEN**: support `gvcf_metrics` ([#2327](https://github.com/MultiQC/MultiQC/pull/2327))
-- **fastp**: Improve detection of JSON files ([#2334](https://github.com/MultiQC/MultiQC/pull/2334))
+- **fastp**: fix detection of JSON files ([#2334](https://github.com/MultiQC/MultiQC/pull/2334))
+- **HTSeq Count**: robust file reading loop, ignore `.parquet` files ([#2364](https://github.com/MultiQC/MultiQC/pull/2364))
 - **Illumina InterOp Statistics**: do not set `'scale': False` as a default ([#2350](https://github.com/MultiQC/MultiQC/pull/2350))
-- **mosdepth**: Fix regression in showing general stats ([#2346](https://github.com/MultiQC/MultiQC/pull/2346))
+- **mosdepth**: fix regression in showing general stats ([#2346](https://github.com/MultiQC/MultiQC/pull/2346))
+- **Picard**: Crosscheck Fingerprints updates ([#2388](https://github.com/MultiQC/MultiQC/pull/2388))
+  - add a heatmap for LOD scores besides a table
+  - if too many pairs in table, skip those with `Expected` status
+  - use the `warn` status for `Inconclusive`
+  - add a separate sample-wise table instead of general stats
+  - sort tables by status, not by sample name
+  - add a column "Best match" and "Best match LOD" in tables
+  - hide the LOD Threshold column
+- **PURPLE**: support v4.0.1 output without `version` column ([#2366](https://github.com/MultiQC/MultiQC/pull/2366))
+- **Samtools**: support new `coverage` command ([#2356](https://github.com/MultiQC/MultiQC/pull/2356))
+- **UMI-tools**: support new `extract` command ([#2296](https://github.com/MultiQC/MultiQC/pull/2296))
+- **Whatshap**: make robust when a stdout is appended to TSV ([#2361](https://github.com/MultiQC/MultiQC/pull/2361))
 
 ## [MultiQC v1.20](https://github.com/ewels/MultiQC/releases/tag/v1.20) - 2024-02-12
 
@@ -102,7 +205,6 @@ The v1.20 release is also the first release we've had since we moved the MultiQC
 - **BCL Convert**: add index, project names to sample statistics and calculate mean quality for lane statistics. ([#2261](https://github.com/MultiQC/MultiQC/pull/2261))
 - **BCL Convert**: fix duplicated `yield` for 3.9.3+ when the yield is provided explicitly in Quality_Metrics ([#2253](https://github.com/MultiQC/MultiQC/pull/2253))
 - **BCL Convert**: handle samples with zero yield ([#2297](https://github.com/MultiQC/MultiQC/pull/2297))
-- **Bismark**: fix old link in Bismark docs ([#2252](https://github.com/MultiQC/MultiQC/pull/2252))
 - **Bismark**: fix old link in docs ([#2252](https://github.com/MultiQC/MultiQC/pull/2252))
 - **Cutadapt**: support JSON format ([#2281](https://github.com/MultiQC/MultiQC/pull/2281))
 - **HiFiasm**: account for lines with no asterisk ([#2268](https://github.com/MultiQC/MultiQC/pull/2268))

CSP.txt

@@ -4,6 +4,12 @@
 script-src 'self'
     # 1.21
     'sha256-bHPK47f6TsuMbS8pR8jVPzEK4orYin9an6P5OkYf+n8=' # class BarPlot extends Plot {  constructor(dump) {    super(dump);    this.filter
+    'sha256-yRjvY4tSJ5z2ggAL/i5pJj+peALSAG9PD51Z0Hf7wzQ=' # ////////////////////////////////////////////////// Plotly Plotting Code/////////
+    'sha256-gf6XBM6wOXcVJjumRhlnj7J/GEpeABZmKQla1g2feYk=' # class BoxPlot extends Plot {  constructor(dump) {    super(dump);    this.filter
+    'sha256-hT8GW76gPbw230zK0dy6dUPxVfhN9QKm3T8EyNblQjw=' # class ViolinPlot extends Plot {  constructor(dump) {    super(dump);    this.vio
+    'sha256-DrHBQbs5duODlWwXxhFtOOpsMfANjqepi9PBfA4om/8=' # ////////////////////////////////////////////////// MultiQC Table code///////////
+    'sha256-skoLPw9W6GGVScPkzbYwz7laPwL8dOpznqFPCoUHVxA=' # ////////////////////////////////////////////////// MultiQC Report Toolbox Code//
+    'sha256-ZAR6nlDF3X1oyyS2SMzxupvbW+MVuc+OzhyS2AqJmbY=' # ////////////////////////////////////////////////// MultiQC Report Toolbox Code//
 
     # 1.20
     'sha256-tTUP7XMWeShHDbCfkNfh2MTlcpIsSP6ybV/ChnAPZyo=' # ////////////////////////////////////////////////// Base JS for MultiQC Reports//

README.md

@@ -1,6 +1,6 @@
  
-![MultiQC](docs/images/MultiQC_logo.png#gh-light-mode-only)
-![MultiQC](docs/images/MultiQC_logo_darkbg.png#gh-dark-mode-only)
+![MultiQC](https://github.com/MultiQC/MultiQC/raw/main/docs/images/MultiQC_logo.png#gh-light-mode-only)
+![MultiQC](https://github.com/MultiQC/MultiQC/raw/main/docs/images/MultiQC_logo_darkbg.png#gh-dark-mode-only)
  
 
 ### Aggregate bioinformatics results across many samples into a single report
@@ -24,7 +24,7 @@ it ideal for routine fast quality control.
 A very large number of Bioinformatics tools are supported by MultiQC. Please see the MultiQC website for a [complete list](https://multiqc.info/modules/).
 MultiQC can also easily parse data from custom scripts, if correctly formatted / configured - a feature called [Custom Content](https://multiqc.info/docs/custom_content/).
 
-More modules are being written all of the time. Please suggest any ideas as a new
+More modules are being written all the time. Please suggest any ideas as a new
 [issue](https://github.com/MultiQC/MultiQC/issues) _(please include example log files)_.
 
 ## Installation
@@ -67,7 +67,7 @@ and produce a report detailing whatever it finds.
 
 <!-- RICH-CODEX fake_command: "multiqc ." -->
 
-![`cd test_data/data/modules/fastqc/v0.10.1 && multiqc .`](docs/images/screenshots/fastqc-run.svg)
+![`cd test_data/data/modules/fastqc/v0.10.1 && multiqc .`](https://github.com/MultiQC/MultiQC/raw/main/docs/images/screenshots/fastqc-run.svg)
 
 The report is created in `multiqc_report.html` by default. Tab-delimited data
 files are also created in `multiqc_data/`, containing extra information.