Versions API (#2092)

* Rewrite version checker to use new API endpoint

* Move version_check_url into the config, for local testing

* Version check URL

* Clean up

* Typo

* Parse CI env var

* Fix

* Catch timeout error

* UMI-tools: support `extract` command (#2296)

* UMI-tools: support extract command

* [automated] Update CHANGELOG.md

* Add barplot

* Barplot with extract stats

* Doc

* Changelog

* Lint

* Fix duplicated IDs in linting

* Merge conflict

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Infinite dmax or dmin fail JSON dump load in JavaScript (#2354)

* Infinite dmax or dmin fail JSON dump load in JavaScript

* [automated] Update CHANGELOG.md

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Fix changelog.py for module updates

* Whatshap: robust to stdout appended to TSV (#2361)

* Drop `full_figure_for_development` the call to avoid Kaleido issues (#2359)

* Wrap `full_figure_for_development` in try-except to handle Kaleido errors

* [automated] Update CHANGELOG.md

* Get around read-only /tmp

* Remove the full_figure_for_development call for good, just re-calcualte range from data

* Changelog

* Comments

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* PURPLE: support v4.0.1 output without `version` column (#2366)

* Purple: support v4.0.1 output without version column

* [automated] Update CHANGELOG.md

* [automated] Update CHANGELOG.md

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Ignore `.parquet` files. More robust `htseq` file reading loop (#2364)

* Ignore .parquet files. Handle htseq unicode errors

* [automated] Update CHANGELOG.md

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Add box plot (#2358)

* Add box plot

* [automated] Update CHANGELOG.md

* Fix linting

* CSP

* Fix sample order

* Allow overriding title for a dataset tab

* Violin: color dots blue in monochrome violins to indiciate interactivity

* Mono color for boxes, support highliting

* Rename beeswarm to violin

* Barplot: stacking: normal should mean stacked, not grouped

* Update plotting documentation

* Docs

* Redirect beeswarm

* CSP

* Update the flat plot section and remove highcharts mention

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Replace `setup.py` with `pyproject.toml` (#2353)

* Replace setup.py with pyproject.toml

* [automated] Update CHANGELOG.md

* Add myself as author

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Put matplotlib back into main deps (#2370)

* Put matplotlib back into main deps

* [automated] Update CHANGELOG.md

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

* Generic font family for Plotly (#2368)

* Generic font family for Plotly

* [automated] Update CHANGELOG.md

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Samtools `coverage` submodule (#2356)

* Allow override title per-dataset

* Add samtools coverage submodule

* [automated] Update CHANGELOG.md

* Fix changelog

* Update multiqc/modules/samtools/rmdup.py

Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Fix rmdup regex

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

* Update `modules.md`: `pyproject.toml`, `"shared_key": "read_count"`, ruffify (#2371)

* Update modules.md: `pyproject.toml`, `"shared_key": "read_count"`, ruffify

* [automated] Update CHANGELOG.md

* Apply suggestions from code review

---------

Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>
Co-authored-by: Phil Ewels <phil.ewels@seqera.io>

---------

Co-authored-by: Vlad Savelyev <vladislav.sav@gmail.com>
Co-authored-by: MultiQC Bot <multiqc-bot@seqera.io>

README.md

@@ -24,7 +24,7 @@ it ideal for routine fast quality control.
 A very large number of Bioinformatics tools are supported by MultiQC. Please see the MultiQC website for a [complete list](https://multiqc.info/modules/).
 MultiQC can also easily parse data from custom scripts, if correctly formatted / configured - a feature called [Custom Content](https://multiqc.info/docs/custom_content/).
 
-More modules are being written all of the time. Please suggest any ideas as a new
+More modules are being written all the time. Please suggest any ideas as a new
 [issue](https://github.com/MultiQC/MultiQC/issues) _(please include example log files)_.
 
 ## Installation

multiqc/multiqc.py

@@ -11,6 +11,7 @@
 import errno
 import io
 import os
+import platform
 import re
 import shutil
 import subprocess
@@ -19,9 +20,9 @@
 import time
 import json
 import traceback
-from urllib.request import urlopen
 
 import jinja2
+import requests
 import rich
 import rich_click as click
 from packaging import version
@@ -30,6 +31,7 @@
 from .modules.base_module import ModuleNoSamplesFound
 from .plots import table
 from .utils import config, log, megaqc, plugin_hooks, report, software_versions, strict_helpers, util_functions
+from .utils.util_functions import strtobool
 
 # Set up logging
 start_execution_time = time.time()
@@ -404,14 +406,43 @@ def run(
     # Check that we're running the latest version of MultiQC
     if config.no_version_check is not True:
         try:
-            response = urlopen(f"http://multiqc.info/version.php?v={config.short_version}", timeout=5)
-            remote_version = response.read().decode("utf-8").strip()
-            if version.parse(re.sub(r"[^0-9.]", "", remote_version)) > version.parse(
-                re.sub(r"[^0-9.]", "", config.short_version)
-            ):
-                logger.warning(f"MultiQC Version {remote_version} now available!")
+            # Fetch the version info from the API
+            meta = {
+                "version_multiqc": config.short_version,
+                "version_python": platform.python_version(),
+                "operating_system": platform.system(),
+                "is_docker": os.path.exists("/.dockerenv"),
+                "is_singularity": os.path.exists("/.singularity.d"),
+                "is_conda": os.path.exists(os.path.join(sys.prefix, "conda-meta")),
+                "is_ci": strtobool(os.getenv("CI", False)),
+            }
+            wait_seconds = 2
+            try:
+                r = requests.get(config.version_check_url, params=meta, timeout=wait_seconds)
+            except requests.exceptions.Timeout as e:
+                logger.debug(
+                    f"Timed out after waiting for {wait_seconds}s for multiqc.info to check latest version: {e}"
+                )
+            except requests.exceptions.RequestException as e:
+                logger.debug(f"Could not connect to multiqc.info for version check: {e}")
             else:
-                logger.debug(f"Latest MultiQC version is {remote_version}")
+                release_info = r.json()
+                # Broadcast log messages if found
+                for msg in release_info.get("broadcast_messages", []):
+                    if msg.get("message"):
+                        level = msg.get("level")
+                        if level not in ["debug", "info", "warning", "error", "critical"]:
+                            level = "info"
+                        getattr(logger, level)(msg["message"])
+                # Available update log if newer
+                remove_version = version.parse(re.sub(r"[^0-9.]", "", release_info["latest_release"]["version"]))
+                this_version = version.parse(re.sub(r"[^0-9.]", "", config.short_version))
+                if remove_version > this_version:
+                    logger.warning(f"MultiQC Version {release_info['latest_release']['version']} now available!")
+                logger.debug(
+                    f"Latest MultiQC version is {release_info['latest_release']['version']}, "
+                    f"released {release_info['latest_release']['release_date']}"
+                )
         except Exception as e:
             logger.debug(f"Could not connect to multiqc.info for version check: {e}")
 

multiqc/utils/config.py

@@ -79,6 +79,7 @@
 prepend_dirs_sep: str
 file_list: bool
 require_logs: bool
+version_check_url: str
 
 make_data_dir: bool
 zip_data_dir: bool

multiqc/utils/config_defaults.yaml

@@ -47,6 +47,7 @@ prepend_dirs_depth: 0
 prepend_dirs_sep: " | "
 file_list: false
 require_logs: false
+version_check_url: https://api.multiqc.info/version
 
 make_data_dir: true
 zip_data_dir: false

multiqc/utils/util_functions.py

@@ -165,10 +165,10 @@ def strtobool(val) -> bool:
     are 'n', 'no', 'f', 'false', 'off', and '0'.  Raises ValueError if
     'val' is anything else.
     """
-    val = val.lower()
-    if val in ("y", "yes", "t", "true", "on", "1"):
+    val_str = str(val).lower()
+    if val_str in ("y", "yes", "t", "true", "on", "1"):
         return True
-    elif val in ("n", "no", "f", "false", "off", "0"):
+    elif val_str in ("n", "no", "f", "false", "off", "0"):
         return False
     else:
         raise ValueError(f"invalid truth value {val!r}")