nf-core · mashehu · Nov 8, 2025 · Nov 8, 2025 · Nov 8, 2025 · Nov 8, 2025
diff --git a/.github/actions/nf-test/action.yml b/.github/actions/nf-test/action.yml
@@ -68,7 +68,7 @@ runs:
           --changed-since HEAD^ \
           --verbose \
           --tap=test.tap \
-          --shard ${{ inputs.shard }}/${{ inputs.total_shards }}
+          --shard ${{ inputs.shard }}/${{ inputs.total_shards }} --debug
 
           # Save the absolute path of the test.tap file to the output
           echo "tap_file_path=$(realpath test.tap)" >> $GITHUB_OUTPUT

diff --git a/.github/workflows/nf-test.yml b/.github/workflows/nf-test.yml
@@ -50,7 +50,7 @@ jobs:
         env:
           NFT_VER: ${{ env.NFT_VER }}
         with:
-          max_shards: 7
+          max_shards: 10
 
       - name: debug
         run: |

diff --git a/.gitignore b/.gitignore
@@ -7,3 +7,14 @@ testing/
 testing*
 *.pyc
 null/
+.nf-test*
+.idea/
+.vscode/
+taggers/
+tokenizers/
+corpora/
+.github/act.custom_runner.Dockerfile
+.ruff_cache
+galaxy/test_output/
+TODO
+test/
diff --git a/.nf-core.yml b/.nf-core.yml
@@ -6,16 +6,22 @@ lint:
     - conf/igenomes_ignored.config
   files_unchanged:
     - assets/nf-core-stableexpression_logo_light.png
+    - docs/images/nf-core-stableexpression_logo_light.png
+    - docs/images/nf-core-stableexpression_logo_dark.png
     - .github/PULL_REQUEST_TEMPLATE.md
   nextflow_config:
     - params.input
+  template_strings:
+    - tests/test_data/genorm/compute_m_measure/input/std.0.0.parquet
+    - tests/test_data/genorm/compute_m_measure/input/std.1.2.parquet
+    - tests/test_data/genorm/compute_m_measure/input/std.1.2.parquet
   schema_lint: false
-nf_core_version: 3.5.1
+
+nf_core_version: 3.5.2
 repository_type: pipeline
 template:
   author: Olivier Coen
-  description: This pipeline is dedicated to finding the most stable genes across
-    count datasets
+  description: This pipeline is dedicated to identifying the most stable genes within a single or multiple expression dataset(s). This is particularly useful for identifying the most suitable RT-qPCR reference genes for a specific species.
   force: false
   is_nfcore: true
   name: stableexpression
@@ -24,4 +30,4 @@ template:
   skip_features:
     - igenomes
     - fastqc
-  version: 1.0dev
+  version: 1.0.0
diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml
@@ -1,10 +1,12 @@
 repos:
   - repo: https://github.com/pre-commit/mirrors-prettier
-    rev: "v3.1.0"
+    rev: "v4.0.0-alpha.8"
     hooks:
       - id: prettier
         additional_dependencies:
           - prettier@3.6.2
+        exclude: galaxy/
+
   - repo: https://github.com/pre-commit/pre-commit-hooks
     rev: v6.0.0
     hooks:
@@ -25,3 +27,15 @@ repos:
               subworkflows/nf-core/.*|
               .*\.snap$
           )$
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.14.1
+    hooks:
+      # Run the linter.
+      - id: ruff
+        files: \.py$
+        args: [--fix]
+        exclude: bin/old/
+      # Run the formatter.
+      - id: ruff-format
+        files: \.py$
diff --git a/.prettierignore b/.prettierignore
@@ -14,3 +14,6 @@ bin/
 ro-crate-metadata.json
 modules/nf-core/
 subworkflows/nf-core/
+galaxy/
+docs/
+tests/act
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -3,9 +3,13 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## v1.0dev - [date]
+## v1.0.0 - 18/03/2026
 
-Initial release of nf-core/stableexpression, created with the [nf-core](https://nf-co.re/) template.
+First complete, official release of nf-core/stableexpression.
+
+## v1.0dev - 26/01/2025
+
+Initial pre-release of nf-core/stableexpression, created with the [nf-core](https://nf-co.re/) template.
 
 ### `Added`
 

diff --git a/CITATIONS.md b/CITATIONS.md
@@ -10,6 +10,26 @@
 
 ## Pipeline tools
 
+- [EBI Expression Atlas](https://www.ebi.ac.uk/gxa/home)
+
+> Papatheodorou I, Fonseca NA, Keays M, Tang YA, Barrera E, Bazant W, Burke M, Füllgrabe A, Muñoz-Pomer Fuentes A, George N, Huerta L, Koskinen S, Mohammed S, Geniza M, Preece J, Jaiswal P, Jarnuczak AF, Huber W, Stegle O, Vizcaino JA, Brazma A, Petryszak R. Expression Atlas: gene and protein expression across multiple studies and organisms. Nucleic Acids Res. 2017 Nov 20;46(Database issue):D246–D251. doi: 10.1093/nar/gkx1158. PubMed PMID: 29165655.
+
+- [NCBI GEO](https://www.ncbi.nlm.nih.gov/geo/)
+
+> Ron Edgar, Michael Domrachev & Alex E Lash. Gene Expression Omnibus: NCBI gene expression and hybridization array data repository. Nucleic Acids Res. 2002 Jan 1;30(1):207-10. doi: 10.1093/nar/30.1.207. PubMed PMID: 11752295.
+
+- [g:Profiler](https://biit.cs.ut.ee/gprofiler/gost)
+
+> Reimand J, Kull M, Peterson H, Hansen J, Vilo J. g:Profiler—a web-based toolset for functional profiling of gene lists from large-scale experiments. Nucleic Acids Res. 2007 May 3;35(Web Server issue):W193–W200. doi:10.1093/nar/gkm226. PubMed PMID: 17478515.
+
+- [Normfinder](https://rdrr.io/github/dhammarstrom/generefer/man/normfinder.html)
+
+> Claus Lindbjerg Andersen, Jens Ledet Jensen, Torben Falck Ørntoft. Normalization of Real-Time Quantitative Reverse Transcription-PCR Data: A Model-Based Variance Estimation Approach to Identify Genes Suited for Normalization, Applied to Bladder and Colon Cancer Data Sets. Cancer Res (2004) 64 (15): 5245–5250. doi:10.1158/0008-5472.CAN-04-0496. PubMed PMID: 15289330.
+
+- [GeNorm](https://pypi.org/project/rna-genorm/)
+
+> Jo Vandesompele, Katleen De Preter, Filip Pattyn, Bruce Poppe, Nadine Van Roy, Anne De Paepe, Frank Speleman. Accurate normalization of real-time quantitative RT-PCR data by geometric averaging of multiple internal control genes. Genome Biol. 2002 Jun 18;3(7):RESEARCH0034. doi: 10.1186/gb-2002-3-7-research0034 Pubmed PMID: 12184808.
+
 - [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/)
 
 > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.

diff --git a/README.md b/README.md
@@ -12,6 +12,7 @@
 
 [![Nextflow](https://img.shields.io/badge/version-%E2%89%A525.04.0-green?style=flat&logo=nextflow&logoColor=white&color=%230DC09D&link=https%3A%2F%2Fnextflow.io)](https://www.nextflow.io/)
 [![nf-core template version](https://img.shields.io/badge/nf--core_template-3.5.1-green?style=flat&logo=nfcore&logoColor=white&color=%2324B064&link=https%3A%2F%2Fnf-co.re)](https://github.com/nf-core/tools/releases/tag/3.5.1)
+[![run with apptainer](https://custom-icon-badges.demolab.com/badge/run%20with-apptainer-4545?logo=apptainer&color=teal&labelColor=000000)](https://apptainer.org/)
 [![run with conda](http://img.shields.io/badge/run%20with-conda-3EB049?labelColor=000000&logo=anaconda)](https://docs.conda.io/en/latest/)
 [![run with docker](https://img.shields.io/badge/run%20with-docker-0db7ed?labelColor=000000&logo=docker)](https://www.docker.com/)
 [![run with singularity](https://img.shields.io/badge/run%20with-singularity-1d355c.svg?labelColor=000000)](https://sylabs.io/docs/)
@@ -21,68 +22,144 @@
 
 ## Introduction
 
-**nf-core/stableexpression** is a bioinformatics pipeline that ...
+**nf-core/stableexpression** is a bioinformatics pipeline aiming to aggregate multiple count datasets for a specific species and find the most stable genes. The datasets can be either downloaded from public databases (EBI, NCBI) or provided directly by the user. Both RNA-seq and Microarray count datasets can be utilised.
 
-<!-- TODO nf-core:
-   Complete this sentence with a 2-3 sentence summary of what types of data the pipeline ingests, a brief overview of the
-   major pipeline sections and the types of output it produces. You're giving an overview to someone new
-   to nf-core here, in 15-20 seconds. For an example, see https://github.com/nf-core/rnaseq/blob/master/README.md#introduction
--->
+<p align="center">
+    <img title="Stableexpression Workflow" src="docs/images/nf_core_stableexpression.metromap.png" width=100%>
+</p>
 
-<!-- TODO nf-core: Include a figure that guides the user through the major workflow steps. Many nf-core
-     workflows use the "tube map" design for that. See https://nf-co.re/docs/guidelines/graphic_design/workflow_diagrams#examples for examples.   -->
-<!-- TODO nf-core: Fill in short bullet-pointed list of the default steps in the pipeline -->2. Present QC for raw reads ([`MultiQC`](http://multiqc.info/))
+It takes as main inputs :
 
-## Usage
+- a species name (mandatory)
+- keywords for Expression Atlas / GEO search (optional)
+- a CSV input file listing your own raw / normalised count datasets (optional).
+
+**Use cases**:
+
+- **find the most suitable genes as RT-qPCR reference genes for a specific species (and optionally specific conditions)**
+- download all Expression Atlas and / or NCBI GEO datasets for a species (and optionally keywords)
+
+## Pipeline overview
+
+The pipeline is built using [Nextflow](https://www.nextflow.io/) and processes data using the following steps:
+
+#### 1. Get accessions from public databases
+
+- Get [Expression Atlas](https://www.ebi.ac.uk/gxa/home) dataset accessions corresponding to the provided species (and optionally keywords)
+  This step is run by default but is optional. Set `--skip_fetch_eatlas_accessions` to skip it.
+- Get NBCI [GEO](https://www.ncbi.nlm.nih.gov/gds) **microarray** dataset accessions corresponding to the provided species (and optionally keywords)
+  This is optional and **NOT** run by default. Set `--fetch_geo_accessions` to run it.
+
+#### 2. Download data (see [usage](./conf/usage.md#3-provide-your-own-accessions))
+
+- Download [Expression Atlas](https://www.ebi.ac.uk/gxa/home) data if any
+- Download NBCI [GEO](https://www.ncbi.nlm.nih.gov/gds) data if any
 
 > [!NOTE]
-> If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/usage/installation) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline) with `-profile test` before running the workflow on actual data.
+> At this point, datasets downloaded from public databases are merged with datasets provided by the user using the `--datasets` parameter. See [usage](./conf/usage.md#4-use-your-own-expression-datasets) for more information about local datasets.
 
-<!-- TODO nf-core: Describe the minimum required steps to execute the pipeline, e.g. how to prepare samplesheets.
-     Explain what rows and columns represent. For instance (please edit as appropriate):
+#### 3. ID Mapping (see [usage](./conf/usage.md#5-custom-gene-id-mapping--metadata))
 
-First, prepare a samplesheet with your input data that looks as follows:
+- Gene IDs are cleaned
+- Map gene IDS to NCBI Entrez Gene IDS (or Ensembl IDs) for standardisation among datasets using [g:Profiler](https://biit.cs.ut.ee/gprofiler/gost) (run by default; optional)
+- Rare genes are filtered out
 
-`samplesheet.csv`:
+#### 4. Sample filtering
 
-```csv
-sample,fastq_1,fastq_2
-CONTROL_REP1,AEG588A1_S1_L002_R1_001.fastq.gz,AEG588A1_S1_L002_R2_001.fastq.gz
-```
+Samples that show too high ratios of zeros or missing values are removed from the analysis.
+
+#### 5. Normalisation of expression
+
+- Normalize RNAseq raw data using TPM (necessitates downloading the corresponding genome and computing transcript lengths) or CPM.
+- Perform quantile normalisation on each dataset separately using [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.quantile_transform.html)
+
+#### 6. Merge all data
+
+All datasets are merged into one single dataframe.
+
+#### 7. Imputation of missing values
+
+Missing values are replaced by imputed values using a specific algorithm provided by [scikit-learn](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.quantile_transform.html). The user can choose the method of imputation with the `--missing_value_imputer` parameter.
+
+#### 8. General statistics for each gene
+
+Base statistics are computed for each gene, platform-wide and for each platform (RNAseq and microarray).
+
+#### 9. Scoring
 
-Each row represents a fastq file (single-end) or a pair of fastq files (paired end).
+- The whole list of genes is divided in multiple sections, based on their expression level.
+- Based on the coefficient of variation, a shortlist of candidates genes is extracted for each section.
+- Run optimised, scalable version of [Normfinder](https://www.moma.dk/software/normfinder)
+- Run optimised, scalable version of [Genorm](https://genomebiology.biomedcentral.com/articles/10.1186/gb-2002-3-7-research0034) (run by default; optional)
+- Compute stability scores for each candidate gene
 
--->
+#### 10. Reporting
 
-Now, you can run the pipeline using:
+- Result aggregation
+- Make [`MultiQC`](http://multiqc.info/) report
+- Prepare [Dash Plotly](https://dash.plotly.com/) app for further investigation of gene / sample counts
 
-<!-- TODO nf-core: update the following command to include all required parameters for a minimal example -->
+## Test pipeline
+
+You can test the execution of the pipeline locally with:
+
+```bash
+nextflow run nf-core/stableexpression -profile test,<docker/apptainer/conda/micromamba/...>
+```
+
+## Basic usage
+
+> [!NOTE]
+> If you are new to Nextflow and nf-core, please refer to [this page](https://nf-co.re/docs/usage/installation) on how to set-up Nextflow. Make sure to [test your setup](https://nf-co.re/docs/usage/introduction#how-to-run-a-pipeline) with `-profile test` before running the workflow on actual data.
+
+To search the most stable genes in a species considering all public datasets, simply run:
 
 ```bash
 nextflow run nf-core/stableexpression \
-   -profile <docker/singularity/.../institute> \
-   --input samplesheet.csv \
-   --outdir <OUTDIR>
+   -profile <PROFILE (examples: docker / apptainer / conda / micromamba)> \
+   --species <SPECIES (examples: arabidopsis_thaliana / "drosophila melanogaster")> \
+   --outdir <OUTDIR (example: ./results)> \
+   -resume
 ```
 
-> [!WARNING]
-> Please provide pipeline parameters via the CLI or Nextflow `-params-file` option. Custom config files including those provided by the `-c` Nextflow option can be used to provide any configuration _**except for parameters**_; see [docs](https://nf-co.re/docs/usage/getting_started/configuration#custom-configuration-files).
+## More advanced usage
+
+For more specific scenarios, like:
 
-For more details and further functionality, please refer to the [usage documentation](https://nf-co.re/stableexpression/usage) and the [parameter documentation](https://nf-co.re/stableexpression/parameters).
+- **fetching only specific conditions**
+- **using your own expression dataset(s)**
+
+please refer to the [usage documentation](https://nf-co.re/stableexpression/usage).
+
+## Resource allocation
+
+For setting pipeline CPU / memory usage, see [here](./docs/configuration.md).
+
+## Profiles
+
+See [here](https://nf-co.re/stableexpression/usage#profiles) for more information about profiles.
 
 ## Pipeline output
 
 To see the results of an example test run with a full size dataset refer to the [results](https://nf-co.re/stableexpression/results) tab on the nf-core website pipeline page.
 For more details about the output files and reports, please refer to the
 [output documentation](https://nf-co.re/stableexpression/output).
 
+## Support us
+
+If you like nf-core/stableexpression, please make sure you give it a star on GitHub!
+
+[![stars - stableexpression](https://img.shields.io/github/stars/nf-core/stableexpression?style=social)](https://github.com/nf-core/stableexpression)
+
 ## Credits
 
 nf-core/stableexpression was originally written by Olivier Coen.
 
-We thank the following people for their extensive assistance in the development of this pipeline:
+We thank the following people for their assistance in the development of this pipeline:
 
-<!-- TODO nf-core: If applicable, make list of people who have also contributed -->
+- Rémy Costa
+- Shaheen Acheche
+- Janine Soares
 
 ## Contributions and Support
 
@@ -95,8 +172,6 @@ For further information or help, don't hesitate to get in touch on the [Slack `#
 <!-- TODO nf-core: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file. -->
 <!-- If you use nf-core/stableexpression for your analysis, please cite it using the following doi: [10.5281/zenodo.XXXXXX](https://doi.org/10.5281/zenodo.XXXXXX) -->
 
-<!-- TODO nf-core: Add bibliography of tools and data used in your pipeline -->
-
 An extensive list of references for the tools used by the pipeline can be found in the [`CITATIONS.md`](CITATIONS.md) file.
 
 You can cite the `nf-core` publication as follows:

diff --git a/assets/methods_description_template.yml b/assets/methods_description_template.yml
@@ -3,8 +3,6 @@ description: "Suggested text and references to use when describing pipeline usag
 section_name: "nf-core/stableexpression Methods Description"
 section_href: "https://github.com/nf-core/stableexpression"
 plot_type: "html"
-## TODO nf-core: Update the HTML below to your preferred methods description, e.g. add publication citation for this pipeline
-## You inject any metadata in the Nextflow '${workflow}' object
 data: |
   <h4>Methods</h4>
   <p>Data was processed using nf-core/stableexpression v${workflow.manifest.version} ${doi_text} of the nf-core collection of workflows (<a href="https://doi.org/10.1038/s41587-020-0439-x">Ewels <em>et al.</em>, 2020</a>), utilising reproducible software environments from the Bioconda (<a href="https://doi.org/10.1038/s41592-018-0046-7">Grüning <em>et al.</em>, 2018</a>) and Biocontainers (<a href="https://doi.org/10.1093/bioinformatics/btx192">da Veiga Leprevost <em>et al.</em>, 2017</a>) projects.</p>