Skip to content

Releases: koefoeden/multiomeR

multiomeR 1.3.1

Choose a tag to compare

@koefoeden koefoeden released this 21 Oct 12:22

multiomeR 1.3.1 corrects Harmony with several covariates and fixes two problems in the differential analyses. It also corrects the manual where it no longer matched the code. Results change only for aggregations that set several Harmony covariates or a count model with a one-column design. A feature model that pools several cell types without a donor term now stops. Coming from 1.2.x, also follow the 1.3.0 notes.

What you need to do

  1. Update to this release. Releases tag stable, validated commits, so update to the tag rather than pulling main, which can carry development work that is not yet validated. This release needs Pixi 0.82.1 or later, so run pixi self-update first if pixi --version shows an older one. Commit or stash local changes first, and resolve any merge conflicts as usual. While no other R process uses the environment, run:

    cd multiomeR  # your clone
    git fetch origin tag v1.3.1
    git merge --ff --no-edit v1.3.1
    pixi install --locked --run-post-link-scripts
    pixi run --use-environment-activation-cache install-r-github-packages

    The environment is unchanged since 1.3.0.

  2. Give feature models over several cell types a donor term. A model in differential_analyses_pseudobulk_models tests all cell types together unless its cell_type_subset names exactly one. One donor's pseudobulks of different cell types are not independent replicates. Such a model therefore stops the module's models target, unless it sets a paired cell-type design (cell_type_formula with pairing_variable) or a random_effect. To test each cell type on its own, give each model a single cell_type_subset. Before, only a formula that named cluster was refused.

  3. Coming from 1.2.x? Also follow steps 2 to 8 of the 1.3.0 notes.

What happens to your aggregations

  • Harmony embeddings rerun. With one covariate or none, as in the example configurations, they reproduce their values, so nothing downstream reruns. With several covariates, the embeddings and everything downstream change.
  • The differential analyses' models and feature_matrix_fit targets rerun. Configured models reproduce their fits, except count models with a one-column design, whose results change.
  • The Seurat/Signac exports rerun to record the corrected residual label. Their data are unchanged.

What changed

Fixes

  • Harmony with several covariates: each covariate is now corrected as its own batch variable, as Harmony 2.0.5 allows. Before, each observed combination, such as a GEM well and a sex, was one batch, which made many small batches. On a synthetic embedding with well and sex shifts, the remaining well and sex R² fell from 0.014 to 0.004 and from 0.008 to 0.0002.
  • Pooled cell types: a feature model over several cell types needs a donor term, as step 2 describes. Before, its cell types' pseudobulks counted as independent replicates unless the formula named cluster.
  • One-column count designs, such as ~ 0 + x, are filtered, TMM-normalised and fitted with voom, like other count designs. Before, limma fitted their raw counts without filtering or normalisation, since edgeR's tests need two design columns.
  • Seurat export: the BPCells_native GEX export labels its Pearson residuals as clipped at plus or minus the square root of the number of PCA nuclei, as the code clips them, instead of [-10, 10].

Documentation

  • Methods statements that no longer matched the code are corrected. For example, the WNN cell set restricts neighbours without recomputing them, cellsnp-lite thresholds are per-GEM-well defaults, and SCAVENGE uses a shared-nearest-neighbour graph from the WNN neighbours.
  • Module requirements are corrected: peak–gene tests need at least eight donors with 40 nuclei of a cell type, and abundance models test cell types found in at least half of the donors. The extended donor table replaces the aggregation's, so it must hold every model variable.
  • scDblFinder and SCAVENGE are cited where each methods page first names them, and each methods sentence has its own line.
  • The test-suite description, the gallery paths and the configuration README are corrected.

Development

  • Helpers that no target uses leave the pipeline's helpers: render_figures() is removed, and the cell-level UCell scorers move next to the parity comparisons and the annotation diagnostics. No target reruns.
  • Agents run worktree commands through the main checkout's Pixi manifest, since Pixi 0.82 rejects a worktree's .pixi link to that environment. AGENTS.md gains sections on validation and new tests and on GitHub destinations.
  • The release validation is recorded in VALIDATION-1.3.1.md.

multiomeR 1.3.0

Choose a tag to compare

@koefoeden koefoeden released this 21 Oct 10:44

Update to multiomeR 1.3.1 instead. It adds fixes for Harmony with several covariates and for the differential analyses. Follow its notes, which point back here for the steps below.

multiomeR 1.3.0 counts ATAC fragments instead of insertions, makes clustering and UMAPs reproducible for any thread count, and roughly halves the time of a full build. A 91-sample analysis with all modules now takes an estimated 5.0 hours on the benchmark's workers, against 12.9 hours before. Genetic enrichment adds S-LDSC heritability enrichment per cell type, and its cell-type chromVAR test now takes P-values from moved copies of each GWAS's loci. Modules can now analyse chosen datasets only, donors can give several samples, and an aggregation can read several donor tables. Every aggregation's results change, so plan a full rebuild after updating. Four parameters and two ATAC QC metrics are renamed or removed, so update your configuration first.

What you need to do

  1. Update to this release. Releases tag stable, validated commits, so update to the tag rather than pulling main, which can carry development work that is not yet validated. This release needs Pixi 0.82.1 or later, so run pixi self-update first if pixi --version shows an older one. Commit or stash local changes first, and resolve any merge conflicts as usual. While no other R process uses the environment, run:

    cd multiomeR  # your clone
    git fetch origin tag v1.3.0
    git merge --ff --no-edit v1.3.0
    pixi install --locked --run-post-link-scripts
    pixi run --use-environment-activation-cache install-r-github-packages

    The environment changes only its CA certificate bundle, which no longer trusts eight compromised eMudhra certificate authorities. Native kernels now compile once into your user cache folder instead of in every worker.

  2. Split the cluster-annotation margin. aggregation_cluster_annotation_min_advantage becomes aggregation_GEX_cluster_annotation_min_advantage, aggregation_ATAC_cluster_annotation_min_advantage and aggregation_WNN_cluster_annotation_min_advantage, each 0.05 by default. Set all three to your old value, for example:

    aggregation_GEX_cluster_annotation_min_advantage: 0.02
    aggregation_ATAC_cluster_annotation_min_advantage: 0.02
    aggregation_WNN_cluster_annotation_min_advantage: 0.02

    With equal values, no target command changes. The WNN margin sets only the final labels, so you can later tune it without rebuilding the GEX and ATAC processing.

  3. Remove three retired parameters: aggregation_ATAC_peak_matrix_mode, aggregation_UMAP_GEX_PCs and aggregation_UMAP_ATAC_PCs. Peaks now always count fragments. UMAPs reuse the clustering neighbours, so they use aggregation_GEX_data_PCs and aggregation_ATAC_data_PCs. Their defaults, 1:30 and 2:30, match the old UMAP defaults. A configuration that still sets a parameter of step 2 or 3 stops when the graph is built, with "defines parameter(s) absent from the manifest".

  4. Move your ATAC QC filters to the fragment metrics. atac_peak_counts_frac and atac_peak_counts_blacklist_frac are gone. A filter that names them stops QC_excluded_BCs_list.ATAC with "object not found" when that target runs. The graph still builds, so check aggregation_QC_exclude_list_combined_object before you run. nCount_ATAC remains, but it now counts fragments, so an unchanged threshold on it removes more nuclei. These replacements of the former example filters exclude nearly the same nuclei (Jaccard 0.93–0.95 in three aggregations):

    Before, with the default insertion counting After
    nCount_ATAC < 1000 atac_fragments_in_peaks < 590
    atac_peak_counts_frac < 0.1 atac_fragments_in_peaks_frac < 0.12
    atac_peak_counts_blacklist_frac > 0.05 atac_fragments_in_blacklist_frac > 0.012

    In another calibration, nCount_ATAC < 200 and atac_peak_counts_frac < 0.02 became atac_fragments_in_peaks < 125 and atac_fragments_in_peaks_frac < 0.025. Count limits thus fall to about 0.6 times their old value, and fraction limits rise about 1.2-fold. Confirm other values in the checkpoint 4 QC plots. If you counted with aggregation_ATAC_peak_matrix_mode: overlaps, which counted a fragment about once, keep about the same count and in-peak fraction limits. The blacklist fraction now divides by all fragments rather than by the peak counts, so for human data 0.05 becomes about 0.012 in either mode. It also depends on the genome: the mm10 blacklist covers 239 Mb against 72 Mb for GRCh38, so a typical mouse nucleus has 1.7% instead of 0.03% of its fragments there. The mouse demo therefore uses 0.025. Set a mouse limit from its distribution, never the human value. Also replace the old names in aggregation_continuous_vars and in your own scripts.

  5. Turn on ATAC gene activity scores if you use them. They are now off by default. This drops their only plot, marker_gene_activity_dot_plot.7_ATAC_QC, and saves about 2.5 hours of worker time per 740,000-nucleus build. Set aggregation_ATAC_gene_activity_scores: true to keep the plot. The Seurat/Signac export does not use the scores.

  6. Update code that reads pipeline outputs:

    • The cell metadata tables, metadata_w_clusters_tibble.* and metadata_w_cell_types_tibble.*, no longer carry UMAP coordinates. Join UMAP_embeddings_tibble.GEX, .GEX_non_harmony, .ATAC or .WNN, or read metadata_w_cell_types_analysis_tibble.*, which the plots use. The Seurat exports keep their UMAPs.
    • The ATAC QC columns are renamed as in step 4, and atac_peak_counts_blacklist is gone. atac_fragments_in_peaks is new. atac_fragments_in_blacklist_frac is part of each GEM well's metadata from checkpoint 1.
    • nCount_ATAC and the ATAC peak matrices count fragments, so their values are lower than the former insertion counts.
    • GEX count folders store unsigned integers instead of doubles. Targets that open them still return doubles. If you open the folders yourself, open_BPCells_counts_as_double() does the same.
    • The cell-type chromVAR deviation tables take z, z_p and z_q from the locus-shift test and gain background_z, null_mean and null_sd. Their support_label is gone. The peak, variant and locus contribution tables now split the deviation only, so they lose z_contribution, their z and support_label.
  7. Give genetic-enrichment workers network access to Zenodo and the GWAS Catalog. S-LDSC runs in every aggregation with the genetic_enrichment module. It downloads a 1.5 GB LD reference once, and each GWAS's summary statistics once for all aggregations. To use other summary statistics, name a path, URL or GWAS Catalog accession in the study's optional summaryStatistics field. Studies without summary statistics stay as grey rows.

  8. Coming from 1.1.0? Also follow steps 2 to 6 of the 1.2.0 notes.

What happens to your aggregations

  • Every aggregation rebuilds, and its results change. The rebuild starts at each GEM well's full_metadata_tibble, which gains the blacklist fraction, and at the per-well peak matrices, which now count fragments. Review the labels at checkpoints 3, 7 and 8 before the modules run.
  • What changes:
    • ATAC QC exclusions, as step 4 describes.
    • Peaks: each peak regains its first base, MACS3's maximum gap is fixed, and each peak-calling group draws its random halves anew once. 98.1% of consensus peaks are unchanged by the fixed gap. The new halves change the consensus peaks as much as a new seed would (Jaccard 0.93).
    • Clusters, UMAPs and cell-type labels: neighbour indexes are built on one thread, WNN draws one set of modality weights instead of two, and the GEX and ATAC UMAPs start from a PCA layout. Before, two identical rebuilds already gave different clusters (adjusted Rand index 0.85 for GEX and 0.77 for ATAC). Rebuilds now reproduce for any thread count.
    • All module results, through these inputs. The differential analyses and peak–gene tests themselves compute the same values faster. Log fold changes agree within 2.1e-9 and peak–gene P-values within 9.5e-14.
    • Cell-type chromVAR z-scores, P-values, q-values and heatmaps, as Genetic enrichment describes. Their deviations change only through their inputs.
  • Targets added: S-LDSC and the locus-shift test build for every aggregation with the genetic_enrichment module. The first run downloads the LD reference and the summary statistics: on the benchmark, the LD reference took 2 minutes and the 26 GWAS sources 38 minutes of work in parallel branches. S-LDSC's own steps then took 33 minutes of work at 31 samples and 45 minutes at 91.
  • Per-GEM-well targets that rerun and reproduce their results:
    • cellsnp-lite reruns once for every genotyped well, about 1 to 1.5 hours per large well on 6 cores. Vireo then reruns once with identical calls.
    • AMULET reruns once, since its native kernel now loads from the user cache.
    • Each well's GEX counts are rewritten as unsigned integers.
    • Cell Ranger, CellBender and the downloads stay current.
  • Targets built once: gene annotations, blacklists and chromatin states per reference genome, in about 9 minutes.
  • Targets removed: the four ATAC gene-activity targets of each aggregation unless you enable them (step 5), reference_Ensembl_annotations_GRanges_list and cross_UMAP_barcodes.ATAC. prune_orphaned_plots() lists the images that removed or renamed targets leave behind.
  • Mouse demo: brain_mouse excludes nuclei whose blacklist fraction exceeds 0.025, about 1.5 times its median. It keeps 3,532 instead of 2,751 nuclei, because the former limit sat close to the mouse median and removed more.
  • Demo: immune_human_2x keeps clusters of 50 nuclei or more, so its MAIT cluster, now 99 nuclei, stays. A WNN-only `Cy...
Read more

multiomeR 1.2.2

Choose a tag to compare

@koefoeden koefoeden released this 14 Oct 13:39

multiomeR 1.2.2 is the release that the multiomeR manuscript cites. It redesigns the manuscript figures, labels the islet cluster and narrows the GWAS panel of the public example mixed_human_31x, and corrects, clarifies and references the methods text of the manual. Only code comments changed in the pipeline code, so the results of your aggregations do not change and nothing in your stores reruns.

What you need to do

  1. Update to this release. Releases tag stable, validated commits, so update to the tag rather than pulling main, which can carry development work that is not yet validated. Commit or stash local changes first; resolve any merge conflicts as usual. While no other R process uses the environment, run:

    cd multiomeR  # your clone
    git fetch origin tag v1.2.2
    git merge --ff --no-edit v1.2.2
    pixi install --locked --run-post-link-scripts
    pixi run --use-environment-activation-cache install-r-github-packages

    Nothing else is required: the environment and the parameter schema are unchanged.

  2. Coming from 1.1.0? Also follow steps 2 to 6 of the 1.2.0 notes.

What happens to your aggregations

  • Existing stores stay current. No pipeline function or target changed, since targets ignores the new code comments, and configuration/ is unchanged.
  • In configuration_dev/, mixed_human_31x reruns its WNN cell-type labels and everything downstream of them, and its genetic enrichment module drops 13 studies. The manuscript targets of pipeline_manuscript_figures/targets.R rerun; Figure 1's panel targets are renamed after their new letters.

What changed

Public example

  • Islet cells: the WNN labels of mixed_human_31x score an islet endocrine set (INS, SLC30A8, PCSK2, ABCC8, KCNMB2) instead of the beta-cell set, through aggregation_WNN_marker_genes. The islet cluster (371 nuclei, almost all from pancreas) tied beta and acinar cells on ambient INS and stayed unassigned; it is now labelled, and no other cluster changes. The example has 19 cell types.
  • GWAS panel: the genetic enrichment module tests 26 instead of 39 studies. Studies with fewer than 20 effective peaks in the aggregation, (Σw)²/Σw² of their variant-weighted peaks, are left out, because their enrichments rest on one or a few loci, as is FastingGlucose_Chen2023, whose samples FastingGlucose_Jee2025 contains. 12 studies are enriched in at least one of 10 cell types.

Manuscript results

  • Figure 1 opens with a marker panel (A): for each WNN cell type, the configured marker gene whose expression and promoter accessibility, in cell-type pseudobulks summed over donors, are most specific to it. The other panels move one letter on. Panel D orders its cell types like panel A and its traits by their selected cell type, so the selected pairs form a staircase. Panels E and F show colorectal cancer only, side by side, so the figure fits one page, and F labels only the cell types enriched for it. The panels have tidier labels and ticks, and an empty locus window names the Open Targets locus-to-gene prediction.
  • Supplementary Figure S2 shows multiomeR with its Seurat/Signac export against the Seurat/Signac baseline, the measure that matches the baseline's output, in four tagged panels with direct line labels and the fold difference wherever the lines are far enough apart; its units are CPU-hours and GB-hours.
  • Supplementary Figure S3 and the Table S3 data follow the 26 studies. S3 keeps only the trait-category and loci tracks, since Table S3 gives each study's method, sample size and ancestry, and fits a portrait page. Supplementary Table S4 counts nuclei.

Documentation

  • The workflow overview is redrawn on one grid with lighter styling, heads its primary module like the downstream modules and numbers the module's four parts.
  • The methods pages are checked against the code: they state that genetic enrichment uses the 95% credible sets, that differential pseudobulks are not those of the Seurat/Signac export, when the TF-expression comparison is available, which models the paired-cell-type route uses, and how WNN annotation draws its own controls. They give the key defaults of peak–gene filtering, SCAVENGE, doublet removal and clustering, and drop details that the manual gives elsewhere.
  • The methods pages also spell out the differential-analysis abbreviations, give each peak–gene support preset its own clause, and cite HNSW, SCTransform v2, SuSiE-inf, PICS, the Open Targets locus-to-gene score and Kenward–Roger inference, with complete author lists and issue numbers.
  • The methods pages are easier to read: sentences are short and free of semicolons, everyday words replace jargon, and short reasons accompany choices a reader may question, such as one overdispersion value for all genes, per-group peak calling, the measurement-support filter for peak–gene pairs and joint SuSiE fine-mapping.
  • The methods pages cite the published work behind statements that lacked a reference, among them AMULET's threshold, the cell-cycle gene sets, UMAP, the genome blacklist, ArchR's peak rules, depth-normalized peak scores, the edgeR and limma procedures (TMM, voom, duplicateCorrelation, quasi-likelihood F-tests, robust empirical Bayes), MSigDB Hallmark and Reactome, Cicero co-accessibility and the trade-off of batch correction.
  • The README shows how to update a clone to a release tag, by merging rather than rebasing, and states the versioning rules. Release notes now live on the GitHub releases page only, which already carried the same notes; NEWS.md is gone from main, and tags up to v1.2.1 keep it.

Development

  • Code comments credit the methods that multiomeR adapts, such as AMULET, UCell, Seurat WNN, SCAVENGE, analytic Pearson residuals and ArchR-style consensus peaks, and record the reasons for key design choices. No function changed.
  • The release convention moved from dev/RELEASES.md into the README and the release skill, and the documentation rules from AGENTS.md into a documentation skill. CONTRIBUTING.md lists the BPCells convention check among the CI checks.
  • The agent skills share one wording with the private repository's, and stale agent guidance and a finished review checklist were removed.

multiomeR 1.2.1

Choose a tag to compare

@koefoeden koefoeden released this 12 Oct 12:52

multiomeR 1.2.1 lowers the memory requirement from 60 GB to 32 GB of RAM, labels the brain_mouse example with embryonic marker sets, and fixes the UMAP parameter sweeps on large aggregations. The results of your aggregations do not change: existing stores stay current apart from a few plots.

What you need to do

  1. Update to this release. Releases tag stable, validated commits, so update to the tag rather than pulling main, which can carry development work that is not yet validated. Commit or stash local changes first; resolve any merge conflicts as usual. While no other R process uses the environment, run:

    cd multiomeR  # your clone
    git fetch origin tag v1.2.1
    git merge --ff --no-edit v1.2.1
    pixi install --locked --run-post-link-scripts
    pixi run --use-environment-activation-cache install-r-github-packages

    Nothing else is required: the environment and the parameter schema are unchanged.

  2. Coming from 1.1.0? Also follow steps 2 to 6 of the 1.2.0 notes. They now include the rename of AMULET's fragment-count column from nFrags to amulet_nFrags, which 1.2.0 made like the other AMULET columns but its notes first omitted.

What happens to your aggregations

  • Existing stores stay current. Resource requests, controllers and target descriptions do not invalidate targets.
  • Only plots rebuild. The GEX and ATAC UMAP sweeps (cross.UMAPs.3_GEX_QC, cross.UMAPs.7_ATAC_QC) and TRS_cluster_summary_plot rerun with new plots. UMAP_embeddings_tibble.GEX, .GEX_non_harmony and .ATAC rerun and reproduce their values.
  • brain_mouse rebuilds from its marker lists onward.
  • In configuration_dev/, only targets of pipeline_manuscript_figures/targets.R rerun: Figure 1 panels A and C–E with its plot and data, and the Table S4 writer. figure_1_traits, supplementary_table_S3 and supplementary_figure_S3 are new.

What changed

Memory requirement

  • Each target's get_tar_resources() request is about 1.5 times the peak memory it reached on aggregations of 213,000 and 713,000 nuclei, in 16- and 32-GB steps. With the default GEX backend, no target requests more than 32 GB, against 49 requests of 40 or 60 GB before; the largest targets need about 7 GB plus 2.5 GB per 100,000 nuclei. Unmeasured targets request at most 32 GB.
  • GEX normalization and the two Seurat objects request 60 GB only when an aggregation of the configuration uses the optional Seurat_SCT GEX backend, whose GEX normalization needs about 0.25 GB per 1,000 nuclei.
  • The default local-heavy controller is declared with 32 GB, and a new local-large controller (6 cores, 60 GB, one worker) takes those requests. A custom crew_controllers.R needs a tier of at least 60 GB only for aggregations that use Seurat_SCT.
  • Built from an empty store in a 16-CPU Slurm job limited to 32 GiB, the demo's processes used at most 17 GiB.
  • Target descriptions record the measurements behind the requests as [resource_observation: ...] tags. A target may carry several, and the manual suggests common keys.

Examples

  • The inactive brain_mouse example (10x Genomics' E18 mouse brain) has embryonic marker sets: radial glia, intermediate progenitors, upper- and deep-layer and hippocampal excitatory neurons, and inhibitory neurons. The adult glial sets left every WNN cluster unassigned; the new sets label these populations, leaving at most a small mixed cluster unassigned, and its ATAC marker TFs follow.

Fixes

  • The GEX and ATAC cross-parameter UMAP sweeps run over the first n configured UMAP components, so the ATAC sweep no longer includes the LSI depth component and its largest setting matches the main UMAP. They are initialized with spectral PCA (init = "spca"), because uwot's default spectral initialization could crash on embeddings of several hundred thousand nuclei.
  • The SCAVENGE TRS interval plot uses at most four facet columns, fewer for more than 30 groups, so its width stays within the 50,000-pixel image limit.

Manuscript results

  • Figure 1C shows the cell-type enrichments at q < 0.05 that rest on many loci (top locus below half of the signal, at least 20 effective weighted peaks), one per study and per cell type and trait category, with stars at q < 0.05; panels D and E show LDL cholesterol in hepatocytes and colorectal cancer in colonocytes. Trait names are in sentence case, and panel A names its cell type as a plural noun.
  • New Supplementary Figure S3 shows every tested study's cell-type deviations on Figure 1C's scale, and Supplementary Table S3 is generated from the stored GWAS inputs of mixed_human_31x.
  • Supplementary Figure S2's reserved CPU and RAM are computed from the worker sizes of the measured runs, recorded in seurat_signac_comparison_allocations.tsv, so later request changes leave them unchanged.
  • The Table S4 data leave out the compute node's name.
  • The manuscript figure helpers wrap patchwork panels with patchwork::wrap_elements(), so their legends and labels are measured on the saving device and no longer overlap or clip.

Development

  • Agents start long pipeline drivers detached and follow their log, since an agent tool's time limit stops the driver and its running targets.

multiomeR 1.2.0

Choose a tag to compare

@koefoeden koefoeden released this 12 Oct 08:15

multiomeR 1.2.0 revises the statistical methods after an audit: GEX normalization, WNN modality weights, clustering, consensus peaks, doublet detection, and the differential, gene-set, composition, genetic-enrichment and peak–gene tests. Every aggregation's results change, from QC and cell-type labels to all module results, so plan a full rebuild after updating. The release also adds an optional WNN-specific marker list, per-cluster composition tables, lower memory use for CellBender input, targets stores that can move between disks, and the public configuration, figures and results of the multiomeR manuscript.

What you need to do

  1. Update to this release. Releases tag stable, validated commits, so update to the tag rather than pulling main, which can carry development work that is not yet validated. Commit or stash local changes first; resolve any merge conflicts as usual. While no other R process uses the environment, run:

    cd multiomeR  # your clone
    git fetch origin tag v1.2.0
    git merge --ff --no-edit v1.2.0
    pixi install --locked --run-post-link-scripts
    pixi run --use-environment-activation-cache install-r-github-packages

    This installs Seurat 5.6.0 from GitHub, with its build dependencies r-pbapply and r-rcppthread from the lock file.

  2. Give your configuration directory its own crew_controllers.R. The root crew_controllers.R is gone: each configuration directory now carries controllers that suit its aggregations, and the default moved to configuration/crew_controllers.R. If you selected your own directory in configuration.local, copy configuration/crew_controllers.R into it and adapt it, moving in any changes you made to the former root file.

  3. Update moved paths in your configuration and scripts:

    • cfg_pipeline_parameters.tsv, QC_checkpoint_manifest.tsv, QC_metric_manifest.tsv and the demo download lists are in manifests/.
    • reference_metadata/ is resources/reference_metadata/; add new Cell Ranger ARC references there.
    • module_<name>/ is extra_targets/module_<name>/.
    • The example donor metadata tables formerly in example_data/ are in configuration/ and configuration_dev/, and the download script is in scripts/. example_data/ now only receives downloads; three unused tables were removed.
    • mixed_human_31x and the nested comparison_1x to comparison_20x aggregations moved to the new configuration_dev/, with their donor metadata and GWAS selection; select it to rebuild them. manuscript_figures/ is replaced by pipeline_manuscript_figures/.
  4. Check your configuration against the changed defaults and checks:

    • The cell-cycle score is no longer regressed out by default. Set aggregation_SCT_regress_vars: [CC.Difference] to keep it.
    • A differential-analysis formula that uses cluster without cell_type_formula or random_effect now stops with an error: it treated each donor's cell types as independent samples. Rewrite such formulas.
    • CellBender counts are used whenever a GEM well gives a CellBender H5 file; GEM_well_add_cellbender is no longer needed and a leftover column is ignored. Set GEM_well_cellbender_h5_file to NA to keep Cell Ranger counts.
    • Marker symbols that several genes share now stop marker validation, as missing symbols do, because they were scored on only one of the genes. Replace them.
    • The example configurations now exclude nuclei with more than 5% of their peak counts in blacklisted regions (was 1%, which removed cell types unevenly) and AMULET multiplets at the authors' q < 0.01 (was 0.05). Configurations copied from the old examples keep the old limits; change them to follow the examples.
  5. Update code that reads pipeline outputs:

    • Each unassigned cluster is labeled Unassigned_cluster_<id> instead of one merged Unassigned.
    • AMULET's fragment-count column is amulet_nFrags instead of nFrags, like the other AMULET columns.
  6. Demo users: run pixi run --use-environment-activation-cache setup-demo again to download controller_PBMC_human.

What happens to your aggregations

  • Everything after the per-GEM-well inputs rebuilds. Every GEM well reruns amulet_metrics_tibble and full_metadata_tibble, and every aggregation target rebuilds. Cell Ranger, CellBender, cellsnp-lite, Vireo and the other per-GEM-well inputs stay current. The first run downloads the AMULET excluded-region list.
  • Cell-type labels may change. GEX normalization, WNN weights, clustering and doublet calls all change, and each unassigned cluster keeps its own label. Review the labels at checkpoints 3, 7 and 8 before the modules run; the new aggregation_WNN_marker_genes lets you tune the final labels without rebuilding the GEX and ATAC processing.
  • Peaks change. Each group's peaks must be called in both random halves of its nuclei, so small groups lose their weakly supported peaks.
  • Module results change: differential expression and accessibility, gene-set and composition tests, genetic enrichment, and peak–gene links, which are now calibrated and fewer.
  • Targets that rerun and keep their results. Targets that open BPCells matrices or fragments now record project-relative paths, so they rerun with unchanged data and their consumers rerun with them: GEX_counts_BPCells_matrix, fragments_w_prefix_bpcells, aggregated_counts_BPCells_matrix.GEX, consensus_peak_BPCells_matrix.ATAC, motif_family_accessibility_BPCells_matrix.ATAC, gene_score_archr_BPCells_matrix.ATAC and the pseudobulk count matrices. GEM_well_config_tsv, GEM_well_metadata_tibble, GEM_well_annotation_metadata_tibble, GEX_counts_BPcells_matrix_dir, chromHMMs_list_proj.ATAC, the five downloaded reference-file targets, the file targets of the moved manifests, reference metadata and donor metadata, and each GEM well's Cell Ranger reference lookup (cellranger_reference_json_file) rerun and reproduce their results. The exclusion lists (excluded_barcodes_by_type_list, excluded_cellranger_only_barcodes_by_type_list) rerun for GEM wells whose QC list lacks not_found_in_GEX_matrix, which includes every GEM well of the public configurations; their changes stop at the checkpoint 1 plots and retention tables. The new composition tables build without rerunning anything else.
  • Demo: its aggregation targets rebuild with the new controller_PBMC_human well.
  • In configuration_dev/, the Figure 1 targets rebuild. Supplementary Figure S2 reads the recorded runtimes of the compared workflows from the store, so rebuild its data before rerunning targets of the comparison_ aggregations.

What changed

Every change under Processing, Quality control, Differential analyses, Genetic enrichment and Peak–gene links alters results.

Processing

  • The BPCells backend's Pearson residuals use one overdispersion, theta = 100, for every gene and are clipped at ±√N, the analytic Pearson residuals of Lause et al. (2021). The per-gene method-of-moments theta counted differences between cell types as overdispersion and, with the ±10 clip, flattened marker variation: on a multi-tissue test set, 24 of 27 canonical markers now enter the 3,000 variable genes, against 10.
  • aggregation_SCT_regress_vars is unset by default, so the cell-cycle score, mostly noise in post-mitotic nuclei, is no longer regressed out; CC.Difference remains an example value.
  • WNN modality weights are estimated within seeded random subsets of at most 10,000 nuclei, so they no longer drift towards 0 or 1 as an aggregation grows. Neighbours are still chosen among all nuclei, and smaller aggregations are unchanged. Each nucleus is now first in its own neighbour list.
  • Leiden clustering runs for ten iterations, as Seurat does, instead of igraph's default two.
  • Consensus peaks are built as ArchR does. Each group's discovery nuclei are split into two random halves, MACS3 calls peaks in each half, and only peaks called in both are kept; each group then keeps its strongest peaks up to 500 per nucleus and 150,000 in all, and overlapping peaks across groups are ranked by the quantile of their significance within their group. Deep groups no longer dominate, and small groups lose their weakly supported tail (a 1,200-nucleus group keeps 54% of its peaks). A half takes about half the time and memory of a full call, so MACS3 workers request 16 GB. Consensus peaks keep their group in cluster.
  • The JASPAR familial root motifs lose flank columns with less than half of the largest column count; these low-support columns restricted the matches of 56 of 233 families to a sub-family.
  • Each unassigned cluster is labeled Unassigned_cluster_<id> instead of all being merged into one Unassigned group, which peak calling, pseudobulks, composition and genetic enrichment analysed as a real cell type.

Quality control

  • AMULET runs with chrM, chrX and chrY excluded in full and, for GRCh38, the authors' list of repeats, segmental duplications and blacklisted regions, downloaded once into the store and checked against its SHA-256 checksum. The example configurations exclude nuclei at the authors' q < 0.01 instead of 0.05.
  • In insertions mode, the peak fraction is divided by twice the fragment count, so atac_peak_counts_frac is a fraction and filters on it take effect.
  • ATAC scDblFinder groups peaks on singular-value-scaled LSI loadings, as scDblFinder itself does.
  • The example configurations exclude nuclei with more than 5% of their peak counts in blacklisted regions instead of 1%, which removed cell types unevenly.
  • Marker symbols that several genes share are rejected; they were scored on one of the genes.

Differential analyses

  • GEX markers and ATAC differential peaks are tested on counts per 10,000 instead of raw counts, with Seurat-style fold changes, so they no longer follow each group's sequencing depth.
  • Motif-family pseudobulks are tested on bias-corrected chromVAR ...
Read more

multiomeR 1.1.0

Choose a tag to compare

@koefoeden koefoeden released this 29 Sep 15:55

A documentation and demo release. The public demo aggregation is rebuilt with new data and markers; other aggregations need no recomputation.

Demo

  • Combine two healthy PBMC GEM wells from the same 10x Genomics series in the demo: the granulocyte-sorted well as before and the unsorted 3k well, now unsorted_PBMC_human, which replaces the lymphoma lymph node. The download shrinks from about 4.1 to 1.3 GB and the stored results from about 6.4 to 3.3 GB.
  • Annotate the demo with marker sets whose genes single nuclei capture: B, Mono, NK, DC and T. The previous two-gene markers left most demo nuclei Unassigned; every GEX, ATAC and WNN cluster is now labeled.
  • Plot only the sorting annotation in the demo's categorical UMAPs, because its other GEM-well annotations are now constant.

Documentation

  • Merge the implementation book into the manual. Its method descriptions form an In depth part with one primary-module page organized by checkpoint stage, one page per optional module and the algorithm-validation page; the implementation conventions become one Change the pipeline page. Each checkpoint of the running guide and each module page links to its methods.
  • Mark the reading layers in the sidebar: the guides, a Reference part that now includes the parameter browser, and the In depth part with an advanced badge.
  • Show the printed results of every command on the demo-results page, refreshed from the built demo with pixi run refresh-demo-outputs.
  • Give the GEM well, donor and aggregation reference pages one layout, and document pooled GEM wells, the rules checked and when, and aggregation inheritance.
  • Use US spelling throughout the manual.
  • Deploy the documentation only from main.

Validation

  • Retry each live preflight check of validate-local up to four times, also after refused connections, so an intermittent outage of a remote source no longer fails the whole validation.

Migration

  • Run pixi run --use-environment-activation-cache setup-demo again to download the new demo well; example_data/lymphoma_lymph_human is no longer used and can be deleted.
  • Target invalidation: every immune_human_2x target and the per-GEM-well targets of unsorted_PBMC_human build anew. Changed target descriptions invalidate nothing, so other aggregations stay current.

multiomeR 1.0.0

Choose a tag to compare

@koefoeden koefoeden released this 29 Sep 10:27

multiomeR 1.0.0 (2026-09-29)

The first stable release. The manuscript describing multiomeR is under peer review (link to come). From this release on, incompatible changes to configuration, target names or output schemas require a new major version; see RELEASES.md.

This release changes cell-type labels, peak–gene inference, genetic-enrichment inputs and several target and parameter names. Expect a full rebuild of existing aggregations and review the migration notes first.

Configuration

  • Keep each project's settings in its own configuration directory, selected by a single path in the ignored configuration.local; configuration/ holds the public defaults and examples. Module settings use flat cfg_module_<module>.yaml files, and disabled modules need no file.
  • Pass scalar component counts to GEX PCA and ATAC LSI, so changing only the selected dimensions, for example dropping the first LSI component, no longer recomputes either reduction.
  • Stop before building the target graph, with the install command, when the Bioconda BSgenome data packages are missing because Pixi skipped their post-link scripts.
  • Add the output gallery's mixed_human_31x aggregation of public 10x Genomics and ENCODE data as an inactive example with its module settings. Its GEM wells are processed locally with Cell Ranger ARC 2.1.0 and GRCh38-2024-A.

QC and cell-type annotation

  • Title the QC-exclusion UpSet plots with the number of retained barcodes and plot the nuclei per donor that reach the final WNN object.
  • Centre the motif-family accessibility heatmaps on each family's mean across groups, so abundant groups do not set their zero point.
  • Unify pre-filter QC plots across modalities, show doublet evidence beside GEX cluster markers, and report the selected wells, QC rules and configuration values in checkpoint captions.
  • Diagnose WNN modality weights by their association with metadata within clusters, and show WNN markers and UCell evidence as cluster dot plots.

Differential analyses

  • Use WNN-derived cell-type labels throughout, with named abundance and feature models, predictor-only abundance formulas and descriptive target names.
  • Test gene sets on the existing contrast results.
  • Give donors absent from the extended donor metadata missing model variables, so models exclude them instead of the module stopping.
  • Estimate intra-block correlations on at most 50,000 evenly spaced features and give each forked cell-type fit one BLAS thread, so a 590,000-peak chromatin-accessibility fit takes minutes instead of hours.

Genetic enrichment

  • Report SCAVENGE trait-relevance scores without permutation P-values, summarized by WNN cluster and cell type in four heatmaps.
  • Decompose cell-type chromVAR deviations additively into peak, variant and locus contributions, with ordinary and automatic absolute-effect weighting, and screen locus detail plots with a configurable z-score threshold.
  • Compare the credible sets of all configured GWAS by their shared normalized posterior-probability mass, to show which traits give non-independent enrichment results.
  • Build each GWAS's peak weights once and derive the chromVAR annotation from the peak-to-variant allocation. This also fixes the ordinary posterior- probability path, whose normalized inputs had lost their posteriorProbability column, and reports PIPs rather than effect weights as absolute-effect locus leads.
  • Sort the Open Targets credible-set records deterministically and allocate peak contributions to variants without per-variant summaries.
  • Weight every cell type equally in the expected accessibility of the cell-type pseudobulk chromVAR background, so an abundant cell type is no longer compared mainly with itself and pinned near zero.

Peak–gene correlation

  • Make peak–gene correlation an optional module on WNN cell types.
  • Replace the HC3 scan with a hierarchical donor-slope model fitted by native REML with Kenward–Roger inference over all eligible pairs; links require a positive, reliable hierarchical estimate at FDR < 0.05.
  • Filter hypotheses by RNA, ATAC and shared-donor measurement support, and show genomic context, coverage and donor-adjusted scatterplots in top-link figures. The moderate support preset is the default.
  • Fit the hierarchical model in donor space, without aggregate-by-aggregate matrices, so a branch projected at about a day takes minutes.

Runtime and dependencies

  • Export normalized GEX data in the Seurat objects: RNA gains a lazy log-normalized data layer, the BPCells backend adds the regressed Pearson residuals as RNA scale.data, and the Seurat backend adds the SCTransform() assay as SCT. The cell metadata includes the Seurat cell-cycle scores.
  • Build the multimodal Seurat export without rehashing every fragment file and count each assay once, which takes about a fifth of the time for large aggregations.
  • Save plots through one staged path that removes only obsolete outputs recorded in each target's inventory, and build each ggplot once.
  • Save plots as images only. The mirrored plot_objects/*.rds copies re-serialized each plot's inputs, reaching hundreds of gigabytes for large aggregations, and took about half of all plot-saving time.
  • Score the cluster UCell evidence with a compiled kernel, and compute the consensus peak matrix, ATAC blacklist counts and gene scores per GEM well; both return identical results, several times faster.
  • Compute ATAC coverage tracks from an in-memory copy of the plotted regions, which takes seconds instead of minutes to hours per region set on large merged fragment objects and returns identical data.
  • Make the project bootstrap stateless and share one checksum-keyed loader for the standalone native sources.
  • Rerun the dependents of opened BPCells directories when the directory content changes; previously they could keep results computed from the old content, and a cheap CI check now rejects bare BPCells opens in targets.
  • Update the locked environment within R 4.5 and Bioconductor 3.22 (among them Seurat 5.5.1, scDblFinder 1.24.10, arrow 25 and Python 3.13), move BPCells, Signac and betterChromVAR to current revisions, and drop 31 unused dependencies. multiomeRCore 0.2.0 narrows get_tar_resources().
  • Keep only reference-parity tests of the reimplemented algorithms, which pass with the updated reference packages.
  • Run only cheap checks in cloud CI: documentation source links and, also weekly, the example-data download URLs. Run the reference-parity tests locally, and build the documentation site with the repository builder.

Documentation

  • Restructure the manual around Configure, Run and Review steps, with one page per checkpoint and module, a standalone parameter browser, an output gallery generated from saved plots and an implementation book with one methods page per stage.
  • Share the method descriptions with the manuscript supplement and describe the validation contract of each reimplemented algorithm.

Migration

  • The Seurat objects no longer contain the counts-only SCTregr assay. Use RNA (BPCells backend) or SCT (Seurat backend); the WNN weight column is RNA.weight or SCT.weight instead of SCTregr.weight.
  • Move existing settings into a configuration directory and select it in configuration.local.
  • Remove these parameters, which configuration validation now rejects: aggregation_GALAXY_track_upload_API_KEY, aggregation_GALAXY_track_upload_HISTORY_ID, aggregation_tar_make_skip_regex_patterns, differential_analyses_bulk_RNA_rds_file_path, differential_analyses_OLINK_parquet_file_path, genetic_enrichment_posterior_probability_weighting_function_name and genetic_enrichment_SCAVENGE_permutation_times, and the per-study variant_weighting_mode GWAS field.
  • Reinstall the environment with pixi install --locked --run-post-link-scripts and the pinned GitHub packages with pixi run install-r-github-packages.
  • Update scripts that read HC3 peak–gene results, SCAVENGE P-values or renamed differential-analysis targets.
  • Redraw plots from their data instead of reading plot_objects/*.rds, as the demo-output page shows.

Validation: VALIDATION-1.0.0.md.

multiomeR 0.5.0

Choose a tag to compare

@koefoeden koefoeden released this 11 Sep 09:00
c3733a9

multiomeR 0.5.0 (2026-09-11)

Released through PR #5.

This release changes cluster annotations, cluster IDs, peak–gene inference and
several target/output names. Review the migration notes before rerunning an
existing analysis.

QC and interpretation

  • Organize review outputs into eight numbered QC checkpoints, separating GEX
    PCA and ATAC LSI review from clustering. Add cumulative, per-action nuclei
    retention flows and validate matching gene definitions before aggregation.
  • Select QC violin metrics from the shared manifest. Correct UpSet percentages,
    align RNA/ATAC/WNN confusion matrices, improve marker-set dot plots, and show
    all computed PCA/LSI dimensions in review plots.
  • Add within-cell-type cluster-marker comparisons and volcano plots. Number
    GEX, ATAC and WNN Leiden clusters by decreasing size without changing their
    membership; break size ties by the original cluster ID.
  • Add readable motif-family labels, palettes that accommodate many cell groups,
    and aligned genetic-enrichment panels and shared legends.

Cluster annotation

  • Replace the former standardized module-score labeller with matched-control
    UCell evidence across GEX, ATAC and WNN. All three use a shared GEX control
    reference. There is one annotation method, with no legacy selector.
  • Assign the leading label only when its adjusted score exceeds both matched
    background and competing labels by aggregation_cluster_annotation_min_advantage
    (default 0.05). Otherwise retain Unassigned, the candidate and a reason.
    Stability and GEM-well agreement are diagnostics, not assignment gates.
  • Preserve positive and negative UCell marker signatures. Signed scores are
    clipped per cell before averaging for observed and control signatures,
    including marker-deletion diagnostics. Positive-only panels retain the
    faster exact rank-summary calculation.
  • Export cluster evidence, marker-set support and competition, and clearer
    pre-doublet-filtering review plots. Unassigned clusters remain separate for
    doublet detection. Adjusted scores are not identity probabilities or validated
    biological error rates.

Peak–gene analysis

  • Allow exploratory discovery with few donors, including one donor, when
    aggregate counts and residual variation permit estimation. Adjust for donor
    and library depth; report conditional HC3 association statistics and separate
    donor-support diagnostics. State is no longer a nuisance-design term.
  • Retain nonpromoter gene-body candidates and plot the donor/depth residuals
    used in scoring. These associations do not establish population-level
    replication or causal enhancer–gene relationships.
  • Add SuSiE conditional peak prioritization and compact global-FDR reconstruction
    from chromosome branch inputs. Handle empty candidate sets with explicit
    plot outputs. SuSiE PIPs remain exploratory conditional prioritization.

Migration

  • Remove aggregation_cluster_annotation_method and
    aggregation_allow_multiple_cell_types from configuration. Signed marker
    panels remain supported; check the new evidence before interpreting labels.
  • Recheck external references to numeric cluster IDs. Annotation changes can
    propagate through doublet grouping, retained cells and downstream analyses.
  • Find plots and compatibility exports in numbered checkpoint folders, including
    GEX_Seurat_object.3_GEX_QC and multimodal_Seurat_object.8_multimodal_QC.
    Update scripts selecting the former target names. Old output files are not
    automatically deleted; consult current target metadata when reviewing results.
  • Subgroup reprocessing is removed from the public workflow. Remove its six
    aggregation_subgroups_*/contamination-filter settings from public configs.
  • Peak–gene results replace cluster_robust_SE with association_SE and add
    donor-support and inference-status fields. Recompute affected targets before
    comparing results; this is a method change, not just a speed improvement.

Principal recomputation boundaries (destination manifests provide full names):

cascading_target_breaking: cell_retention_tibble.GEX_input
cascading_target_breaking: QC_metric_manifest_tsv
cascading_target_breaking: PCA_clusters.GEX
cascading_target_breaking: LSI_clusters.ATAC
cascading_target_breaking: clusters_tibble_raw.WNN
cascading_target_breaking: metadata_w_cell_types_unfiltered_tibble.GEX
cascading_target_breaking: metadata_w_cell_types_unfiltered_tibble.ATAC
cascading_target_breaking: metadata_w_cell_types_tibble.WNN
cascading_target_breaking: peak_gene_correlation_donor_state_records.peak_gene_correlation.WNN
contained_target_breaking: GEX_Seurat_object.3_GEX_QC
contained_target_breaking: multimodal_Seurat_object.8_multimodal_QC

Validation

Public algorithm CI and documentation builds passed on the release implementation
at 43dac20; the final preparation commit changes only the release date and PR
link. Focused numerical validation includes 91 passing scoring/peak-gene
assertions, signed UCell parity and a nonempty SuSiE smoke fit. Default and
peak-gene opt-in manifests and both documentation books were validated.
See the validation record
for evidence and limits. No full demo/cohort pipeline replay was performed.

CI data v2: ARC 2.0.0 subsets with reference headers

Choose a tag to compare

@koefoeden koefoeden released this 11 Sep 10:55
c3733a9

Public Cell Ranger ARC fixtures for CI and local testing. Each archive contains 300 cells, without BAMs.

Dataset ARC version Reference package
Healthy PBMC 2.0.0 refdata-cellranger-arc-GRCh38-2020-A-2.0.0
Lymphoma lymph node 2.0.0 refdata-cellranger-arc-GRCh38-2020-A-2.0.0
Embryonic mouse brain 2.0.0 refdata-cellranger-arc-mm10-2020-A-2.0.0

Created with create_cellranger_arc_count_subset().

Source data: 10x Genomics, CC BY 4.0. Attribution and license are included in each archive.

Download SHA-256 checksums.