OpenOmicsBench v13.0.1

OpenOmicsBench v13.0.1 packages the improvements committed after v13.0.0.

• Added Bioconda badges and package links to the README and website.
• Added a worked RNA-seq example near the top of the README.
• Updated the website catalog so its benchmarks load from the site itself.
• Added a brief repository-star request to the CLI help footer.
• Added a code of conduct, security policy, and pull request template.

The benchmark datasets, reference results, and scoring behavior are unchanged from v13.0.0.

Install with python -m pip install openomicsbench==13.0.1 or see the Bioconda package once its recipe has updated.

OpenOmicsBench v13.0.0

OpenOmicsBench v13 makes the biological RNA-seq collection easier to use in a real workflow.

• Export one study or all 12 as checked compact or full input packs. Each case keeps counts, sample annotations, the declared contrast, source details, rights and checksums together.
• Check a count matrix, sample sheet and contrast file before analysis. The preflight reports ID mismatches, empty libraries, under-replicated groups and confounded fixed-effect designs.
• Score native DESeq2, edgeR and limma style CSV or TSV results directly against a named benchmark. Missing effects are counted, ambiguous headers and duplicate IDs fail, and contrast reversal is always explicit.
• Use the same features from an installed wheel without a repository checkout or network connection.

The collection still has 12 biological RNA-seq objects from seven Expression Atlas studies plus the declared synthetic fixtures. A passing result means agreement with the archived reference under its stated thresholds, not biological ground truth. External workflow compatibility has not been asserted without a corresponding pipeline run.

Install: python -m pip install openomicsbench==13.0.0

OpenOmicsBench v12.0.0

Version 12 adds a way to check RNA-seq count tables after moving them between analysis tools.

• omicsbench rnaseq check-counts ID FILE aligns rows and columns by gene and sample ID, then reports missing or extra IDs and changed integer counts. Reordering alone passes.
• omicsbench rnaseq verify-export ID DIRECTORY checks the complete three-file export: file hashes, count values, sample metadata, dimensions and source details.
• Both checks are available through the Python Benchmark API and work from the installed package without a checkout.

The 211 tests, collection and overlap audits, installed-wheel trial, and release preflight passed. Exact count agreement checks data transfer; it does not establish biological method performance.

Install: python -m pip install openomicsbench==12.0.0

OpenOmicsBench v11.0.0

Version 11 makes the 12 biological bulk RNA-seq benchmarks easier to use in analysis tools.

• omicsbench rnaseq export ID DIRECTORY writes an aligned count matrix and sample table, plus source details and SHA-256 hashes. Use --tier full for the complete matrix.
• Collection().get(ID).rnaseq() returns checked integer counts, gene and sample IDs, ordered metadata, attribution and limitations for Python notebooks and pipelines.
• Both paths verify the source file and refuse mismatched sample order. Existing export directories are preserved.

The 207 tests, collection and overlap audits, clean installed-wheel trial, and release preflight passed. These exports do not change the underlying biological validation; each benchmark's stated limitations still apply.

Install: python -m pip install openomicsbench==11.0.0

OpenOmicsBench v10.0.0

Version 10 makes it easier to use OpenOmicsBench with an existing pipeline or a Python notebook.

• omicsbench suite discover WORKSPACE finds supported outputs for selected cases, checks required table headers, and reports missing or ambiguous matches. Add --write to create a hash-locked result index before grading. Exact path and header mappings remain available when a pipeline uses different names.
• The new Collection Python API lists benchmarks, returns checksum-verified files, and exposes workspace preparation, discovery, grading, validation and evaluation without a CLI subprocess.
• Discovery and grading work across the collection's six comparable assay families from an installed wheel. Discovery checks routing and headers; scientific comparison remains the job of suite grade.

The 204 tests, collection and overlap audits, installed-wheel trial, and GitHub CI passed. The synthetic fixtures retain their stated limits, and the workspace is not a blinded challenge because the installed package includes its truth sets.

Install: python -m pip install openomicsbench==10.0.0

OpenOmicsBench v9.0.0

Version 9 makes it easier to test an external pipeline against the collection. omicsbench suite prepare creates a checked workspace for the benchmarks you choose. It copies declared input files, records their hashes, and gives the expected result filenames, columns, sample design, source citation, rights and limitations. Expected answers are not copied into the workspace.

After placing your pipeline's outputs in results/, run omicsbench suite grade. It verifies the staged inputs again, scores exactly the selected cases, and writes JSON, HTML, CSV and JUnit reports with a verifiable evidence ZIP. Missing and incorrect results remain visible. Existing suite index mappings work for native output paths and headers.

The installed package still includes the truth sets, so this is a practical test setup rather than a blinded challenge.

Install: python -m pip install openomicsbench==9.0.0

OpenOmicsBench v8.0.0

Version 8 adds a synthetic paired-read taxonomy assignment benchmark. Its 48 read pairs have declared read-level truth across three project-local mock reference codes, shared-reference ambiguity, and absent-reference background. The fixture is deterministic and rebuilds byte for byte.

omicsbench taxon compare scores a canonical two-column TSV or standard five-column Kraken 2 per-read output, including gzip-compressed files. It reports coverage, accuracy, per-code precision and recall, macro F1, and unclassified recall. omicsbench taxon classify is an exact-match example for these mock references. The codes are not NCBI taxids, and this fixture does not measure biological profiling accuracy.

Taxonomy results now work with result manifests, multi-assay evaluation, policy gates, method comparisons, regression checks, reports, and evidence ZIPs. omicsbench demo includes the sixth assay and labels the bundled-reference self-comparison in its reports.

Install: python -m pip install openomicsbench==8.0.0

See CHANGELOG.md for the complete change list.

v7.0.0

Version 7 makes OpenOmicsBench easier to try and browse.

• omicsbench demo OUTPUT_DIRECTORY creates a five-assay example with hash-locked inputs, JSON and HTML reports, and a verifiable evidence ZIP. It works from the installed package without a checkout or network connection.
• The new public catalog lets visitors search all 20 benchmark objects by assay, species, study and identifier, then inspect each dataset's files and manifest.
• The repository and PyPI pages now point directly to the first-run path and catalog.

The demo compares bundled reference answers with their own truth sets. It is a workflow example, not evidence of accuracy for an external method. The collection still contains 12 biological bulk RNA-seq objects and eight declared synthetic fixtures.

Install with python -m pip install openomicsbench, then run omicsbench demo my-first-run and open my-first-run/report.html.

OpenOmicsBench v6.0.0

Version 6 adds a deterministic genomic peak-calling benchmark for ATAC-seq, ChIP-seq, CUT&RUN and CUT&Tag software.

• interval-001 provides a compact bedGraph signal track, chromosome sizes and 12 narrowPeak truth intervals.
• omicsbench interval qc validates BED3, BED6, narrowPeak and gzip-compressed files.
• omicsbench interval compare reports reciprocal-overlap and base-pair precision, recall and F1.
• Genomic intervals now work with unified evaluation, result manifests, policy gates, regression checks, dashboards and evidence crates.
• The installed-wheel smoke test covers bulk RNA-seq, single-cell RNA-seq, genomic intervals, SNVs and peptide identifications together.

Validation: 187 tests, 22 registered objects, deterministic fixture rebuilds, clean wheel installation and a five-assay policy gate.

Install: python -m pip install openomicsbench==6.0.0

OpenOmicsBench 5.0.0

OpenOmicsBench 5.0.0 adds a deterministic single-cell RNA-seq benchmark to the existing bulk RNA, DNA and protein collection.

• singlecell-001: 90 synthetic cells, 60 features, three cell states and two batches in a 10x-style feature-barcode layout
• strict Matrix Market, barcode and feature validation with sparse cell-level QC
• label-invariant clustering comparison using ARI, NMI, purity and cell coverage
• single-cell support in result manifests, native column mappings, unified evaluations, policy gates, regression checks, dashboards and RO-Crate evidence
• 182 passing tests and a clean installed-wheel trial across four assay families

The single-cell object tests matrix handling, QC and recovery of a known synthetic partition. It is not a biological cell-type annotation benchmark.

Install or upgrade:

python -m pip install --upgrade openomicsbench
omicsbench validate singlecell-001
omicsbench cell compare singlecell-001 clusters.tsv

See CHANGELOG.md for the full release notes.