Context
As we add perturbation-prediction models, we need a standardized, honest way to evaluate them.
There is strong evidence in the field that trivial baselines (control mean, additive model) often match or beat elaborate models, so baselines must be first-class, not an afterthought.
PerturbationComparison is a start but is limited.
What's missing
- Standard train/test splits for perturbation prediction: held-out perturbations, held-out combinations, held-out cell types.
- Built-in baselines: control mean, additive/linear, nearest-perturbation (reuse
DistanceSpace.nearest_perturbations).
- Standard metrics: per-DEG delta / logFC correlation, E-distance to ground truth, direction (sign) accuracy, top-k DEG overlap.
- A simple leaderboard-style summary over models × metrics × splits.
Proposal / API
A pertpy.tools evaluator that takes ground-truth and predicted AnnData (as returned by the prediction models) and returns a tidy results DataFrame, reusing the existing Distance metrics.
Why it matters
Gives users — and reviewers — an honest, reproducible answer to "is this model actually better than doing nothing clever?".
This is exactly the rigor scverse is trusted for, and it is the natural companion to a broader prediction module.
Related
Prior discussion in #173 (benchmarking tool speed) is about runtime, not predictive accuracy — this is complementary.
Context
As we add perturbation-prediction models, we need a standardized, honest way to evaluate them.
There is strong evidence in the field that trivial baselines (control mean, additive model) often match or beat elaborate models, so baselines must be first-class, not an afterthought.
PerturbationComparisonis a start but is limited.What's missing
DistanceSpace.nearest_perturbations).Proposal / API
A
pertpy.toolsevaluator that takes ground-truth and predictedAnnData(as returned by the prediction models) and returns a tidy resultsDataFrame, reusing the existingDistancemetrics.Why it matters
Gives users — and reviewers — an honest, reproducible answer to "is this model actually better than doing nothing clever?".
This is exactly the rigor scverse is trusted for, and it is the natural companion to a broader prediction module.
Related
Prior discussion in #173 (benchmarking tool speed) is about runtime, not predictive accuracy — this is complementary.