Skip to content

perf: chan's parallel mean-var algorithm for dask#4143

Open
ilan-gold wants to merge 6 commits into
mainfrom
ig/chan_mean_var_main
Open

perf: chan's parallel mean-var algorithm for dask#4143
ilan-gold wants to merge 6 commits into
mainfrom
ig/chan_mean_var_main

Conversation

@ilan-gold
Copy link
Copy Markdown
Contributor

@ilan-gold ilan-gold commented Jun 5, 2026

See https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Parallel_algorithm

Based on a #4118 (comment) with @zboldyga

This has two benefits - it allows us to calculate mean/var in one pass instead of effectively two (square sum and sum squared) and gets rid of a numerical instability issue that @zboldyga found the solution to (see removed comment)

  • Closes #
  • Tests included or not required because:

@ilan-gold ilan-gold added this to the 1.12.2 milestone Jun 5, 2026
@codecov
Copy link
Copy Markdown

codecov Bot commented Jun 5, 2026

Codecov Report

❌ Patch coverage is 96.49123% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.66%. Comparing base (d66537f) to head (17f706e).
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/scanpy/get/_aggregated.py 96.49% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4143      +/-   ##
==========================================
+ Coverage   79.59%   79.66%   +0.07%     
==========================================
  Files         120      120              
  Lines       12786    12833      +47     
==========================================
+ Hits        10177    10224      +47     
  Misses       2609     2609              
Flag Coverage Δ
hatch-test.low-vers 78.62% <29.82%> (-0.22%) ⬇️
hatch-test.pre 79.53% <96.49%> (+0.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/scanpy/get/_aggregated.py 93.72% <96.49%> (+0.42%) ⬆️

... and 1 file with indirect coverage changes

@ilan-gold ilan-gold marked this pull request as ready for review June 5, 2026 15:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant