Stats hypothesis testing

Development build. This page describes main, not a released package. The latest published Lodestar.Stats is 0.4.0 — read its documentation.

Home › Stats

Hypothesis testing

Lodestar.Stats answers three questions. Ten families ask is this difference more than noise? Three ask are these two variables related? — the same machinery, pointed at a pair of measurements rather than at two groups. Five more ask does the assumption underneath the answer hold?, which is the question a reader reaches last and should have reached first.

What happens to a missing value

By default a NaN anywhere in a sample reaches the statistic and the p-value, which is scipy's nan_policy='propagate' and what every test here did before the parameter existed. Pass NanPolicy.Omit to drop the missing observations instead, or NanPolicy.Raise to refuse the input. For a paired test, Omit drops the pair — filtering the two samples separately would change what is being tested, which is the mistake the parameter exists to prevent.

Which test

you have	and you assume	use
two independent samples	roughly normal	TTest.Independent
two independent samples	nothing about the shape	MannWhitney.Test
the same subjects measured twice	roughly normal differences	TTest.Paired
the same subjects measured twice	nothing about the shape	Wilcoxon.Paired
counts in categories	a stated expected distribution	ChiSquare.GoodnessOfFit
a contingency table	cells large enough for the approximation	ChiSquare.Contingency
a 2×2 table with small cells	nothing	FisherExact.Test
two samples, whole distributions	nothing	KolmogorovSmirnov.TwoSample
three or more groups	roughly normal, similar spread	OneWayAnova.Test
three or more groups	nothing about the shape	KruskalWallis.Test
one sample, and a normality assumption to check	nothing	ShapiroWilk.Test
many p-values at once	nothing	MultipleComparisons
two measurements per subject	a linear relationship, roughly normal	Pearson.Test
two measurements per subject	only that the relationship rises or falls	Spearman.Test
two rankings of the same items	nothing, and the sample is short	KendallTau.Test
several groups, before trusting an ANOVA	nothing about the shape	Levene.Test
several groups, before trusting an ANOVA	each group roughly normal	Bartlett.Test
three or more treatments on the same subjects	nothing about the shape	Friedman.Test
a count of successes out of a total	nothing; it is exact at any size	Binomial.Test
one sample, and a normality assumption to check	nothing	AndersonDarling.Test

The assumption under the answer

Two of the tests above are not there to answer a question of their own. They are there because another test on this page has already assumed something, and until now this package gave a caller no way to find out whether the assumption held.

OneWayAnova.Test assumes the groups share one variance. TTest.Independent assumes it too, unless Variance.Welch is asked for. When that is false, their p-values are not conservative — they are simply wrong, and they are wrong in the direction that produces findings.

using Lodestar.Stats;

double[] first = [20.1, 19.8, 20.3, 20.0, 19.9, 20.2, 20.1, 19.7];
double[] second = [20.4, 18.9, 21.2, 19.1, 21.0, 18.7, 20.8, 19.4];
double[] third = [20.0, 20.1, 19.9, 20.2, 19.8, 20.1, 20.0, 19.9];

double meansAgree = Math.Round(OneWayAnova.Test(first, second, third).PValue, 4);
double spreadsDiffer = Math.Round(Levene.Test(first, second, third).PValue, 10);

meansAgree is 0.9654 and spreadsDiffer is 2.4E-08. Three machines fill the same bottle; they agree on where they aim and disagree, by a factor of forty million in the p-value, on how well they hold it. A report that ran only the ANOVA would say the three are interchangeable.

Which of the two variance tests to run is a question about the data. Levene.Test around the median assumes nothing about the shape and is the safer default. Bartlett.Test assumes each group is normal, is sharper when that holds, and reports a difference that is not there when it does not — so it is the second question, after AndersonDarling.Test or ShapiroWilk.Test has answered the first.

A test of an assumption is not a licence. Failing to reject is not evidence that the assumption holds — on a short sample these tests reject almost nothing, which says more about the sample than about the variances. Their value is in the other direction: when one of them does reject, the conclusion standing on that assumption has to be withdrawn.

Which correlation

The three differ in what counts as a relationship, and the difference is not a matter of taste.

Pearson.Test measures a line. It reads the distances between the values, so one observation far from the rest moves it a long way, and a relationship that rises steadily but not linearly reads lower than it is.

Spearman.Test measures order, by correlating the mid-ranks instead of the values. Every relationship that only rises answers exactly 1, however it curves.

The same eight pairs, both ways:

using Lodestar.Stats;

double[] dose = [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 40.0];
double[] effect = [2.0, 3.5, 4.0, 5.5, 6.0, 7.5, 8.0, 9.0];

double linear = Math.Round(Pearson.Test(dose, effect).Statistic, 4);
double monotone = Math.Round(Spearman.Test(dose, effect).Statistic, 4);

linear is 0.6749 and monotone is 1. Neither is wrong: the last dose is five times the one before it, so a measure of distance is entitled to say the relationship is not a line, and a measure of order is entitled to say it never once went down. Choose by which of the two claims you meant to make.

KendallTau.Test also measures order, but counts agreeing pairs rather than correlating ranks. It moves less under a single odd observation than rho does, it has an exact null distribution for short untied samples where rho has only an approximation, and its value has a direct reading: the probability that two entries are ordered the same way by both measurements, minus the probability that they are not. Rho is the more familiar number; tau is the better behaved one on a short sample.

A correlation is not a cause, and none of the three says anything about which measurement moved the other — or whether a third one moved both.

What a p-value is, and is not

A p-value is the probability of seeing a difference at least this large if the null hypothesis is true. It is not the probability that the null hypothesis is true, and it is not the probability that your result is a fluke. A p-value of 0.03 does not mean there is a 3 % chance you are wrong.

Two consequences worth acting on:

• 0.049 and 0.051 are the same evidence. The threshold is a convention, not a discovery. Report the number.
• Twenty tests at 5 % produce one significant result by chance. That is what MultipleComparisons is for, and it is not optional once you are testing more than a couple of things.

The step-up procedure below is MultipleComparisons.BenjaminiHochberg, the one most people reach for first:

using Lodestar.Stats;

double[] pValues = [0.001, 0.008, 0.039, 0.041, 0.042];

double[] adjusted = MultipleComparisons.BenjaminiHochberg(pValues);

bool stillSignificant = adjusted[0] < 0.05;   // => True

One default that is not scipy's

TTest.Independent defaults to Welch's test; scipy.stats.ttest_ind defaults to Student's. Pooling the two variances is only correct when the populations really share one, which is an assumption most callers have not checked. Pass Variance.Equal for scipy's default. Everything else in this package matches scipy.stats 1.18.0 exactly, and the equivalence table is the row-by-row map, including the handful of places one call refuses rather than answering — a NaN, a warning, or a number a caller did not ask for.

Exact and asymptotic

Three tests carry both an exact null distribution and a normal approximation to it, selected by ExactMethod:

• Auto — exact for a small, untied sample; asymptotic otherwise. What scipy's method='auto' does, and the same thresholds.
• Exact — always the exact distribution. On tied data the number is only approximate, because ties break the equal-probability argument the enumeration rests on; scipy computes there too rather than refusing, and so does this.
• Asymptotic — always the normal approximation, whatever the sample size.

The branch changes the number, not just the running time, which is why it is a parameter and not a hidden optimisation. Each of the three tests also refuses ExactMethod.Exact past its own size bound — the exact table is O(n·m) or worse to build, and Auto never crosses that bound on its own, falling back to the asymptotic answer instead. The three reference pages (MannWhitney.Test, Wilcoxon.Paired, KolmogorovSmirnov.TwoSample) each state their own bound, because it is not the same number twice.

The incumbents, and the one measured

MathNet.Numerics 5.0.0 (2022-04-03, 74.7M downloads) is the dominant third-party numerical library for .NET, and it ships probability distributions and descriptive statistics — no hypothesis tests. ML.NET does prediction: a t-test exists only in Azure ML Studio (classic), a retired hosted product, and Mann-Whitney only in Kusto/KQL — neither is a .NET library a project can reference.

Three .NET libraries do carry the tests, and this page said for a while that only the first did (decision 0004 has the reading that corrected it):

• Accord.Statistics 3.8.0, LGPL-2.1, last published on 2017-10-19; its framework, accord-net/framework (4.5k stars), was archived by its owner on 2020-11-19.
• Meta.Numerics 4.2.0, MS-PL, netstandard2.0, published 2025-07-14 after five years without a release. It carries eight of this package's ten families — the t-tests, Mann-Whitney, Wilcoxon, Kruskal-Wallis, Kolmogorov-Smirnov, one-way ANOVA, Fisher's exact test and the chi-square table — and Shapiro-Francia rather than Shapiro-Wilk. Measured (#756): this package is ahead on seven of the eight at both 100 and 10,000 samples, from 1.32× on Kruskal-Wallis to 5.95× on the chi-square table, with the signed-rank row a wash at 100 and a win at 10,000. It was behind on two, and both were costs here rather than differences in what the libraries compute: Fisher's exact test went from 8.2× behind to 3.49× ahead, and the equal-size exact Kolmogorov-Smirnov from 12.8× behind to 1.47× ahead — the same p-values, and ExactMethod.Auto still choosing the exact branch that scipy's own method="auto" chooses. Its chi-square applies no continuity correction, so its statistic is 9.09091 where this package's is 7.91919 on the same table, and its signed-rank statistic follows a different convention. Neither is a disagreement about the data: the write-up in docs/guides/performance.md lists all six differences with their causes.
• Numerics.NET 10.7.0, formerly Extreme Optimization, maintained and commercial. Named so its absence from the benchmarks is not mistaken for an absence from .NET.

Accord.Statistics is the one measured so far, because #442's own constraint asks for a named .NET incumbent where one exists at all. TTest.Independent, MannWhitney.Test and ChiSquare.Contingency are benchmarked and cross-checked against it in bench/README.md and docs/guides/performance.md — archived is not the same as absent, and this package's own oracle discipline means the comparison is also a second opinion on scipy, not only a timing.