grandmean

The Hugging Face evaluate mahalanobis metric centers its input with a scalar grand mean while inverting a per-feature covariance. The two terms of the quadratic form then live on incompatible bases, and the metric returns a nonzero distance for a point that sits exactly at the center of the distribution, where the Mahalanobis distance is zero.

The bug

The Mahalanobis distance of a point x from a distribution with per-feature mean vector mu and covariance C is D^2 = (x - mu)^T C^-1 (x - mu). In metrics/mahalanobis/mahalanobis.py:

X_minus_mu = X - np.mean(reference_distribution)   # line 91
cov        = np.cov(reference_distribution.T)      # line 92

np.cov(reference_distribution.T) is a per-feature covariance of shape (n_features, n_features), but np.mean(reference_distribution) has no axis argument, so it returns the scalar mean of every element, the grand mean. Subtracting one scalar from every feature is not the per-feature centering the quadratic form needs. The centering should be np.mean(reference_distribution, axis=0), the per-feature mean vector, which is what the covariance on the next line already assumes.

What it does

For a point at the distribution center the true distance is 0, but the shipped metric returns a large value, and the error grows as the feature means spread apart:

per-feature mean = [-0.027, 9.931], scalar grand mean = 4.952
query at the distribution center:
  correct Mahalanobis D^2 = 0.0000
  shipped metric      D^2 = 52.4758

distance at the center as the two feature means spread apart:
  gap  0.0: correct 0.0000, shipped 0.0009
  gap  1.0: correct 0.0000, shipped 0.3801
  gap  5.0: correct 0.0000, shipped 12.3002
  gap 10.0: correct 0.0000, shipped 48.0756

The error vanishes only when every feature has the same mean, so the scalar grand mean happens to equal the per-feature mean vector. That is exactly the case in the metric's shipped doctest, whose symmetric input has equal feature means, which is why the doctest passes and the bug goes unnoticed.

Layout

• excerpt.py: the centering, the covariance, and the documented Mahalanobis form, quoted with line numbers from metrics/mahalanobis/mahalanobis.py (Apache-2.0).
• mahalanobis.py: the shipped scalar-mean centering and the correct per-feature centering, side by side, both against the per-feature covariance.
• consequence.py: the nonzero distance at the center, its growth with feature-mean spread, and the equal-means case that hides it.
• test_grandmean.py: correct is zero at the center and shipped is not, the error grows with mean spread, equal means agree, and correct matches an explicit quadratic form.

Reproduce

python mahalanobis.py
python consequence.py
python test_grandmean.py

The lines are quoted from the main branch; the fix is np.mean(reference_distribution, axis=0).