Strange behaviour of np.average with negative weights

[apbard]

The current implementation of np.average accepts weights and the only check performed is that the sum of weights is not zero.
This is due to the fact the the result is normalised by the sum of weights.

This however leads to the following strange behaviour when dealing with negative weights:

In [1]: import numpy as np

In [2]: np.average([3,5], weights=[1, -0.99])
Out[2]: -194.99999999999986

In [3]: np.average([3,5], weights=[1, -1.01])
Out[3]: 204.9999999999998

In [4]: np.average([3,5], weights=[0, -1])
Out[4]: 5.0

In [5]: np.average([3,5], weights=[0, +1])
Out[5]: 5.0

wouldn't be better to normalise by the absolute sum or the sum of absolute values? In this way we would get:

# Case absolute sum
In [4]: average([3,5], weights=[0, -1])
Out[4]: -5.0

In [5]: average([3,5], weights=[0, +1])
Out[5]: 5.0

In [6]: average([3,5], weights=[1, -0.99])
Out[6]: -194.99999999999986

In [7]: average([3,5], weights=[1, -1.01])
Out[7]: -204.9999999999998

or

# Case sum of absolute values
In [4]: average([3,5], weights=[0, -1])
Out[4]: -5.0

In [5]: average([3,5], weights=[0, +1])
Out[5]: 5.0

In [6]: average([3,5], weights=[1, -0.99])
Out[6]: -0.97989949748743732

In [7]: average([3,5], weights=[1, -1.01])
Out[7]: -1.0199004975124377

Finally, is there a realistic use case where negative weights are needed?

[kenogo]

I don't know of any practical use for negative weights, and the current results are what's expected from the definition I know: not taking absolute values anywhere. If you'd really want to support a special feature for that kind of thing, I feel like your second example of using a sum of absolute values is more intuitive: A negative weight would be the same as a positive weight with a negative value.

[apbard]

the current results are what's expected from the definition I know: not taking absolute values anywhere

Actually, IMHO, the current behaviour is quite surprising, an "epsilon" change in weights lead to a huge change in the result, or a change in sign of all weights result in no change at all.
The standard definition of weighted average assumes non-negative weights, thus there is no difference between absolute values or not.

I feel like your second example of using a sum of absolute values is more intuitive.

Yes, exactly. Totally agree, I proposed the first one just as an intermediate step, because it solves the "continuity" of the average and the change of sign but not the explosion of values even for small differences in weights absolute values.

I don't know of any practical use for negative weights

I asked this to understand if there is a use case and what would be the expected result in that case.

[kenogo]

@apbard Well ok I'll have to agree that the current results are not intuitive at all. I think it would be a good idea to decide between either throwing an exception for negative weights or dividing by the sum of their absolute values.

[eric-wieser]

Sum of absolute values sounds wrong to me. These two should have the same results IMO

np.average([2, 2, 1], weight=[20,-19,1])
np.average([2, 1], weight=[1, 1])

Absolute of sum would be reasonable, but so would throwing an error if the sum is negative

[apbard]

@eric-wieser Yes, you are right. They should return the same result. At the same time, however, I would expect:

np.average([4, 4], weights=[1, -1])

to return 0.
Probably the best thing to avoid confusion would be to stick with the standard definition and raise an error if weights are negative. I'll make a PR for this.

[apbard]

@eric-wieser In the PR you say that negative weights are useful. Can you show me some use cases?
In any case what about the strange/not intuitive behaviour of the current implementation?
I have nothing against one of the other solutions (both have peculiar cases) but IMHO the current one should somehow be fixed.

[eric-wieser]

I would expect np.average([4, 4], weights=[1, -1]) to return 0.

Your intuition is wrong, for the same reason that np.average([]) should not return 0. An identity that needs to be respected is that average(x + arr, ...) = x + average(arr, ...), which returning 0 in your case would violate.

[pmli]

@apbard Weighted average with nonnegative weights, where at least one weight is positive, can be thought of as a convex combination (the coefficients in this convex combination are then the weights normalized to sum to 1). If some weights are negative, as long as they don't sum to 0, we instead have an affine combination.

I would argue that when the weights are normalized to sum to 1, and the geometric interpretation of convex/affine combination is kept in mind, there is no nonintuitive behavior. Taking your first example with weights (1, -0.99) and (1, -1.01), the normalized weights are (100, -99) and (-100, 101), so there is no more epsilon difference. In the second example, (0, -1) and (0, 1) both normalize to (0, 1), so it's clear that the change in sign has no effect.

Concerning np.average([4, 4], weights=[1, -1]), it is undefined because of division by zero, but since (w1 * 4 + w2 * 4) / (w1 + w2) = 4 when w1 + w2 != 0, it can be extended by continuity to be 4 for all w1 and w2. But since this argument cannot be used for, e.g. np.average([4, 5], weights=[1, -1]), or whenever the points are not repeating, it's probably better just to raise an exception when the sum of weights is zero.

There are other interpretations of the weighted average, e.g. as the minimizer of the weighted sum of squared distances to all points (as opposed to the median, which minimizes the sum of non-squared distances to all points). If I did my math correctly, this definition can be extended to when the sum of all weights is positive, with possibly some negative weights.

There are probably other interpretations, but I don't know of any that would treat the case of negative weights in a special way.

[apbard]

I would argue that when the weights are normalized to sum to 1, and the geometric interpretation of convex/affine combination is kept in mind, there is no nonintuitive behavior. Taking your first example with weights (1, -0.99) and (1, -1.01), the normalized weights are (100, -99) and (-100, 101), so there is no more epsilon difference. In the second example, (0, -1) and (0, 1) both normalize to (0, 1), so it's clear that the change in sign has no effect.

@pmli I do not understand how the interpretation you are suggesting is "solving"/"explaining" my examples...you start from weights [(100, -99) and (100, -101)] that are very close in the "weight space" and you end up with ones that are opposite [(100, -99) and (-100, 101)] and so are the resulting averages.
that's why I think that normalising by the absolute sum would be probably more intuitive.

[pmli]

@apbard What I wanted to say is that affine combination is, to me, more intuitive than weighted average. For this reason, I prefer to think in terms of normalized weights.
I do not understand why would using absolute values make things more intuitive. It seems to me you want to use a continuity argument, but neither (w1 * 3 + w2 * 5) / |w1 + w2|, (w1 * 3 + w2 * 5) / (|w1| + |w2|), nor np.average([3, 5], weights=[w1, w2]) are continuous over the whole R^2, and so there will always be weights very close in the "weight space" that give relatively distant averages. (On the other hand, w1 * 3 + w2 * 5 is continuous over w1 + w2 = 1, but I prefer affine combination already for the geometric interpretation.)

[eric-wieser]

it's probably better just to raise an exception when the sum of weights is zero.

Actually that seems like a corner case that would be nice to support, and might come up in cases where the weights are computed via some metric that degenerates in the all-the-same-case.

[apbard]

@pmli on the sum of absolute values (w1 * 3 + w2 * 5) / (|w1| + |w2|) you are right.
As explained by @eric-wieser it does not satisfy the property average(x + arr, ...) = x + average(arr, ...) thus I now agree that is wrong

instead (w1 * 3 + w2 * 5) / |w1 + w2| basically differs from the current one only when the sum is negative...but preserve the continuity when the sum is close to zero and makes this properties hold...(they seem reasonable to me):
average(x, w) = -average(x, -w) = -average(-x, w) = average(-x, -w)

we also could raise when the sum is negative, this has no negative impact on performance, is robust to rounding error and still allows some negative weights.

@eric-wieser what do you think?