[statistics] Division by 2 in statistics.median

[jfine2358]

BPO	35698
Nosy	@rhettinger, @vstinner, @stevendaprano, @MojoVampire, @remilapeyre, @jfine2358

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/stevendaprano'
closed_at = <Date 2019-01-14.13:08:20.433>
created_at = <Date 2019-01-09.11:46:12.566>
labels = ['type-bug', 'invalid']
title = '[statistics] Division by 2 in statistics.median'
updated_at = <Date 2019-01-14.13:26:42.916>
user = 'https://github.com/jfine2358'

bugs.python.org fields:

activity = <Date 2019-01-14.13:26:42.916>
actor = 'vstinner'
assignee = 'steven.daprano'
closed = True
closed_date = <Date 2019-01-14.13:08:20.433>
closer = 'steven.daprano'
components = []
creation = <Date 2019-01-09.11:46:12.566>
creator = 'jfine2358'
dependencies = []
files = []
hgrepos = []
issue_num = 35698
keywords = []
message_count = 15.0
messages = ['333305', '333309', '333327', '333340', '333349', '333374', '333385', '333386', '333398', '333400', '333410', '333539', '333618', '333619', '333620']
nosy_count = 6.0
nosy_names = ['rhettinger', 'vstinner', 'steven.daprano', 'josh.r', 'remi.lapeyre', 'jfine2358']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35698'
versions = []

[jfine2358]

When len(data) is odd, median returns the average of the two middle values. This average is computed using
i = n//2
return (data[i - 1] + data[i])/2

This results in the following behaviour

>> from fractions import Fraction
>> from statistics import median
>> F1 = Fraction(1, 1)

>>> median([1])
1
>>> median([1, 1]) # Example 1.
1.0

>>> median([F1])
Fraction(1, 1)
>>> median([F1, F1])
Fraction(1, 1)

>>> median([2, 2, 1, F1]) # Example 2.
Fraction(3, 2)

>>> median([2, 2, F1, 1]) # Example 3.
1.5

Perhaps, when len(data) is odd, it would be better to test the two middle values for equality. This would resolve Example 1. It would not help with Examples 2 and 3, which might not have a satisfactory solution.

See also bpo-33084.

[vstinner]

When len(data) is odd, median returns the average of the two middle values.

I'm not sure that I understand your issue. Do you consider that it's a bug? It's part of the definition of the median function, no?
https://en.wikipedia.org/wiki/Median#Finite_set_of_numbers

[remilapeyre]

What do you think median([1, 1.0]) should return?

[MojoVampire]

vstinner: The problem isn't the averaging, it's the type inconsistency. In both examples (median([1]), median([1, 1])), the median is unambiguously 1 (no actual average is needed; the values are identical), yet it gets converted to 1.0 only in the latter case.

I'm not sure it's possible to fix this though; right now, there is consistency among two cases:

1. When the length is odd, you get the median by identity (and therefore type and value are unchanged)
2. When the length is even, you get the median by adding and dividing by 2 (so for ints, the result is always float).

A fix that changed that would add yet another layer of complexity:

1. When the length is odd, you get the median by identity (and therefore type and value are unchanged)
2. When the length is even,
   a. If the two middle values are equal (possibly only if they have equal types as well, to resolve the issue with [1, 1.0] or [1, True]), return the first of the two middle values (median by identity as in Support "bpo-" in Misc/NEWS #1)
   b. Otherwise, you get the median by adding and dividing by 2

And note the required type checking in 2a required to even make it that consistent. Even if we accepted that, we'd pretty quickly get into a debate over whether median([3, 5]) should try to return 4 instead of 4.0, given that the median is representable in the source type (which would further damage consistency).

If anything, I think the best design would have been to *always* include a division step (so odd length cases performed middle_elem / 1, while even did (middle_elem1 + middle_elem2) / 2) so the behavior was consistent regardless odd vs. even input length, but that shipped has probably sailed, given the documented behavior specifically notes that the precise middle data point is itself returned for the odd case.

I think the solution for people concerned is to explicitly convert int values to be median-ed to fractions.Fraction (or decimal.Decimal) ahead of time, so floating point math never gets involved, and the return type is consistent regardless of length.

[vstinner]

vstinner: The problem isn't the averaging, it's the type inconsistency.

>>> type(statistics.median([1]))
<class 'int'>
>>> type(statistics.median([1,2]))
<class 'float'>

Which consistency? :-)

[jfine2358]

I read PEP-450 as saying that statistics.py can be used by "any secondary school student". This is not true for most Python libraries.

In this context, the difference between a float and an int is important. Consider
statistics.median([2] * n)

As a secondary school student, knowing the definition of median, I might expect the value to be 2, for any n > 0. What else could it be. However, the present code gives 2 for n odd, and 2.0 for n even.

I think that this issue is best approached by taking the point of view of a secondary school student. Or perhaps even a primary school student who knows fractions. (A teacher might use statistics.py to create learning materials.)

By the way, 2 and 2.0 are not interchangeable. For example
>>> [1] * 2.0
TypeError: can't multiply sequence by non-int of type 'float'

[remilapeyre]

As a secondary school student, knowing the definition of median, I might expect the value to be 2, for any n > 0.

The secondary school student would be wrong, wouldn't he?

The median of a set is not expected to be a part of the set. Especially for ints since division by 1 or 2 is not closed for integers.

Would the same student expect median([2, 4, 6, 8]) to be part of the set of even integers?

I think one taking the median of a set should always ready to deal with floating point arithmetic the result is not guaranteed to be an integer. Going from hoops to make it so when it is equivalent to an integer is rather misleading.

[vstinner]

I suggest to close the issue as "not a bug". IMHO statistics.median() respects the defintion of the mathematical median function.

[jfine2358]

Here's the essence of a patch.

Suppose the input is Python integers, and the output is a mathematical integer. In this case we can make the output a Python integer by using the helper function

>>> def wibble(p, q):
...     if type(p) == type(q) == int and p%q == 0:
...         return p // q
...     else:
...         return p / q
... 
>>> wibble(4, 2)
2
>>> wibble(3, 2)
1.5

This will also work for average.

[remilapeyre]

This does not do what you want:

    >>> class MyInt(int): pass
    >>> wibble(MyInt(4), MyInt(2))
    2.0

and a patch is only needed if something is broken.

I'm with vstinner of the opinion that nothing is broken and vote to close this issue.

[jfine2358]

It might be better in my sample code to write
isinstance(p, int)
instead of
type(p) == int
This would fix Rémi's example. (I wanted to avoid thinking about (False // True).)

For median([1, 1]), I am not claiming that 1.0 is wrong and 1 is right. I'm not saying the module is broken, only that it can be improved.

For median([1, 1]), I believe that 1 is a better answer, particularly for school students. In other words, that making this change would improve Python.

As a pure mathematician, to me 1.0 means a number that is close to 1. Whereas 1 means a number that is exactly 1..

[rhettinger]

As a pure mathematician, to me 1.0 means a number that is
close to 1. Whereas 1 means a number that is exactly 1.

Descriptive statistics performed on a computer using actual measurements is pretty far from "pure mathematics" ;-)

Making this change is likely pointless for most users and likely confusing for others (i.e. why the type switch between median([1, 1]) and median([1, 3]).

I concur with Victor and recommend closing.