statistics.median does not work with ordinal scale, add doc

[WdeW]

BPO	33573
Nosy	@terryjreedy, @taleinat, @stevendaprano
PRs	bpo-33573: improve docs to suggest statistics.median() alternatives for non-numeric data #7587 [3.7] bpo-33573: docs to suggest median() alternatives for non-numeric data (GH-7587) #7906 [3.6] bpo-33573: docs to suggest median() alternatives for non-numeric data (GH-7587) #7907
Files	testMedian.py: simple demonstraion of failure

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-06-25.11:27:42.421>
created_at = <Date 2018-05-18.19:29:45.729>
labels = ['3.7', '3.8', 'type-feature', 'library', 'docs']
title = 'statistics.median does not work with ordinal scale, add doc'
updated_at = <Date 2018-06-25.11:27:42.421>
user = 'https://bugs.python.org/WdeW'

bugs.python.org fields:

activity = <Date 2018-06-25.11:27:42.421>
actor = 'taleinat'
assignee = 'docs@python'
closed = True
closed_date = <Date 2018-06-25.11:27:42.421>
closer = 'taleinat'
components = ['Documentation', 'Library (Lib)']
creation = <Date 2018-05-18.19:29:45.729>
creator = 'W deW'
dependencies = []
files = ['47601']
hgrepos = []
issue_num = 33573
keywords = ['patch']
message_count = 10.0
messages = ['317048', '317120', '317122', '317125', '317248', '317694', '319219', '320414', '320416', '320417']
nosy_count = 5.0
nosy_names = ['terry.reedy', 'taleinat', 'steven.daprano', 'docs@python', 'W deW']
pr_nums = ['7587', '7906', '7907']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue33573'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[WdeW]

The 0.5-quantile or median is defined for ordinal, interval, and ratio scales. An Enumerator as derived from Enum and extended with rich comparison methods implements an ordinal scale. Therefore calculating the median over a list of such enum-elements ought to be possible.

The current implementation tries to interpolate the median value by averaging the two middle observations. This is allowed for interval and ratio scales, but since this interpolation involves an addition, not so for ordinal scales. Although computationally it is possible to do this for numeric ordinal variables, logically it is non-sense for the distance between ordinal values is - by definition - unknown. On non-numeric ordinal values it is even computationally impossible.

The correct return value would be: the first value in an ordered set where al least half the number of observations is smaller or equal than it. This is observation[len(observation)//2] for odd and even length ordered lists of values.

Whether the same applies to interval and ratio scales is a matter of opinion. The currently implemented algorith definitely is more popular these days.

[stevendaprano]

For ordinal scales, you should use either median_low or median_high.

I don't think the standard median function ought to choose for you whether to take the low or high median. It is better to be explicit about which you want, by calling the relevant function, than for median to guess which one you need.

[stevendaprano]

By the way, this isn't a crash (that's for things which cause the interpreter to segfault). I'm marking this as Not a bug, but I'm open to suggestions to improve either the documentation or the median functions.

[stevendaprano]

What do you think of adding a note in the documentation for median?

"If your data is ordinal (supports order operations) but not numeric (doesn't support addition), you should use median_low or median_high instead."

[WdeW]

Changing the documentation in tis way seems to me an excellent and easy way to solve the issue.

[terryjreedy]

I agree.

[taleinat]

PR ready for review.

[taleinat]

New changeset fdd6e0b by Tal Einat in branch 'master':
bpo-33573: docs to suggest median() alternatives for non-numeric data (GH-7587)
fdd6e0b

[taleinat]

New changeset 150cd3c by Tal Einat (Miss Islington (bot)) in branch '3.7':
[3.7] bpo-33573: docs to suggest median() alternatives for non-numeric data (GH-7587) (GH-7906)
150cd3c

[taleinat]

New changeset 8fd8cfa by Tal Einat (Miss Islington (bot)) in branch '3.6':
[3.6] bpo-33573: docs to suggest median() alternatives for non-numeric data (GH-7587) (GH-7907)
8fd8cfa