show median in benchmark results

[scoder]

BPO	22881
Nosy	@pitrou, @scoder, @vstinner, @serhiy-storchaka, @wm75
Files	show_median.patch: show median in addition to min and avg timings show_median.patch: update: use average of middle values for even sample size show_median.patch: update: fix median calculation for even number of samples

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-09-01.19:27:35.757>
created_at = <Date 2014-11-16.09:50:17.907>
labels = ['type-feature', 'performance']
title = 'show median in benchmark results'
updated_at = <Date 2016-09-01.19:27:35.756>
user = 'https://github.com/scoder'

bugs.python.org fields:

activity = <Date 2016-09-01.19:27:35.756>
actor = 'scoder'
assignee = 'none'
closed = True
closed_date = <Date 2016-09-01.19:27:35.757>
closer = 'scoder'
components = ['Benchmarks']
creation = <Date 2014-11-16.09:50:17.907>
creator = 'scoder'
dependencies = []
files = ['37206', '37207', '39279']
hgrepos = []
issue_num = 22881
keywords = ['patch']
message_count = 16.0
messages = ['231239', '231241', '231243', '242050', '242078', '242079', '242080', '242081', '242461', '242463', '242464', '242465', '242466', '242467', '242469', '273926']
nosy_count = 5.0
nosy_names = ['pitrou', 'scoder', 'vstinner', 'serhiy.storchaka', 'wolma']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue22881'
versions = []

[scoder]

The median tends to give a better idea about benchmark results than an average as it inherently ignores outliers.

[serhiy-storchaka]

In case of even number of samples the median value is calculated as arithmetic mean of two middle samples.

med_base = (base_times[len(base_times)//2] + base_times[(len(base_times)-1)//2]) / 2

[scoder]

Fair enough, patch updated.

[scoder]

Any more comments on the patch, or can it be applied?

[wm75]

for the even number case, I think you shouldn't do // 2, but / 2.

In general, wouldn't it be good to let the statistics module do all the stats calculations?

[scoder]

In general, wouldn't it be good to let the statistics module do all the stats calculations?

It's not available in older Python versions, e.g. 2.6.

[wm75]

It's not available in older Python versions, e.g. 2.6.

I know, I was talking about 3.5+, of course. This would not be backported to Python2 anyway, would it?

[wm75]

ah sorry, it's late here already and I forgot what file this change is about. So forget my last comment then.

[scoder]

for the even number case, I think you shouldn't do // 2, but / 2.

Right. I updated the patch.

[pitrou]

Have you found the median to be more stable than the minimum here?

[scoder]

I'm actually not sure how it relates to the minimum. The more runs you have, the higher the chance of hitting the actual minimum at least once. And if none of the runs hits the real minimum, you're simply out of luck.

However, it should tend to give a much better result than the (currently printed) average, which suffers from outliers. And outliers are almost always too high for benchmarks and never too low, due to various external influences.

[pitrou]

Then let's just replace the average with the median? I don't think it makes sense to add more statistical information to the output (IMHO, there is already too much of it :-)).

[serhiy-storchaka]

May be just drop 5% of largest values to avoid the impact of outliers?

See also bpo-23552.

[scoder]

Well, we can apply a kludge, or apply statistics.

[pitrou]

May be just drop 5% of largest values to avoid the impact of outliers?

In fast mode (option "-f"), there may not be enough samples for that.

[vstinner]

The new https://github.com/python/performance benchmark suite now displays the median rather than the arithmetic mean (average) by default (it also displays the standard deviation).

See the perf issue for a discussion about median vs mean:
psf/pyperf#1

Can we now close this issue?