Add geometric mean to `statistics` module

[cool-RR]

BPO	27181
Nosy	@rhettinger, @mdickinson, @stevendaprano, @cool-RR, @vadmium, @koobs, @csabella
PRs	bpo-27181: Add statistics.geometric_mean() #12638

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/stevendaprano'
closed_at = <Date 2019-04-07.16:21:07.856>
created_at = <Date 2016-06-02.12:24:07.764>
labels = ['3.8', 'type-feature', 'library']
title = 'Add geometric mean to `statistics` module'
updated_at = <Date 2019-04-07.16:21:07.854>
user = 'https://github.com/cool-RR'

bugs.python.org fields:

activity = <Date 2019-04-07.16:21:07.854>
actor = 'rhettinger'
assignee = 'steven.daprano'
closed = True
closed_date = <Date 2019-04-07.16:21:07.856>
closer = 'rhettinger'
components = ['Library (Lib)']
creation = <Date 2016-06-02.12:24:07.764>
creator = 'cool-RR'
dependencies = []
files = []
hgrepos = []
issue_num = 27181
keywords = ['patch']
message_count = 45.0
messages = ['266948', '266969', '267051', '267253', '267990', '268008', '268020', '268022', '270025', '270026', '270033', '272214', '272217', '272218', '272224', '272225', '272488', '272504', '272525', '272526', '272640', '272659', '272807', '272880', '272881', '272882', '272913', '272970', '275927', '276057', '276060', '276151', '278056', '278059', '278076', '278078', '300918', '335662', '335720', '338742', '339082', '339094', '339319', '339579', '339580']
nosy_count = 8.0
nosy_names = ['rhettinger', 'mark.dickinson', 'steven.daprano', 'cool-RR', 'python-dev', 'martin.panter', 'koobs', 'cheryl.sabella']
pr_nums = ['12638']
priority = None
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue27181'
versions = ['Python 3.8']

[rhettinger]

Steven, this seems like a reasonable suggestion (though I would expect someone else will immediately suggest a harmonic mean as well). Is this within the scope of what you were trying to do with the statistics module?

[stevendaprano]

On Thu, Jun 02, 2016 at 09:04:54PM +0000, Raymond Hettinger wrote:

Steven, this seems like a reasonable suggestion (though I would expect
someone else will immediately suggest a harmonic mean as well). Is
this within the scope of what you were trying to do with the
statistics module?

Yes, I think it is reasonable too. I'll aim to get this in to 3.6.

[cool-RR]

To complicate things further...

I implemented a geometric mean on my own, and then I figured out what I really want is a weighted geometric mean, so I implemented that for myself. If you'd want to include that, that'll be cool. Actually I'm not sure if the goal of the statistics module is to be comprehensive or minimal. I'm hoping it's meant to be comprehensive. But then I'd guess there would be a lot of things you'd want to add except my little feature.

[cool-RR]

And of course, if the goal of the statistics module is to be comprehensive, one should ask himself what should be the difference between this new module and a mature statistics module like scipy.stats, and whether we should try to copy the features of off scipy.stats.

[mdickinson]

Choice of algorithm is a bit tricky here. There are a couple of obvious algorithms that work mathematically but result in significant accuracy loss in an IEEE 754 floating-point implementation: one is exp(mean(map(log, my_numbers))), where the log calls can introduce significant loss of information, and the other is prod(x**(1./len(my_numbers)) for x in my_numbers), where the **(1./n) operation similarly discards information. A better algorithm numerically is prod(my_numbers)**(1./len(my_numbers)), but that's likely to overflow quickly for large datasets (and/or datasets containing large values).

I'd suggest something along the lines of prod(my_numbers)**(1./len(my_numbers)), but keeping track of the exponent of the product separately and renormalizing where necessary to avoid overflow.

There are also algorithms for improved accuracy in a product, along the same lines as the algorithm used in fsum. See e.g., the paper "Accurate Floating-Point Product and Exponentiation" by Stef Graillat. [1] (I didn't know about this paper: I just saw a reference to it in a StackOverflow comment [2], which reminded me of this issue.)

[1] http://www-pequan.lip6.fr/~graillat/papers/IEEE-TC-prod.pdf
[2] http://stackoverflow.com/questions/37715250/safe-computation-of-geometric-mean

[mdickinson]

On the other hand, apparently exp(mean(log(...))) is good enough for SciPy: its current implementation looks like this:

def gmean(a, axis=0):
    a, axis = _chk_asarray(a, axis)
    log_a = ma.log(a)
    return ma.exp(log_a.mean(axis=axis))

[stevendaprano]

On Thu, Jun 09, 2016 at 09:24:04AM +0000, Mark Dickinson wrote:

On the other hand, apparently exp(mean(log(...))) is good enough for SciPy:

Hmm, well, I don't have SciPy installed, but I've found that despite
their (well-deserved) reputation, numpy (and presumably scipy) often
have rather naive algorithms that can lose accuracy rather
spectacularly.

py> statistics.mean([1e50, 2e-50, -1e50, 2e-50])
1e-50
py> np.mean(np.array([1e50, 2e-50, -1e50, 2e-50]))
5e-51

py> statistics.mean([1e50, 2e-50, -1e50, 2e-50]*1000)
1e-50
py> np.mean(np.array([1e50, 2e-50, -1e50, 2e-50]*1000))
5.0000000000000002e-54

On the other hand, np is probably a hundred times (or more) faster, so I
suppose accuracy/speed makes a good trade off.

[mdickinson]

Hmm, well, I don't have SciPy installed, but I've found that despite
their (well-deserved) reputation, numpy (and presumably scipy) often
have rather naive algorithms that can lose accuracy rather
spectacularly.

Agreed. And as Ram Rachum hinted, there seems little point aiming to duplicate things that already exist in the de facto standard scientific libraries. So I think there's a place for a non-naive carefully computed geometric mean in the std. lib. statistics module, but I wouldn't see the point of simply adding an exp-mean-log implementation (not that anyone is advocating that).

[stevendaprano]

Does anyone have any strong feeling about the name for these functions?

gmean and hmean;

geometric_mean and harmonic_mean

And "subcontrary_mean" is not an option :-)

[rhettinger]

I would like to see them spelled-out: geometric_mean and harmonic_mean

[mdickinson]

I would like to see them spelled-out: geometric_mean and harmonic_mean

+1

[python-dev]

New changeset 9eb5edfcf604 by Steven D'Aprano in branch 'default':
bpo-27181 add geometric mean.
https://hg.python.org/cpython/rev/9eb5edfcf604