weighted mean

[bgrayburn]

A "weights" labeled parameter in the df.mean method would be extremely useful. In numpy this functionality is provided via np.average instead of np.mean which I'm assuming is how similar functionality would be added to pandas.

ex requested feature:

> a = np.array([[2,4,6],[8,10,12]])
> w = [.5, 0, .5]
> np_avg = np.average(a, weights = w, axis=1)
#output ->  array([ 4., 10.])
> pd_avg = pandas.DataFrame(a).mean(weights = w, axis=1)
#desired output -> series with entries 4 and 10

If this is a desired feature I'll complete this and submit a pull request if desired. If this is somewhere already I've overlooked, my apologies, I've tried to look thoroughly.

[shoyer]

Why not just write:

pd_avg = (np.array(w) * pandas.DataFrame(a)).mean(axis=1)

[benjello]

I agree with @bgrayburn, weighted statistics would be very useful in pandas. One can use statsmodel but extending DataFrame methods to use weight would be very useful for people using weighted survey data.

[bgrayburn]

@stephan : I agree your code snippet accomplishes (nearly) the same thing
from a functional perspective, but from a code-readability standpoint, and
a code reuse standpoint, including a weights parameter seems optimal. Also
numpy's weight parameter automatically normalizes the weights vector which
is also extremely useful.

On Sat, May 2, 2015 at 3:04 PM, Mahdi Ben Jelloul notifications@github.com
wrote:

I agree with @bgrayburn https://github.com/bgrayburn, weighted
statistics would be very useful in pandas. One can use statsmodel but
extending DataFrame methods to use weight would be very useful for people
using weighted survey data.

—
Reply to this email directly or view it on GitHub
#10030 (comment).

[shoyer]

Okay, fair enough. This seems within scope for pandas. We recently added a sample method which includes a similar weights argument (#9666) which might be useful as a starting point.

[benjello]

@shoyer asked elsewhere (#10000) about a list of methods that could be enhanced by a 'weighted' version.
Almost all the statistical functions that can be found here are candidates. I can also think of describe, value_counts, qcut, hist and margin computations in pivot tables.

[shoyer]

Again, I think we're open to most of these changes (all of which are backward compatible with weights=None). The main obstacle is that we need implementations, documentation and benchmarks to show we aren't slowing anything down. PRs would be welcome.although, It would also be worth checking if any of these could be pushed upstream to numpy.

[bgrayburn]

@shoyer @benjello sorry for the delay on this, still planning on submitting a PR, I'll be coding this weekend.

in regards to the numpy thing, for weighted means they use .average which you can see here. My plan was to implement

pd_avg = (np.array(w) * pandas.DataFrame(a)).mean(axis=1)

pretty much as written, by multiplying the input dataframe's columns by the weight vector. Alternatively we could call np.average when a weights parameter is present, OR (3rd option) we could implement a pandas.DataFrame(a).average(weights=[...]) to mirror pandas.

one last question, should weighting be applicable in either axis=0 or axis=1 mode? I'm assuming yes, but wanted to check.

Let me know your preferences, or if somehow this should be incorporated with a larger change as mentioned above.

Best

[jreback]

why are you not just adding a weights keyword to .mean?

much more consistent in the API and we don't implement average I suspect because it's just confusing