BUG weightstats has wrong cov #370

Closed
josef-pkt opened this Issue Jul 12, 2012 · 2 comments

Comments

Projects
None yet
1 participant
Owner

josef-pkt commented Jul 12, 2012

The first time that I saw someone ask for it http://stackoverflow.com/questions/11442962/python-package-that-supports-weighted-covariance-computation

but cov in stats.weightstats doesn't do the weighting correctly, corrcoef is also wrong as consequence

test coverage is only for the basic properties, not for cov, corcoef

see if we can write tests compared to Stata, class was written with SPSS manual as reading

Owner

josef-pkt commented Jul 12, 2012

# -*- coding: utf-8 -*-
"""

Created on Thu Jul 12 16:57:14 2012

Author: Josef Perktold
"""

import numpy as np
from statsmodels.stats.weightstats import DescrStatsW


np.random.seed(987467)
x = np.random.multivariate_normal([0, 1.], [[1., 0.5], [0.5, 1]], size=20)
weights = np.random.randint(1, 4, size=20)

xlong = np.repeat(x, weights, axis=0)

ds = DescrStatsW(x, weights=weights)

print 'cov bug'
print ds.cov  #BUG

self = ds
ds_cov = np.dot(self.weights * self.demeaned.T, self.demeaned) / self.sum_weights

print '\nddof=0'
print ds_cov
print np.cov(xlong.T, bias=1)

ds_cov0 = np.dot(self.weights * self.demeaned.T, self.demeaned) / \
              (self.sum_weights - 1)
print '\nddof=0'
print ds_cov0
print np.cov(xlong.T, bias=0)

@josef-pkt josef-pkt added a commit to josef-pkt/statsmodels that referenced this issue Sep 11, 2012

@josef-pkt josef-pkt BUG: fix weighted cov and corrcoef, fix std_mean and ttest ddof!=0, a…
…dd tests, closes: #370
8cc249f

@josef-pkt josef-pkt added a commit to josef-pkt/statsmodels that referenced this issue Mar 13, 2013

@josef-pkt josef-pkt BUG: fix weighted cov and corrcoef, fix std_mean and ttest ddof!=0, a…
…dd tests, closes: #370
81003d2
Owner

josef-pkt commented Mar 24, 2013

thhe fix in the above commits was merged in #700 (rebased of #535 )

josef-pkt closed this Mar 24, 2013

@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014

@josef-pkt josef-pkt BUG: fix weighted cov and corrcoef, fix std_mean and ttest ddof!=0, a…
…dd tests, closes: #370
f1a71ce
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment