Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ENH: moments for the zipf distribution #2904

Closed
wants to merge 2 commits into from
Closed

Conversation

ev-br
Copy link
Member

@ev-br ev-br commented Sep 20, 2013

Should close gh-1886.
A remaining issue is that expect reports the lack of convergence for small enough values of a. Which is a known issue (and the knownfail I'm leaving in this PR can serve as a useful smoke test if/when discrete expect is improved).

Also fix the 4th moment.
@josef-pkt
Copy link
Member

Can you copy the code for the failing expect kurtosis test in an issue? It's faster to replicate than looking at the changeset or test code.

zipf.expect(...
zipf.stats(...

It's possible that the expect in expect stops to early in the tail in this case, (x - mean)**4 can get pretty large.
(but I still find it a bit surprising)

@josef-pkt
Copy link
Member

expect is hitting maxcount which seems to be too low in this case, and it's not an option currently

@ev-br
Copy link
Member Author

ev-br commented Sep 23, 2013

Here it is:

>>> m, v, s, k = stats.zipf.stats(6.5, moments='mvsk')
(array(1.0130420979439105), array(0.015940729460401704), array(12.561822558088409), array(279.4461024600989))
>>> stats.zipf.expect(lambda x: np.power(x-m, 5), args=(6.5,))
/home/br/virtualenvs/scipy-dev/local/lib/python2.7/site-packages/scipy/stats/distributions.py:7065: RuntimeWarning: expect(): sum did not converge
  warnings.warn('expect(): sum did not converge', RuntimeWarning)
0.666271192853372
>>> (k+3)*v*v
0.071771491009249733

Sure, the power is low, the sum does not converge all that well.

@josef-pkt
Copy link
Member

you did power 5 not 4

using still scipy 0.9

>>> m, v = stats.zipf.stats(6.5)
>>> m, v
(array(1.0130420979439105), array(0.015940729460401704))
>>> m3 = stats.zipf.expect(func= lambda x: (x-m)**4, args=(6.5,))
sum did not converge
>>> m3
0.071753510989155736
>>> ke = m3 / v**2 - 3
>>> ke
279.37534474660725
>>> ke / 279.4461024600989 - 1   # relative error
-0.00025320701512299149

@ev-br
Copy link
Member Author

ev-br commented Sep 23, 2013

Argh.what was it about learning to copy-paste if I can't do the numbers. It's 4 in the test (common_tests.py)

@josef-pkt
Copy link
Member

Ok expect bug, needs better or optional termination condition. `maxcount`` is mainly a safeguard and I picked the number under the assumption that it won't be triggered under regular circumstances.

@rgommers
Copy link
Member

Rebased and fixed a couple of minor PEP8/style issues at https://github.com/rgommers/scipy/compare/pr;2904. This fixes a test failure I've been seeing for weeks, so can we merge this?

@ev-br
Copy link
Member Author

ev-br commented Oct 12, 2013

IMO, it's ready to go in. The issue with the 4th order moment is an expect problem: the convergence rate of the sum is slow. Maybe needs a Levine u-transform --- but that's a separate story.

@ev-br
Copy link
Member Author

ev-br commented Oct 12, 2013

Also, it seems to me that gh-1329 can now be closed --- I'm not claiming there are no problems left :-), but at least all problems smoked out in there are either fixed or marked as a known failure (of which there's only a handful left.)

@josef-pkt
Copy link
Member

looks fine to me

@rgommers
Copy link
Member

OK merged in 9dd997c. Thanks Evgeni, Josef.

@rgommers rgommers closed this Oct 12, 2013
thouis pushed a commit to thouis/scipy that referenced this pull request Oct 30, 2013
@ev-br ev-br deleted the zipf branch November 11, 2013 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

stats.zipf floating point warnings (Trac #1361)
3 participants