TestProbitCG failure on Ubuntu #109

Closed
mikix opened this Issue Nov 15, 2011 · 12 comments

Projects

None yet

4 participants

@mikix
mikix commented Nov 15, 2011

On Ubuntu Precise (haven't tested on stable version), test_discrete.py fails, causing a build failure.

https://launchpadlibrarian.net/83193398/buildlog_ubuntu-precise-i386.statsmodels_0.3.1-2_FAILEDTOBUILD.txt.gz

Simply bumping TestProbitCG's maxiter value from 250 to 500 fixes it. 500 matches the same iters that TestProbitNM is afforded, so it doesn't seem unreasonable.

@jseabold jseabold closed this in 590bd62 Nov 15, 2011
@richbwood

Hi,

I am using the latest build as of 19/12/2012:
statsmodels-c9062e43b8a5f7385537ca95f875c4f19e7ac6dd.zip

TestProbitCG is failing again, even though I can confirm that maxiter is set to 500.

FAIL: statsmodels.discrete.tests.test_discrete.TestProbitCG.test_conf_int

Traceback (most recent call last):
File "/home/woodri/build/out/lib/python2.7/site-packages/nose/case.py", line 187, in runTest
self.test(*self.arg)
File "/home/woodri/build/out/lib/python2.7/site-packages/statsmodels-0.5.0-py2.7-linux-x86_64.egg/statsmodels/discrete/tests/test_discrete.py", line 46, in test_conf_int
assert_almost_equal(self.res1.conf_int(), self.res2.conf_int, DECIMAL_4)
File "/home/woodri/build/out/lib/python2.7/site-packages/numpy/testing/utils.py", line 452, in assert_almost_equal
return assert_array_almost_equal(actual, desired, decimal, err_msg)
File "/home/woodri/build/out/lib/python2.7/site-packages/numpy/testing/utils.py", line 800, in assert_array_almost_equal
header=('Arrays are not almost equal to %d decimals' % decimal))
File "/home/woodri/build/out/lib/python2.7/site-packages/numpy/testing/utils.py", line 636, in assert_array_compare
raise AssertionError(msg)
AssertionError:
Arrays are not almost equal to 4 decimals

(mismatch 12.5%)
x: array([[ 0.26584602, 2.98583433],
[ -0.11269245, 0.21615338],
[ 0.26008598, 2.5926075 ],
[-12.43566596, -2.46925458]])
y: array([[ 0.2658255, 2.985795 ],
[ -0.1126929, 0.2161508],
[ 0.2600795, 2.592585 ],
[-12.43547 , -2.469166 ]])


Thank you.

@josef-pkt
Member

which scipy and numpy versions are you using?

Since this tests is skipped on Windows, maybe there are more serious problems with fmin_cg and we should rethink what to do with this test.
One change that might have affected this is that we are now renormalizing the likelihood function by 1/nobs for the optimization.

@josef-pkt
Member

can you try to lower gtol to gtol=1e-8 and run the tests?

@josef-pkt
Member

It looks like there is still a version issue.
This test doesn't fail on Travis-CI nor in the pythonxy Ubuntu tests https://code.launchpad.net/~pythonxy/+recipe/statsmodels-daily-current

@josef-pkt josef-pkt reopened this Dec 19, 2012
@richbwood

uname -a
Linux xldn5103pap 2.6.18-194.11.4.el5 #1 SMP Fri Sep 17 04:57:05 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux

numpy.version
'1.6.2'

scipy.version
'0.11.0'

Changing gtol=1e-8 fixed the tests:

Ran 1857 tests in 244.987s

OK (SKIP=12)

@josef-pkt
Member

Good, lowering gtol compensates for the normalization, and requires roughly the same precision as before.

Thank you.

@josef-pkt
Member

correction: that builds statsmodels_0.4.2-1
There shouldn't be a failure with that. ?

@jseabold
Member

Would this fail in master? Should we focus on getting a release out? Features / clean up / PR review is a neverending job...

@josef-pkt
Member

Yes, it's still on current master, just lowering the gtol should be enough.

I wasn't set up for changing master at the time and had forgotten about it.

I can start in 2 weeks with release preparation, then I have hopefully larger time blocks available.
I want to go through the errors in
#613
I saw several bugs in the code (in untested code paths) where I would also like to check why they are not tested.

@josef-pkt josef-pkt referenced this issue Feb 3, 2013
Merged

Misc fixes 05 #640

@josef-pkt josef-pkt added a commit to josef-pkt/statsmodels that referenced this issue Feb 20, 2013
@josef-pkt josef-pkt TST: TestProbitCG: increase optimization precision, closes #109 44a93ea
@josef-pkt josef-pkt closed this in 44a93ea Feb 20, 2013
@josef-pkt
Member

test failure on Windows 64bit MingW binaries
http://mail.scipy.org/pipermail/numpy-discussion/2014-April/070043.html
maxiter is too low, needs 766 iterations instead of less than maxiter=500, but then the result has atol 1e-6 instead of 4 decimals as in test suite.

looks like fmin_cg needs larger maxiter

@josef-pkt josef-pkt reopened this Apr 26, 2014
@josef-pkt
Member

see new issue #1690
fixed again in PR #1699 and test failures on some machines in PR #1766

Unit tests are just working around the problems in scipy's fmin_cg.

Problems in using fmin_cg in "not nice" cases still persist. Just make the estimation problem "nice".

@josef-pkt josef-pkt closed this Jun 18, 2014
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@jseabold jseabold TST: Bump maxiter in TestProbitCG. Closes #109. 1db133c
@PierreBdR PierreBdR pushed a commit to PierreBdR/statsmodels that referenced this issue Sep 2, 2014
@josef-pkt josef-pkt TST: TestProbitCG: increase optimization precision, closes #109 b4a8059
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment