Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TST: stats: add kstwo, ksone to slow tests. #12140

Merged
merged 1 commit into from
May 18, 2020

Conversation

rgommers
Copy link
Member

This speeds up stats.test() by almost 50% (60 sec to 31 sec)
when running locally. The kstwo test is also the single slowest test
on CI, see gh-12132.

Output before this commit:

========================================================= slowest 40 test durations =========================================================
17.48s call     tests/test_continuous_basic.py::test_cont_basic[kstwo-arg59]
5.02s call     tests/test_stats.py::TestMGCStat::test_random_state
3.46s call     tests/test_stats.py::TestMGCStat::test_workers
3.17s call     tests/test_continuous_basic.py::test_cont_basic[ksone-arg58]
1.84s call     tests/test_stats.py::TestKSTwoSamples::testMediumBoth
1.71s call     tests/test_stats.py::TestKSTwoSamples::testMiddlingBoth
1.34s call     tests/test_continuous_basic.py::test_cont_basic[norminvgauss-arg78]
1.23s call     tests/test_mstats_basic.py::TestCorr::test_kendalltau
1.03s call     tests/test_stats.py::TestKSTwoSamples::test_some_code_paths
0.79s call     tests/test_continuous_basic.py::test_cont_basic[ncx2-arg76]
0.72s call     tests/test_continuous_basic.py::test_cont_basic[geninvgauss-arg35]
0.59s call     tests/test_continuous_basic.py::test_cont_basic[skewnorm-arg90]
0.50s call     tests/test_continuous_basic.py::test_cont_basic[burr-arg7]
0.47s call     tests/test_continuous_basic.py::test_cont_basic[exponweib-arg20]
0.42s call     tests/test_continuous_basic.py::test_cont_basic[tukeylambda-arg97]
0.36s call     tests/test_continuous_basic.py::test_cont_basic[pearson3-arg80]
0.33s call     tests/test_mstats_basic.py::TestCompareWithStats::test_kendalltau
0.33s call     tests/test_continuous_basic.py::test_cont_basic[johnsonsb-arg51]
0.31s call     tests/test_continuous_basic.py::test_cont_basic[powernorm-arg83]
0.29s call     tests/test_continuous_basic.py::test_cont_basic[exponpow-arg19]
0.29s call     tests/test_continuous_basic.py::test_cont_basic[crystalball-arg13]
0.27s call     tests/test_continuous_basic.py::test_cont_basic[beta-arg4]
0.27s call     tests/test_continuous_basic.py::test_cont_basic[triang-arg93]
0.26s call     tests/test_continuous_basic.py::test_cont_basic[exponnorm-arg18]
0.26s call     tests/test_continuous_basic.py::test_cont_basic[fatiguelife-arg22]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[burr12-arg8]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[gengamma-arg33]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[betaprime-arg5]
0.23s call     tests/test_continuous_basic.py::test_cont_basic[halfgennorm-arg38]
0.22s call     tests/test_continuous_basic.py::test_cont_basic[genhalflogistic-arg34]
0.22s call     tests/test_continuous_basic.py::test_cont_basic[f-arg21]
0.19s call     tests/test_stats.py::TestFOneWay::test_nist
0.19s call     tests/test_continuous_basic.py::test_cont_basic[chi-arg10]
0.19s call     tests/test_continuous_basic.py::test_cont_basic[invgauss-arg49]
0.19s call     tests/test_continuous_basic.py::test_cont_basic[nct-arg75]
0.18s call     tests/test_continuous_basic.py::test_cont_basic[genpareto-arg39]
0.18s call     tests/test_multivariate.py::TestMultivariateNormal::test_broadcasting
0.18s call     tests/test_distributions.py::TestLevyStable::test_pdf_alpha_equals_one_beta_non_zero
0.16s call     tests/test_distributions.py::TestSkewNorm::test_moments
0.16s call     tests/test_continuous_basic.py::test_cont_basic[gompertz-arg41]

I'm leaving TestMGCStat::test_workers as the only test that takes >1.5 sec, to not make test coverage for MGC too low.

This speeds up ``stats.test()`` by almost 50% (60 sec to 31 sec)
when running locally. The `kstwo` test is also the single slowest test
on CI, see scipygh-12132.

Output before this commit:
```
========================================================= slowest 40 test durations =========================================================
17.48s call     tests/test_continuous_basic.py::test_cont_basic[kstwo-arg59]
5.02s call     tests/test_stats.py::TestMGCStat::test_random_state
3.46s call     tests/test_stats.py::TestMGCStat::test_workers
3.17s call     tests/test_continuous_basic.py::test_cont_basic[ksone-arg58]
1.84s call     tests/test_stats.py::TestKSTwoSamples::testMediumBoth
1.71s call     tests/test_stats.py::TestKSTwoSamples::testMiddlingBoth
1.34s call     tests/test_continuous_basic.py::test_cont_basic[norminvgauss-arg78]
1.23s call     tests/test_mstats_basic.py::TestCorr::test_kendalltau
1.03s call     tests/test_stats.py::TestKSTwoSamples::test_some_code_paths
0.79s call     tests/test_continuous_basic.py::test_cont_basic[ncx2-arg76]
0.72s call     tests/test_continuous_basic.py::test_cont_basic[geninvgauss-arg35]
0.59s call     tests/test_continuous_basic.py::test_cont_basic[skewnorm-arg90]
0.50s call     tests/test_continuous_basic.py::test_cont_basic[burr-arg7]
0.47s call     tests/test_continuous_basic.py::test_cont_basic[exponweib-arg20]
0.42s call     tests/test_continuous_basic.py::test_cont_basic[tukeylambda-arg97]
0.36s call     tests/test_continuous_basic.py::test_cont_basic[pearson3-arg80]
0.33s call     tests/test_mstats_basic.py::TestCompareWithStats::test_kendalltau
0.33s call     tests/test_continuous_basic.py::test_cont_basic[johnsonsb-arg51]
0.31s call     tests/test_continuous_basic.py::test_cont_basic[powernorm-arg83]
0.29s call     tests/test_continuous_basic.py::test_cont_basic[exponpow-arg19]
0.29s call     tests/test_continuous_basic.py::test_cont_basic[crystalball-arg13]
0.27s call     tests/test_continuous_basic.py::test_cont_basic[beta-arg4]
0.27s call     tests/test_continuous_basic.py::test_cont_basic[triang-arg93]
0.26s call     tests/test_continuous_basic.py::test_cont_basic[exponnorm-arg18]
0.26s call     tests/test_continuous_basic.py::test_cont_basic[fatiguelife-arg22]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[burr12-arg8]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[gengamma-arg33]
0.25s call     tests/test_continuous_basic.py::test_cont_basic[betaprime-arg5]
0.23s call     tests/test_continuous_basic.py::test_cont_basic[halfgennorm-arg38]
0.22s call     tests/test_continuous_basic.py::test_cont_basic[genhalflogistic-arg34]
0.22s call     tests/test_continuous_basic.py::test_cont_basic[f-arg21]
0.19s call     tests/test_stats.py::TestFOneWay::test_nist
0.19s call     tests/test_continuous_basic.py::test_cont_basic[chi-arg10]
0.19s call     tests/test_continuous_basic.py::test_cont_basic[invgauss-arg49]
0.19s call     tests/test_continuous_basic.py::test_cont_basic[nct-arg75]
0.18s call     tests/test_continuous_basic.py::test_cont_basic[genpareto-arg39]
0.18s call     tests/test_multivariate.py::TestMultivariateNormal::test_broadcasting
0.18s call     tests/test_distributions.py::TestLevyStable::test_pdf_alpha_equals_one_beta_non_zero
0.16s call     tests/test_distributions.py::TestSkewNorm::test_moments
0.16s call     tests/test_continuous_basic.py::test_cont_basic[gompertz-arg41]
```
@rgommers rgommers added scipy.stats maintenance Items related to regular maintenance tasks labels May 17, 2020
@rgommers
Copy link
Member Author

@pvanmulbregt I think some of these performance regressions for kstwo were expected and necessary because of accuracy fixes. Can't remember though, would be good if you could have a look.

@pvanmulbregt
Copy link
Contributor

kstwo is a new distribution, for better accuracy than kstwobign.
The _cdf and sf implementations seem reasonable.
The _pdf() method is implemented as the derivative of the CDF, and the ppf uses optimize.brent to find the root.
The moments are calculated via integration.
The upshot is that the entropy and stats calls in the test_cont_basic test take quite some time to evaluate. Switching from momtype=0 to momtype=1 made very little difference.

Copy link
Member

@larsoner larsoner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like a useful gain in speed to me

@rgommers
Copy link
Member Author

Thanks @pvanmulbregt. I figured it was something like that. accuracy > speed, so let's leave it as is for now. Just wanted to make sure you saw I moved these to @slow.

Thanks for the approval @larsoner . Merging then.

@rgommers rgommers merged commit ebb98bc into scipy:master May 18, 2020
@rgommers rgommers deleted the stats-testspeed branch May 18, 2020 18:37
@rgommers rgommers added this to the 1.5.0 milestone May 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
maintenance Items related to regular maintenance tasks scipy.stats
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants