-
-
Notifications
You must be signed in to change notification settings - Fork 25.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[MRG] Libsvm and liblinear rand() fix for convergence on windows targets (and improvement on all targets) #13511
[MRG] Libsvm and liblinear rand() fix for convergence on windows targets (and improvement on all targets) #13511
Conversation
Continuous integration reports failures concerning SVM scores: Windows py35_32 and Windows py37_64 ______________ test_estimators[LinearSVR-check_regressors_train] ______________
estimator_checks.py:1870: in check_regressors_train
'0.48870169385239753 not greater than 0.5'
___ test_estimators[LinearSVR-check_regressors_train(readonly_memmap=True)] ___
estimator_checks.py:1870: in check_regressors_train
'0.48870169385239753 not greater than 0.5' It is a bit strange, because on my Windows PC I am not able to reproduce these issues... I tried to restart all CI jobs by pushing again - I obtain exactly the same result. Could anyone try on his windows machine to see if they can reproduce the CI bug ? I have to admit that I am clueless here. |
…hecks for random generation fix are now static.
…ter random number generator as suggested by https://codeforces.com/blog/entry/61587
…++11 support. liblinear extension.
…ig.add_extension` so now pre-creating liblinear library first with the flag.
Following @jnothman review, this is the summary of where this PR is:
Help from a core team member with knowledge of the libsvm / liblinear estimators would therefore be needed at this stage. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The failures certainly look substantive.
The only place I can see they might come from is that you no longer seed the object with each new fit. Replace srand(
with mt_rand.seed(
in
srand(seed); |
scikit-learn/sklearn/svm/src/libsvm/svm.cpp
Line 2351 in 1e052e9
srand(param->random_seed); |
scikit-learn/sklearn/svm/src/libsvm/svm.cpp
Line 2631 in 1e052e9
srand(param->random_seed); |
Isn't there a method to re-seed?
http://www.cplusplus.com/reference/random/mersenne_twister_engine/seed/
|
There is, but this is a method instance that works on a given mersenne twister engine object. Since a new object is created when the library is loaded, I'll see if I can access it from the files you mention. If not, I'll have to pass this seed alon in the parameters struct. |
… interface function `set_seed`.
It seems all good now. I'll just try to restart CI to get rid of the random download failures of circleci |
Quick link to my last comment, for those ending up here: #13511 (comment) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @smarie
Small question but LGTM
assert_almost_equal(_average_precision(y_true, probas_pred), | ||
precision_recall_auc, decimal=3) | ||
precision_recall_auc, decimal=2) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
is this related?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I remember well, it is. Changing the random number generation seems to make this test fail. When I investigated a few months ago, I could not find a good reason except for precision of the _average_precision
in case of 0.5 ties.
I would certainly appreciate a second pass of checking on that one, to confirm my impression.
It sounds like we could do with some benchmarking before (or even after)
this gets merged.
|
In addition to the benchmark suggestion, I suggest to create a python wrapper for the new |
@scikit-learn/core-devs this is a quite old PR thrice-approved, tagged in 0.23 milestone. Apart from syncing to upstream is there anything else to do but merge? |
We talked about it in one of the meetings and I believe it can be merged as is. @smarie would you mind fixing the merge conflicts? |
…into libsvm_liblinear_rand_fix � Conflicts: � doc/whats_new/v0.23.rst
…/scikit-learn into libsvm_liblinear_rand_fix
Done. I updated the headers, as we are now in 2020 :) I also created a header for the |
There is a doctest failure that seems unrelated:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @smarie , happy to merge if there are no objections.
thanks for the review! we'll have to create a new dedicated issue to reference the idea of creating SVM benchmarks. I have made significant progress in the direction of creating reproducible benchmarks thanks to pytest plugins ( |
Thanks for your persistence @smarie . Could you please open a new issue for the remaining stuff? |
…s (and improvement on all targets) (scikit-learn#13511) * Fixed random number generation on windows. See cjlin1/liblinear#28 * Added correct PR number * Fixed random number generator for libsvm on windows targets * Updated comments * Suppressed C4293 warnings using explicit cast. * Suppressed C4293 warnings using explicit cast. * Following code review: `myrand` is now inline, and all integer size checks for random generation fix are now static. * Dummy comment edit to re-trigger build * Fixed include error * Attempt to provide a better and faster result by using a mersene twister random number generator as suggested by https://codeforces.com/blog/entry/61587 * Attempt to integrate the extra compiler arg requesting for explicit C++11 support * Attempt to integrate the extra compiler arg requesting for explicit C++11 support. liblinear extension. * the `extra_compile_args` for C++11 was not working when used in `config.add_extension` so now pre-creating liblinear library first with the flag. * Fixed liblinear extension build: there was a missing library dependency. * Fixed liblinear library dependency for extension compilation. * Fixed liblinear library compilation: added tron * Now correctly setting the seed in libsvm * Now correctly setting the seed in liblinear. This is done using a new interface function `set_seed`. * Updated what's new accordingly * Increased tolerance when comparing `_average_precision` with `_average_precision_slow`. Indeed `_average_precision` is based on predictions ordering and has errors in case of ties (e.g. several 0.5 prediction values) * Minor improvement of this test method for readability (no change in output) * Minor doc update as per review * dropped commented line * Reverted test readability improvement change * Using double barticks * Clarified all comments * removed useless space * Clarified all comments, and included a `set_seed` method as per code review. * Updated what's new about changed models as suggested by review. * Replaced random number generator: now using a `bounded_rand_int` using tweaked Lemire post-processor from http://www.pcg-random.org/posts/bounded-rands.html) * Updated readme rst and moved it to 0.22 * Whats new moved from 0.222 to 0.23 * Updated docs * Update doc/whats_new/v0.23.rst Co-Authored-By: Chiara Marmo <cmarmo@users.noreply.github.com> * Dummy change to re-trigger the CI build * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.21.rst * Update doc/whats_new/v0.22.rst * As per code review: added LogisticRegression in the list of changes, and repeated the list of classes affected in changelog. * New random number generator used in libsvm / liblinear is now in an independent header file `newrand/newrand.h`. * Update sklearn/svm/src/liblinear/linear.cpp * Fixed dates in headers and added a header to newrand.h * Added a sentence in the changelog to explain the impact in user-friendly terms. * Added link to PR in readme file and removed commented lines as per code review Co-authored-by: Sylvain MARIE <sylvain.marie@se.com> Co-authored-by: Chiara Marmo <cmarmo@users.noreply.github.com>
This PR fixes part of #11536 on windows targets.
On windows platforms, liblinear and libsvm have strong convergence issues because of the way random numbers are generated: max random number in Windows is 15 bits (even on 64 bit windows), which is 32767, while max random number in linux+GCC is 31 bits (resp. 63 bits in 64 bits systems I guess) so that's 2147483647 (resp 9223372036854775807).
If I understand correctly, these random numbers are used in the coordinate gradient descent algorithms, to find the next coordinate to act upon. When the dimensionality (e.g. number of samples) is large, the random number generator on windows has a hard time to explore all dimensions.
This is a known bug documented in liblinear FAQ (strangely enough, not the libsvm FAQ) but the proposed workaround was wrong.
I made a patch for this years ago, that was approved by several users yet never merged: cjlin1/liblinear#28 .
Note that another user reported it on libsvm, I proposed a similar PR there: cjlin1/libsvm#140.
Since I realized that sklearn has it own internal copy of both libraries, this is the fix.