Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multinomial LogisticRegressionCV with lbfgs non-deterministic on Travis Mac OS #11924

Open
jnothman opened this issue Aug 27, 2018 · 8 comments
Open

Comments

@jnothman
Copy link
Member

@jnothman jnothman commented Aug 27, 2018

See MacPython/scikit-learn-wheels#7 (comment)

@jnothman jnothman added the Bug label Aug 27, 2018
@jnothman
Copy link
Member Author

@jnothman jnothman commented Aug 27, 2018

These kinds of cross-platform inconsistencies often come from things like uninitialised memory... Not sure where to look.

@jnothman jnothman changed the title Multinomial logistic regression with lbfgs non-deterministic on Travis Mac OS Multinomial LogisticRegressionCV with lbfgs non-deterministic on Travis Mac OS Aug 27, 2018
jnothman added a commit to jnothman/scikit-learn that referenced this issue Aug 28, 2018
@jnothman
Copy link
Member Author

@jnothman jnothman commented Aug 28, 2018

At #11925 I was trying to identify whether the discrepancy came from the initial fits for each fold, or from the refit process in LogisticRegressionCV (as this would make sense for why it's broken only for the CV variant).

From https://travis-ci.org/scikit-learn/scikit-learn/jobs/421403473#L375 we see that while the binary logistic fits produce an identical model each time, the mean coefficients of the best model from CV are slightly different across repeated runs (and I'm not sure we should be using the mean here):

(multi_class='auto')

[[ -0.59306404   3.09940484  -6.811572    -2.96272071  26.26385125]
 [ -1.55696142  -2.36361588   0.09320993  -5.3674105   29.95119552]
 [  2.15002545  -0.73578896   6.71836207   8.33013121 -56.21504677]]

vs
(multi_class='multinomial')

[[ -0.59604468   3.09992681  -6.8005941   -2.98726904  26.26175622]
 [ -1.56230187  -2.34716593   0.08463475  -5.34018104  29.92984087]
 [  2.15834654  -0.75276089   6.71595935   8.32745008 -56.19159708]]

Using these as the initial coefficients results in a vastly discrepant refit:

[[ -1.57069836   2.66750768  -9.14886852  -4.43393543  47.95429526]
 [ -2.90164875  -2.27389574  -1.65205449  -6.94935689  50.24802659]
 [  4.47234712  -0.39361193  10.80092301  11.38329232 -98.20232185]]

vs

[[-2.01577764e+00  2.68288189e+00 -9.15560217e+00 -4.01869064e+00
   5.02771291e+01]
 [-2.84247918e+00 -2.76571395e+00 -1.49917595e+00 -7.49195111e+00
   5.15594962e+01]
 [ 4.85825682e+00  8.28320544e-02  1.06547781e+01  1.15106417e+01
  -1.01836625e+02]]
@jnothman
Copy link
Member Author

@jnothman jnothman commented Aug 28, 2018

I think this can't be just about LogisticRegressionCV. At https://travis-ci.org/scikit-learn/scikit-learn/jobs/421410785, for two of the three KFold splits, we get identical coefficients. For the third split, one fit gets:

 [[-2.15440499e+00  2.59582912e+00 -9.40201575e+00 -3.45705788e+00
    5.14770829e+01]
  [-2.72327380e+00 -2.58044861e+00 -1.36975377e+00 -7.73415753e+00
    5.00827769e+01]
  [ 4.87767879e+00 -1.53805063e-02  1.07717695e+01  1.11912154e+01
   -1.01559860e+02]]

The other gets:

 [[-1.24424826e+00  3.47553540e+00 -9.13287242e+00 -4.34631953e+00
    4.32699505e+01]
  [-3.14665603e+00 -2.61939440e+00 -1.22801012e+00 -7.34972009e+00
    5.14471399e+01]
  [ 4.39090429e+00 -8.56140993e-01  1.03608825e+01  1.16960396e+01
   -9.47170904e+01]]]

Which seems quite different.

@jnothman
Copy link
Member Author

@jnothman jnothman commented Aug 28, 2018

The fact that this occurs when multi_class='multinomial' and not with ovr or binary problems sent me to investigate the gradient function _multinomial_loss_grad, but I can't see an issue there.

I'm wondering if we should just skip the test on mac: I presume that this same bug was present, but not tested for, in 0.19, so it's not really release sensitive!!

@rth
Copy link
Member

@rth rth commented Aug 28, 2018

Thanks for investigating this! +1 to skip that assert for now.

@ksslng
Copy link
Contributor

@ksslng ksslng commented Jan 28, 2020

I have the same problem on MacOS, that the test of LogisticRegressionCV with newton-cg fail nearly all the time and rarely the test LogisticRegression with newton-cg fails as well.

Not equal to tolerance rtol=1e-07, atol=0
Mismatched elements: 6 / 12 (50%)
Max absolute difference: 9.82197427e-08
Max relative difference: 4.4415004e-07
@jnothman
Copy link
Member Author

@jnothman jnothman commented Jan 28, 2020

@ksslng
Copy link
Contributor

@ksslng ksslng commented Jan 29, 2020

Sorry, the solver lbfgs doesn't actually fail, so I opened a new Issue #16279.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
3 participants
You can’t perform that action at this time.