New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multinomial LogisticRegressionCV with lbfgs non-deterministic on Travis Mac OS #11924

Open
jnothman opened this Issue Aug 27, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@jnothman
Member

jnothman commented Aug 27, 2018

@jnothman jnothman added the Bug label Aug 27, 2018

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 27, 2018

Member

These kinds of cross-platform inconsistencies often come from things like uninitialised memory... Not sure where to look.

Member

jnothman commented Aug 27, 2018

These kinds of cross-platform inconsistencies often come from things like uninitialised memory... Not sure where to look.

@jnothman jnothman changed the title from Multinomial logistic regression with lbfgs non-deterministic on Travis Mac OS to Multinomial LogisticRegressionCV with lbfgs non-deterministic on Travis Mac OS Aug 27, 2018

jnothman added a commit to jnothman/scikit-learn that referenced this issue Aug 28, 2018

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 28, 2018

Member

At #11925 I was trying to identify whether the discrepancy came from the initial fits for each fold, or from the refit process in LogisticRegressionCV (as this would make sense for why it's broken only for the CV variant).

From https://travis-ci.org/scikit-learn/scikit-learn/jobs/421403473#L375 we see that while the binary logistic fits produce an identical model each time, the mean coefficients of the best model from CV are slightly different across repeated runs (and I'm not sure we should be using the mean here):

(multi_class='auto')

[[ -0.59306404   3.09940484  -6.811572    -2.96272071  26.26385125]
 [ -1.55696142  -2.36361588   0.09320993  -5.3674105   29.95119552]
 [  2.15002545  -0.73578896   6.71836207   8.33013121 -56.21504677]]

vs
(multi_class='multinomial')

[[ -0.59604468   3.09992681  -6.8005941   -2.98726904  26.26175622]
 [ -1.56230187  -2.34716593   0.08463475  -5.34018104  29.92984087]
 [  2.15834654  -0.75276089   6.71595935   8.32745008 -56.19159708]]

Using these as the initial coefficients results in a vastly discrepant refit:

[[ -1.57069836   2.66750768  -9.14886852  -4.43393543  47.95429526]
 [ -2.90164875  -2.27389574  -1.65205449  -6.94935689  50.24802659]
 [  4.47234712  -0.39361193  10.80092301  11.38329232 -98.20232185]]

vs

[[-2.01577764e+00  2.68288189e+00 -9.15560217e+00 -4.01869064e+00
   5.02771291e+01]
 [-2.84247918e+00 -2.76571395e+00 -1.49917595e+00 -7.49195111e+00
   5.15594962e+01]
 [ 4.85825682e+00  8.28320544e-02  1.06547781e+01  1.15106417e+01
  -1.01836625e+02]]
Member

jnothman commented Aug 28, 2018

At #11925 I was trying to identify whether the discrepancy came from the initial fits for each fold, or from the refit process in LogisticRegressionCV (as this would make sense for why it's broken only for the CV variant).

From https://travis-ci.org/scikit-learn/scikit-learn/jobs/421403473#L375 we see that while the binary logistic fits produce an identical model each time, the mean coefficients of the best model from CV are slightly different across repeated runs (and I'm not sure we should be using the mean here):

(multi_class='auto')

[[ -0.59306404   3.09940484  -6.811572    -2.96272071  26.26385125]
 [ -1.55696142  -2.36361588   0.09320993  -5.3674105   29.95119552]
 [  2.15002545  -0.73578896   6.71836207   8.33013121 -56.21504677]]

vs
(multi_class='multinomial')

[[ -0.59604468   3.09992681  -6.8005941   -2.98726904  26.26175622]
 [ -1.56230187  -2.34716593   0.08463475  -5.34018104  29.92984087]
 [  2.15834654  -0.75276089   6.71595935   8.32745008 -56.19159708]]

Using these as the initial coefficients results in a vastly discrepant refit:

[[ -1.57069836   2.66750768  -9.14886852  -4.43393543  47.95429526]
 [ -2.90164875  -2.27389574  -1.65205449  -6.94935689  50.24802659]
 [  4.47234712  -0.39361193  10.80092301  11.38329232 -98.20232185]]

vs

[[-2.01577764e+00  2.68288189e+00 -9.15560217e+00 -4.01869064e+00
   5.02771291e+01]
 [-2.84247918e+00 -2.76571395e+00 -1.49917595e+00 -7.49195111e+00
   5.15594962e+01]
 [ 4.85825682e+00  8.28320544e-02  1.06547781e+01  1.15106417e+01
  -1.01836625e+02]]
@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 28, 2018

Member

I think this can't be just about LogisticRegressionCV. At https://travis-ci.org/scikit-learn/scikit-learn/jobs/421410785, for two of the three KFold splits, we get identical coefficients. For the third split, one fit gets:

 [[-2.15440499e+00  2.59582912e+00 -9.40201575e+00 -3.45705788e+00
    5.14770829e+01]
  [-2.72327380e+00 -2.58044861e+00 -1.36975377e+00 -7.73415753e+00
    5.00827769e+01]
  [ 4.87767879e+00 -1.53805063e-02  1.07717695e+01  1.11912154e+01
   -1.01559860e+02]]

The other gets:

 [[-1.24424826e+00  3.47553540e+00 -9.13287242e+00 -4.34631953e+00
    4.32699505e+01]
  [-3.14665603e+00 -2.61939440e+00 -1.22801012e+00 -7.34972009e+00
    5.14471399e+01]
  [ 4.39090429e+00 -8.56140993e-01  1.03608825e+01  1.16960396e+01
   -9.47170904e+01]]]

Which seems quite different.

Member

jnothman commented Aug 28, 2018

I think this can't be just about LogisticRegressionCV. At https://travis-ci.org/scikit-learn/scikit-learn/jobs/421410785, for two of the three KFold splits, we get identical coefficients. For the third split, one fit gets:

 [[-2.15440499e+00  2.59582912e+00 -9.40201575e+00 -3.45705788e+00
    5.14770829e+01]
  [-2.72327380e+00 -2.58044861e+00 -1.36975377e+00 -7.73415753e+00
    5.00827769e+01]
  [ 4.87767879e+00 -1.53805063e-02  1.07717695e+01  1.11912154e+01
   -1.01559860e+02]]

The other gets:

 [[-1.24424826e+00  3.47553540e+00 -9.13287242e+00 -4.34631953e+00
    4.32699505e+01]
  [-3.14665603e+00 -2.61939440e+00 -1.22801012e+00 -7.34972009e+00
    5.14471399e+01]
  [ 4.39090429e+00 -8.56140993e-01  1.03608825e+01  1.16960396e+01
   -9.47170904e+01]]]

Which seems quite different.

@jnothman

This comment has been minimized.

Show comment
Hide comment
@jnothman

jnothman Aug 28, 2018

Member

The fact that this occurs when multi_class='multinomial' and not with ovr or binary problems sent me to investigate the gradient function _multinomial_loss_grad, but I can't see an issue there.

I'm wondering if we should just skip the test on mac: I presume that this same bug was present, but not tested for, in 0.19, so it's not really release sensitive!!

Member

jnothman commented Aug 28, 2018

The fact that this occurs when multi_class='multinomial' and not with ovr or binary problems sent me to investigate the gradient function _multinomial_loss_grad, but I can't see an issue there.

I'm wondering if we should just skip the test on mac: I presume that this same bug was present, but not tested for, in 0.19, so it's not really release sensitive!!

@rth

This comment has been minimized.

Show comment
Hide comment
@rth

rth Aug 28, 2018

Member

Thanks for investigating this! +1 to skip that assert for now.

Member

rth commented Aug 28, 2018

Thanks for investigating this! +1 to skip that assert for now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment