LogisticRegression convert to float64

[GaelVaroquaux]

Looking at the code of LogisticRegression, I have just noticed that it converts automatically the data to float64. I would expect at least the SAG, SAGA, newton-cg and lbfgs solvers to be able to work with float32.

Also, in a similar line of thoughts, the code converts to C-ordered data. How useful / necessary is this for solvers others than liblinear.

I am asking these questions because it seems that we could reduce the memory footprint of LogisticRegression.

Ping @arthurmensch for the SAG part.

[massich]

I can work on this

[arthurmensch]

Great! Handling float32 is not that hard: We need fused type : you can have a look at cd_fast.pyx. Handling F ordered arrays is possible but requires to change the way we access some arrays in place of the code where we use explicit pointers.

…

On Thu, Apr 20, 2017, 5:19 PM Joan Massich ***@***.***> wrote: I can work on this — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD48Y4m29WJ2a8SIOcVzrOsd3E0-AHAsks5rx3dkgaJpZM4NC9c0> .

[mblondel]

Maybe not for L-BFGS:
scipy/scipy#4873

[arthurmensch]

I will take care of introducing fused type in sag.pyx

…

On Wed, Jun 7, 2017, 10:01 AM Olivier Grisel ***@***.***> wrote: Reopened #8769 <#8769>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AD48Y14MUFRBlUQSZU6S1OX2Ig8ebpdkks5sBli5gaJpZM4NC9c0> .

[massich]

Just to keep track of what we the effort:

• randomized_ [MRG + 1] ENH Ensure PCA and randomized_svd_low_rank don't upcast float to double #9067 @raghavrv MRG
• Linear Models
  • Ridge [MRG+2] Ridge linear model dtype consistency (all solvers but sag) #9033 @massich @ncordier MRG
    • svd
    • sparse_cg
    • cholesky
    • lsqr
    • sg
  • Lasso
  • Multi-task Lasso
  • Elastic Net
  • Multi-task Elastic Net
  • Least Angle Regression
  • LARS Lasso
  • Orthogonal Matching Pursuit (OMP)
  • Bayesian Regression @Henley13
    • Bayesian Ridge Regression
  • Automatic Relevance Determination - ARD
  • Logistic regression
    • newton-cg PR [MRG+2] LogisticRegression convert to float64 (newton-cg) #8835 DONE
    • sag/saga PR [MRG] LogisticRegression convert to float64 (sag) #9020 WIP @Henley13 @arthurmensch Review (taken by @NelleV in [MRG] LogisticRegression convert to float64 (sag) #11155)
      • make_dataset PR [MRG] Fused type makedataset #9040 MRG @massich @raghavrv @Henley13 (taken by @NelleV in [MRG] LogisticRegression convert to float64 (sag) #11155) (retaken by @massich in LogisticRegression convert to float64 (for SAG solver) #13243)
    • lbfgs
  • Stochastic Gradient Descent - SGD @ncordier
• Isotonic Regression [MRG+1] fused types in isotonic_regression #9106 @vene

[massich]

@arthurmensch if you take care of fuse type in sag.pyx (see PR #9020) make sure to talk to @Henley13

[ogrisel]

saga and sag should indeed be addressed in the same PR.

[massich]

@TomDLT suggested to add type32 type 64 support for _preprocess_data here

[vene]

can we add isotonic regression to that list? I'll be happy to take it

[massich]

Sure :)

…

On Sun, Jun 11, 2017, 08:57 Vlad Niculae ***@***.***> wrote: can we add isotonic regression to that list? I'll be happy to take it — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#8769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGt-45MeYBb7lVbrJOzOs84Wkl7IUU5Vks5sC4_VgaJpZM4NC9c0> .

[massich]

@ogrisel suggested in #9033 to use [np.float64, np.float32] as the default policy. As here for logistic regression:

if self.solver == 'lbfgs':
    # scipy lbfgs does not support float32 yet:
    # https://github.com/scipy/scipy/issues/4873
    _dtype = np.float64
else:
    # all other solvers work at both float precision levels
    _dtype = [np.float64, np.float32]

As well as adding a test to ensure that type of the predicted output remains consistent. i.e:

    assert_equal(clf_32.predict(X_32).dtype, X_32.dtype)
    assert_equal(clf_64.predict(X_64).dtype, X_64.dtype)

cc: @Henley13, @raghavrv, @ncordier, @vene

[raghavrv]

@ogrisel advised me that

assert clf_32.predict(X_32).dtype == X_32.dtype
...

is better as it leads to more informative error messages in pytests to which we will be switching to soon.

[jnothman]

at which point i assume we'll change all assert_equal?

…

On 15 Jun 2017 1:59 am, "(Venkat) Raghav, Rajagopalan" < ***@***.***> wrote: @ogrisel <https://github.com/ogrisel> advised me that assert clf_32.predict(X_32).dtype == X_32.dtype... is better as it leads to more informative error messages in pytests to which we will be switching to soon. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#8769 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEz65b9bp9Vx25RKzR-B0A_td80OCfjks5sEANcgaJpZM4NC9c0> .

[GaelVaroquaux]

Fixed by #13273