FIX LogisticRegression warm start with newton-cholesky solver

[lorentzenchr]

Reference Issues/PRs

Fixes ##31859.

What does this implement/fix? Explain your changes.

LogisticRegression(warm_start=True, solver="newton-cholesky") for multi-class problems has a bug in that the already provided (warm started) coefficients are partially overwritten by zero instead of "unsymmetrized".

Any other comments?

A corresponding test is added.

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 904de3e. Link to the linter CI: here}

[lorentzenchr]

@david-cortes If you could review this one that would be great!

[ogrisel]

This PR needs to be synced with main to get the required Linux pymin_conda_forge_arm green.

[david-cortes]

Thanks for looking into it. Can confirm that the earlier problem with the intercepts is now solved.

However it seems to be applying the warm-start logic in cases where it shouldn't - not sure if connected to the changes here though:

import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
X, y = load_iris(return_X_y=True)
y_bin = (y == 0).astype(np.float64)

model1 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1
).fit(X, y_bin)
model2 = LogisticRegression(
    solver="newton-cholesky",
    max_iter=1,
    warm_start=True
).fit(X, y).fit(X, y_bin)

# Coefficients are not equal
np.testing.assert_almost_equal(
    model1.coef_,
    model2.coef_
)

[lorentzenchr]

@david-cortes The assert failure of your snippet goes away if you set max_iter=2 in model1 and if you always use y_bin instead of y (model2.fit).

[OmarManzoor]

LGTM. Thank you @lorentzenchr

[david-cortes]

@david-cortes The assert failure of your snippet goes away if you set max_iter=2 in model1 and if you always use y_bin instead of y (model2.fit).

@lorentzenchr Yes, but I don't think it should warm-start with coefficients that do not correspond to the same type of model being fitted.