FIX properly report n_iter_ in case of fallback from Newton-Cholesky to LBFGS

[ogrisel]

Fix for a small bug discovered while reviewing #28840: whenever the Newton-Cholesky solver of LogisticRegression would fall back to LFGS, the reported n_iter_ attribute would be left to zero.

I made it such that any completed iteration from the Newton-Cholesky solver would be subtracted from max_iter before calling LBFGS, and then report the sum of the numbers of iterations completed by the two solvers in the end. In practice, this does not seem to change anything because when Hessian conditioning problem always happens during the first iteration in my experiments with low regularized, rank deficient problems that typically trigger the LBFGS fallback mechanism.

Note that I find that the LBFGS fallback warning quite annoying whenever it is triggered while tuning the regularization level (e.g. using LogisticRegressionCV or RandomizedSearchCV). I have the feeling that this should be a regular verbose print instead, but we can tackle that in a separate PR.

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: e0ec69a. Link to the linter CI: here}

[lorentzenchr]

Thanks for spotting and fixing this bug.
I also would like to see improvements for the fallback case.

[ogrisel]

The global_random_seed parametrized test pass for all seeds locally.

Unfortunately this is not the case on our macOS pylatest_conda_mkl_no_openmp runner, even after having decreased the regularization and increased the data width.

There is a failure with global_random_seed=6 on that host.

        # Trying to fit the same model again with a small iteration budget should
        # therefore raise a ConvergenceWarning:
        lr_nc_limited = LogisticRegression(
            solver="newton-cholesky", C=C, max_iter=n_iter_lbgs - 1
        )
>       with pytest.warns(LinAlgWarning, match="ill-conditioned Hessian matrix"):
E       Failed: DID NOT WARN. No warnings of type (<class 'scipy.linalg._misc.LinAlgWarning'>,) were emitted.
E       The list of emitted warnings is: []

I pushed a new commit with the [all random seeds] commit message for that test to see there is only a problem with this CI environment or if the test is also brittle on other environment.

[ogrisel]

Ok, so the warning is never raised for macOS pylatest_conda_mkl_no_openmp whatever the choice of the random seed, while it is always raised (as expected) for all the seeds in all the other testing environments. So the problem is not related to the statistical properties of the test problem.

[ogrisel]

I tried to look into the scipy version / BLAS version and so, but I could not find anything that would explain why the warning was not raised only in macOS pylatest_conda_mkl_no_openmp while it works for other environments (with or without MKL, with older and newer versions of scipy and so on).

[ogrisel]

Ok, so it seems to be a bug in the propagation of LinAlgWarning but the iteration logic is still somehow respected...

I will push another commit to clean-up the commented out code and keep the ignore_warnings thing for now.