ENH precompute residual and column norm of X in coordinate descent

[lorentzenchr]

Reference Issues/PRs

None.
This PR needed #34514 as preparation.

What does this implement/fix? Explain your changes.

This PR adds precomputing of

• residuals: R = y - X @ coef
• squared column norms of X: norm2_cols_X = np.sum(X ** 2, axis=0)

in the coordinate descent solvers (Cython). The main impact is when computing paths with enet_path or ElasticNetCV because then, except for the very first path value, the initialization (memory) and computation of residuals and column norms is avoided.

Note the at the end of a coordinate descent algo, residuals are always up to date.

AI usage disclosure

None

Any other comments?

No public API change.

[lorentzenchr]

Small benchmark on leukemia dataset

n_alphas = 100

Branch main

┌───────────┬──────────┬───────────┐
│ tolerance ┆ time     ┆ version   │
│ ---       ┆ ---      ┆ ---       │
│ f64       ┆ f64      ┆ str       │
╞═══════════╪══════════╪═══════════╡
│ 0.01      ┆ 0.39445  ┆ 1.10.dev0 │
│ 0.001     ┆ 0.718646 ┆ 1.10.dev0 │
│ 0.0001    ┆ 1.713565 ┆ 1.10.dev0 │
└───────────┴──────────┴───────────┘

Branch PR

┌───────────┬──────────┬───────────┐
│ tolerance ┆ time     ┆ version   │
│ ---       ┆ ---      ┆ ---       │
│ f64       ┆ f64      ┆ str       │
╞═══════════╪══════════╪═══════════╡
│ 0.01      ┆ 0.18151  ┆ 1.10.dev0 │
│ 0.001     ┆ 0.450085 ┆ 1.10.dev0 │
│ 0.0001    ┆ 1.509361 ┆ 1.10.dev0 │
└───────────┴──────────┴───────────┘

This corresponds to a -54%, -37% and -12% fit time reduction.

Code

Details

import time
import numpy as np
import polars as pl
import seaborn as sns
import sklearn
from sklearn.datasets import fetch_openml
from sklearn.linear_model import Lasso, LassoCV
from sklearn.preprocessing import StandardScaler, LabelEncoder

X, y = fetch_openml("leukemia", version=1, return_X_y=True)
n_samples, n_features = X.shape

# Because we apply a penalty, we standarize X (zero mean and unit variance).
X = StandardScaler().fit_transform(X)
X = np.asfortranarray(X)  # Lasso prefers Fortran-contiguous 
y = LabelEncoder().fit_transform(y)
print(f"{n_samples=} {n_features=}")

n_alphas = 100
tols = [1e-2, 1e-3, 1e-4]

timing = np.zeros(len(tols))
for tol_ix, tol in enumerate(tols):
    t0 = time.time()
    m = LassoCV(alphas=n_alphas, tol=tol).fit(X, y)
    timing[tol_ix] = time.time() - t0
df = pl.DataFrame({"tolerance": tols, "time": timing, "version": sklearn.__version__})
df

[OmarManzoor]

@lorentzenchr Thank you for the PR

[OmarManzoor]

LGTM. Thank you @lorentzenchr

[OmarManzoor]

Suggested change

	# initial value "Q w" which will be kept of up to date in the iterations
	# initial value "Q w" which will be kept up to date in the iterations

[lorentzenchr]

I ended up with b50b4ff to align the docstring of the CD routines.
Downside: much larger diff.

[virchan]

Thank you for the PR, @lorentzenchr!

Overall, LGTM. Just have a minor question before merging.