How to Train Your Intercept

Johan Larsson, Frederik Fabricius Bjerre 2026-08-26

How the intercept is updated inside coordinate descent for regularized generalized linear models, and why the choice matters under response imbalance.

Authors

• Johan Larsson (University of Copenhagen)
• Frederik Fabricius Bjerre (University of Copenhagen)

Abstract

Coordinate descent solvers for regularized generalized linear models (GLMs) differ in how they update the intercept. This seemingly routine choice can substantially affect convergence when the response is imbalanced. For direct coordinate descent on the original GLM loss, we compare a conservative step based on worst-case curvature, a safeguarded Newton step based on current curvature, and repeated Newton steps that solve the intercept subproblem. The conservative update can leave much of the interaction between the intercept and the coefficients unresolved, producing a slowdown when that interaction controls convergence. A single Newton step avoids this attenuation and, in our experiments, follows nearly the same outer trajectory as exact minimization while requiring fewer inner steps. Simulated and real-data experiments, including comparisons across six production solvers and a controlled intervention within skglm, support this explanation. We also show how the same mechanism appears in quadratic-approximation methods and extends to multinomial models with rare classes. We recommend a safeguarded Newton update for direct coordinate descent and full intercept optimization within frozen local quadratic surrogates.

Project structure

This project is structured for the Computo journal, a journal for reproducible research in computational science. It is built around a Quarto notebook, which is a document format that allows you to combine text, code, and results in a single document. But the project also contains some code to run experiments and generate results for the notebook:

• contains the scripts to run experiments
• contains a Julia module (Intercepts) which is used to run the experiments.
• contains tests for the Intercepts module.
• contains the results of the experiments, which are generated by the scripts in .

Reproducibility

Our intention is for this project to be fully reproducible. To reproduce the results, you need to follow these steps.

Setup Quarto and the Computo extension

You need quarto installed on your computer, as well as the Computo extension to prepare your document. Computo’s build installs the latest release, so we do not pin a version here:

quarto add computorg/computo-quarto-extension

Install Julia

You need to have Julia installed on your computer. This project is built using Julia 1.11 (currently 1.11.9 — the version CI pins and under which Manifest.toml was resolved), which you can install through juliaup, a tool to manage Julia installations.

After this, you need to activate the Julia environment for this project. You can do this by running the following command in the root directory of the project:

]activate .
]instantiate

This will use the Project.toml and Manifest.toml files to set up the Julia environment with the required dependencies.

(Optional) Retrieve the real-data inputs

The real-data experiments read their inputs from a pinned archive deposited on Zenodo (DOI: 10.5281/zenodo.20315625), rather than fetching datasets at run time. The archive holds the exact inputs behind the real-data figures: the Yeoh2002 gene-expression panel, the LIBSVM datasets (w1a, news20.binary, shuttle.scale, a4a, leukemia, breast-cancer, gisette), and the congress109 phrase-count matrix. To stage them under data/, run:

bash experiments/fetch-data.sh

This downloads the archive, verifies every file against the committed data/MANIFEST.sha256, unpacks it, and derives the Yeoh CSV inputs from the shipped RDS. See <data/README.md> for the provenance of each dataset.

You only need this step to re-run the real-data experiments. The cached results in already render the paper without it.

(Optional) Reproduce results

Many of the results have already been computed and are included in the directory. You can reproduce these results by running the scripts in the directory. For example, to reproduce the results for the intercepts module, you can run:

julia --project=. experiments/sim-mu-extreme.jl

The production-solver comparison uses a two-stage workflow. First, Julia generates the shared standardized logistic problem consumed by every solver:

julia --project=. experiments/sim-real-problem.jl

Then the solver-specific drivers consume those cached inputs and write the CSV outputs used by the paper:

• Rscript experiments/sim-real-glmnet.R
• Rscript experiments/sim-real-biglasso.R
• Rscript experiments/sim-real-adelie.R
• python experiments/sim-real-skglm.py
• python experiments/sim-real-proxnewton.py

The exact tolerance grids and iterate caps for these sweeps are encoded in the driver scripts. The single-$\lambda$ cold-start diagnostic discussed in the paper can be reproduced with:

Rscript experiments/sim-cold-start-real.R

Figure and table reproduction map

The table below maps every empirical figure and cached table in the manuscript to its committed input and the command that regenerates it. Run commands from the repository root inside the pinned environment described below. “Archive” means that bash experiments/fetch-data.sh must first stage the real-data inputs; “none” means that the producer generates its own synthetic data.

Manuscript output	Committed input	Producer command	Environment	Extra data
@fig-parametric	None; computed during render	quarto render intercepts.qmd	Quarto and Julia	None
@fig-first-example	results/first-example.jld2	julia --project=. experiments/first-example.jl	Julia	Archive
@fig-multinomial-summary, @fig-real-multinomial	results/real-multinomial-yeoh.jld2	julia --project=. experiments/sim-real-multinomial-yeoh.jl	Julia	Archive
@fig-multinomial-summary, @fig-real-multinomial-shuttle	results/real-multinomial-shuttle.jld2	julia --project=. experiments/sim-real-multinomial-shuttle.jl	Julia	Archive
@fig-irls-comparison	results/sim-irls-comparison.jld2	julia --project=. experiments/sim-irls-comparison.jl	Julia	None
@fig-mu-extreme	results/sim-mu-extreme.jld2	julia --project=. experiments/sim-mu-extreme.jl	Julia	None
@fig-cold-start	results/sim-cold-start.jld2	julia --project=. experiments/sim-cold-start.jl	Julia	None
@fig-cold-start-poisson	results/sim-cold-start-poisson.jld2	julia --project=. experiments/sim-cold-start-poisson.jl	Julia	None
@fig-real-logreg	results/real-logreg.jld2	julia --project=. experiments/sim-real-logreg.jl	Julia	Archive
@fig-real-poisson	results/real-poisson.jld2	Rscript experiments/sim-real-poisson-problem.R, then julia --project=. experiments/sim-real-poisson.jl	R and Julia	Archive
@fig-mu-reg-gradient, @fig-mu-reg-exact	results/sim-mu-reg-heatmap.jld2	julia --project=. experiments/sim-mu-reg-heatmap.jl	Julia	None
@fig-real-glmnet, @fig-real-biglasso, @fig-real-skglm, @fig-real-proxnewton, @fig-real-adelie	The five solver CSVs in each of results/real-solvers/, w1a/, and news20-3pct/; results/production-references.csv	julia --project=. experiments/sim-real-problem.jl; Rscript experiments/sim-real-glmnet.R; Rscript experiments/sim-real-biglasso.R; Rscript experiments/sim-real-adelie.R; python experiments/sim-real-skglm.py; python experiments/sim-real-proxnewton.py; julia --project=. experiments/production-reference.jl	Julia, R, and Python	Archive
@fig-skglm-controlled	results/skglm-controlled/results.csv	bash experiments/run-skglm-controlled.sh	Julia and Python	None
@fig-rate-gap	results/rate-gap.jld2	julia --project=. experiments/sim-rate-gap.jl	Julia	None
@fig-rho-centering	results/rho-centering.jld2	julia --project=. experiments/sim-rho-centering.jl	Julia	None
@fig-rho-frozen	results/rho-frozen-hessian.csv, results/rho-centering-diagnostics.csv	julia --project=. experiments/sim-rho-centering-diagnostics.jl, then julia --project=. experiments/sim-rho-frozen-hessian.jl	Julia	None
@fig-multinomial-imbalance	results/sim-multinomial-imbalance.jld2	julia --project=. experiments/sim-multinomial-imbalance.jl	Julia	None
@fig-multinomial-sweep, @fig-multinomial-sweep-full	results/sim-multinomial-imbalance-sweep.jld2	julia --project=. experiments/sim-multinomial-imbalance-sweep.jl	Julia	None
@fig-first-example-ordering	results/first-example-ordering.jld2	julia --project=. experiments/sim-first-example-ordering.jl	Julia	Archive
@fig-warm-start, @fig-warm-start-mechanism, @fig-warm-start-full, @fig-warm-start-mechanism-full	results/sim-warmstart-path.jld2	julia --project=. experiments/sim-warmstart-path.jl	Julia	Archive
@fig-per-pass-cost, @fig-per-pass-progress	results/sim-per-pass-cost.jld2	julia --project=. experiments/sim-per-pass-cost.jl	Julia	None
@fig-real-poisson-production	glmnet.csv, adelie.csv, and proxnewton.csv under results/real-solvers-poisson/congress109/; results/production-references.csv	Rscript experiments/sim-real-poisson-problem.R; Rscript experiments/sim-real-poisson-glmnet.R; Rscript experiments/sim-real-poisson-adelie.R; python experiments/sim-real-poisson-proxnewton.py; julia --project=. experiments/production-reference.jl	Julia, R, and Python	Archive
@tbl-cold-start-diag	results/real-solvers/cold-start-diag.csv	Rscript experiments/sim-cold-start-real.R	R	Archive

The CSV files under results/figure-tables/ are not experiment inputs: they are per-figure summary tables exported from the committed caches by dev/figure.jl (task fig -- <label> or task fig-all), tracked so that regenerated results can be diffed at the figure level.

Runtimes and hardware

The synthetic Julia cache producers in the table can be regenerated in one go with task caches, and the real-data ones (after staging the archive) with task caches-real. On a laptop-class machine (see the hardware note in Supplement S3 of the manuscript), the single-$\lambda$ synthetic scripts (sim-mu-extreme, sim-irls-comparison, sim-rate-gap, sim-cold-start, sim-cold-start-poisson, sim-per-pass-cost, first-example, and the multinomial scripts) each finish in roughly one to five minutes, including Julia startup. The larger sweeps—sim-mu-reg-heatmap, sim-warmstart-path, and especially the sim-rho-* rate diagnostics, which run to pass budgets of 40,000–120,000—take substantially longer, up to a few hours in total. The real-data scripts additionally require the staged archive and, for the production-solver comparisons, the pinned R and Python environments.

Render the Quarto document

To render the Quarto document, you can run the following command in the root directory of the project:

quarto render intercepts.qmd

This will generate the HTML and PDF versions of the document in the _site/ directory.

(Optional) Reproduce using devenv

We provide a devenv configuration (devenv.nix and devenv.lock) that reconstructs the complete toolchain. The lock file records the exact Nix revisions and content hashes behind each input; devenv.nix then selects packages from those immutable inputs. The environments map to the drivers as follows.

• Julia 1.11: devenv supplies the interpreter, while Project.toml and Manifest.toml supply the packages used by the tests, notebook, and Julia experiments.
• R: locked nixpkgs supplies R, glmnet, biglasso, adelie, and the other CRAN packages used by experiments/*.R.
• Python: locked nixpkgs and the fixed-output sources in devenv.nix supply Python, skglm 0.5, BlitzL1, Celer, and their dependencies for experiments/*.py.
• Controlled skglm comparison: separate locked inputs supply the pre-fix and post-fix skglm source trees used by run-skglm-controlled.sh.

After installing devenv, enter the environment from the root of the project with:

devenv shell

This provides R, Python, Julia 1.11, Quarto, and every external solver used by the production drivers. Instantiate the Julia packages once with task install. The controlled skglm comparison needs no separate checkout:

bash experiments/run-skglm-controlled.sh

Continuous integration

Two GitHub Actions workflows cover the project, and they guarantee different things. build.yml renders and publishes the paper: it delegates to Computo’s reusable workflows—global-env.yml builds the pinned environment, and publish-render.yml runs quarto render and deploys to GitHub Pages—and it never runs the test suite. It is the journal-managed builder, so per Computo’s author guidelines we do not modify it. test.yml is ours: it builds the package and runs the Intercepts test suite (Pkg.test()) on every push and pull request, which is what guarantees the correctness of the code in src/. Both pin Julia 1.11 to match the version under which Manifest.toml is resolved.