Worklog: #1690 Poisson/Gamma 1-D s(x) outer-loop perf

[SauersML]

Starting investigation of #1690. Poisson 1-D s(x) ~28x slower than mgcv, Gamma ~7x, at equal RMSE. n=600, single smooth. Both REML + double penalty (select=TRUE).

Prior art: #1575 (binomial) traced the same family of problem to the outer-loop work count (seed-grid prepass ~20 full-n solves + multistart + ARC Newton) and the default-ON Firth path. Poisson/Gamma do NOT pay Firth, so the residual here should be dominated by the seed-grid/multistart scaffolding (#1033 territory) plus any family-specific overhead. The 4x Poisson-vs-Gamma gap is the interesting clue — investigating what differs.

Will profile the actual Rust path and look for correctness-preserving levers.

[SauersML]

First Rust-level repro (n=600, single s(x,k=12), REML+double penalty), exact issue DGP:

RECORD poisson: reml_score=-1121.88 edf=6.71 outer_cost_evals=63 inner_pirls_solves=97 grad_norm=1.3e-6 converged=TRUE
RECORD gamma: reml_score=329.26 edf=6.70 outer_cost_evals=60 inner_pirls_solves=98 grad_norm=0.23 converged=FALSE

Two findings:

1. ~97 full-n inner P-IRLS solves for a SINGLE smooth (k=2 smoothing params w/ double penalty). mgcv does ~8 Newton steps. This is the n-free spatial κ-loop residual: clear the #1216 PsiGramTensor tail-cert wall via amplitude-normalized Chebyshev interpolation (rotation wall RESOLVED 82cf6049f+b437d9ff2; Duchon/ThinPlate scope, Matérn=#1274; SAE exact n-independence conceded impossible — amortized ceiling) #1033 seed-grid/multistart scaffolding — the dominant cost, family-agnostic.
2. GAMMA DOES NOT CONVERGE: grad_norm=0.23, converged=false — but score-relative bound 1e-3*(1+329)=0.33, so it's at a score-relative stationary point yet the convergence flag rejects it, so the outer loop runs to max_iter wasting solves. Poisson converges cleanly (1.3e-6). This asymmetry is family-specific and a correctness+perf bug. Digging in.

[SauersML]

Root cause nailed + fix shipped (PR #1730)

The actionable Gamma bug

The Gamma s(x,k=12) n=600 REML fit reaches the genuine optimum (ρ≈[-1.367, 7.533], score=329.259, edf=6.70) but reported outer_converged=false. That false non-convergence then triggered wasted deterministic-replay ARC retries.

Mechanism (verified, not theorized): The in-loop cost-stall guard decides converged from the projected gradient it samples at the best-by-cost iterate during the ρ search. The Gamma log-link profiles the dispersion shape into the inner working weight, so at a weakly-identified flat ρ-valley the inner P-IRLS fixed point — and hence the analytic ρ-gradient — is warm-start sensitive. The guard sampled |g|=0.3646, just above the 0.330 score-relative stationarity bound, and halted non-converged.

But the authoritative gradient — re-measured at the shipped β after the accept-fit re-estimates the Gamma shape (refine_dispersion=true), the value surfaced as outer_gradient_norm — is |g|=0.2297 < 0.330. The point is stationary. Poisson (canonical link, W=μ gives curvature) converges cleanly at |g|≈1e-6 and never hits this.

Disproven dead-ends: (1) re-measuring the gradient inside the bridge at stall-time reproduces the noisy 0.3646 byte-for-byte (the warm-start β is the best-by-cost β); (2) the run_plan finalize re-solve also returns 0.3646 (cache hit on the same β). The authoritative 0.2297 only materializes downstream in optimizer.rs via compute_gradient at the refined-dispersion β. So that is the only site that can reconcile honestly.

Fix (two coordinated edits)

1. optimizer.rs — reconcile outer_converged against the authoritative gradient of the fit actually shipped. A CostStallFlatValley halt whose finalgrad_norm clears the same score-relative bound is re-certified converged. Gated strictly on the flat-valley stop reason; the Gamma/log REML ships a silent NON-CONVERGED full-basis overfit on ~7% of datasets at default k #1426 stuck overfit (honest |g|≈11 ≫ bound) never clears it and stays non-converged.
2. run.rs — treat CostStallFlatValley like RejectFloor in the ARC retry gate. Verified the retry fired twice, each replaying the full trajectory at the same ρ returning |g|=0.3646 byte-for-byte — pure waste. Skipping it drops the repro's inner_pirls_solves 146 → 128.

Verification

• Regression harness poisson_gamma_outer_work_1690.rs: gamma now converged=true grad_norm=0.2297 inner_pirls_solves=128, same reml_score/edf. Tightened to assert the gradient clears the stationarity bound (certificate, not flag flip) and solves < 200 (catches a returning replay).
• Full gam-solve suite: zero new failures vs clean origin/main (diff of failure sets is empty; the 12 pre-existing failures need R/GPU tooling absent in this env; 2 SIGTERMs are GPU test timeouts).
• The Gamma/log REML ships a silent NON-CONVERGED full-basis overfit on ~7% of datasets at default k #1426 guards that don't need R both pass: gamma_log_default_k_never_ships_full_basis_overfit ✅ and survival_outer_converged_agrees_with_pirls_status_not_hardcoded ✅.

Out of scope (→ #1033)

The dominant family-agnostic cost — the seed-grid prepass + multistart's ~80 full-n inner solves for a single smooth (mgcv ~8) — is the larger share of the 7–28x and is load-bearing (#1266/#1548/#1464/#1426). Deferred to #1033.

[SauersML]

✅ SHIPPED — PR #1730 squash-merged to main as 14c503e46 (2026-06-30T07:49:25Z).

Fix (part B, the actionable Gamma-specific bug): The Gamma s(x,k=12) n=600 REML fit reaches the optimum (ρ≈[-1.367, 7.533], score=329.259, edf=6.70) but the in-loop ARC cost-stall guard sampled a warm-start-sensitive gradient |g|=0.3646 (just above the score-relative bound ≈0.330) at the best-by-cost iterate and halted converged=false. The authoritative gradient at the shipped (refined-dispersion) β is |g|=0.2297 < 0.330 — the point IS stationary.

Two-part fix:

1. optimizer.rs (~2586): reconcile outer_converged — when the stop reason is CostStallFlatValley and the authoritative compute_gradient at the shipped β clears the score-relative bound, re-certify converged. Gated strictly on the flat-valley stop reason, so Gamma/log REML ships a silent NON-CONVERGED full-basis overfit on ~7% of datasets at default k #1426 stuck-overfit (honest |g|≈11) never clears it.
2. run.rs retry gate: add CostStallFlatValley alongside RejectFloor as a no-retry stop reason. The false non-convergence had been firing wasted deterministic-replay ARC retries (verified twice, each |g|=0.3646 byte-for-byte). Skipping drops inner_pirls_solves 146 → 128.

Result: gamma converged=true grad_norm=0.2297 inner_pirls_solves=128, identical reml_score 329.2589734249 / edf 6.6989. Regression test poisson_gamma_outer_work_1690.rs asserts a real stationarity certificate (|g| ≤ score-relative bound), not a flag flip.

Part A (dominant, family-agnostic ~80 prepass+multistart solves) deferred to #1033 — every probe is load-bearing (#1266/#1548/#1464/#1426). Zero regressions vs baseline main (failure-set diff empty; the 14 full-suite failures are all pre-existing — GPU SIGTERM timeouts + R/mgcv-absent + numerical-tolerance tests, byte-for-byte identical on clean origin/main).