Repository navigation
SuperGLM 0.42.0
·
100 commits
to master
since this release
0.42 is mostly about SuperLSS: censored and truncated distributions, the editor, and faster fits and predictions.
Highlights
- Censored and truncated distributions for Gaussian, log-normal, generalized Pareto, Gamma and generalized gamma SuperLSS fits, for claims capped at a policy limit or cut off at a deductible (#545, #565).
- The editor opens, edits, refits and exports SuperLSS models, with cross-validation and a Diagnose tab (#581, #587, #588).
- Binned SuperLSS fits: the Tweedie book at 224 coefficients takes 7.70 s, from 19.40 s (#598), after #569, #579, #585, #591 and #594.
- SuperLSS prediction: 638,007 rows take 0.40 s, from 2.31 s, and a one-row quote about 0.84 ms, from about 3 ms (#597).
- Shape constraints fit on the QP: one joint Newton step, faster P-splines, and rules joined with & (#527, #537, #573, #580).
- import superglm takes about 3 ms, from about 1.8 s, and a model loads on a read-only file system (#597).
- Exact REML fits no longer rebuild the weighted Gram matrix they have just built (#602), and pre-solve more families (#542).
Changed behaviour
- SuperLSS models save with joblib; to_bytes and from_bytes are gone (#524).
- The SCOP shape engine is retired: scop specs fit on the QP with a warning, so their held-out results move slightly, and the scalar EFS REML route is deleted (#573).
- A refit of a
scopspec moves by fixture, with no net change: across 84 compared fits, held-out Gini rose on 42 and fell on 42, by more than 0.01 on 6 and 10 of them, all on small or sparse synthetic data. Two NB2 fits with estimated theta stopped converging. - superglm.attach is no longer an accidental public name (#597).
- A SuperLSS Tweedie fit with its power on a wall moves by more than rounding, standard errors of the mean and dispersion coefficients by up to 2.6%, because the power link's derivatives near the wall were inaccurate (#598).
Known
- Compared with 0.41.0, two non-SCOP fits now report not converged, with answers that barely moved: an NB2 fit with estimated theta and a random effect stops on line_search_failed with its gradient 1.6 times over the tolerance (from #549), and a rich GammaLS fit on outer="efs" stops on objective_rejected (from #583). Tracked in #604.
- A Tweedie fit with an all-zero category in mean and dispersion does not converge, on this version or 0.41.0, and row order moves its EDF.
Features and behaviour
- SuperLSS models save with joblib, and to_bytes and from_bytes are gone (#524)
- Shape-constrained fits take one joint Newton step and need half the inner iterations (#527)
- Shape-constrained P-splines fit on the QP that cr and bs use: 1.3-17× faster than SCOP where both engines converge (#537)
- Diagnostics, bands and shape repairs give the same answer in any units (#548)
- Truncated and censored rows for Gaussian, log-normal and generalized Pareto LSS fits, read the same way by the diagnostics (#545)
- GammaLS and GeneralizedGammaLSS take truncation= and censored=, with analytic orders 3-4 on bounded rows (#544 stage 2) (#565)
- Binned SuperLSS fits contract their curvature once per block of chunks, up to five times faster, drop the small-group panels, and certify their terminal rank (#569)
- Retire the SCOP shape engine: scop specs fit on the QP with a warning; delete the scalar EFS REML route and max_analytical_per_w (#573)
- Trim stage 1b: dead code and redundant fit re-checks (src −3,717, tests −2,823) (#575)
- Shape rules joined with &, stretches, flat, relativity caps and band step limits; independent QP working set (#580)
- Editor: open, edit, refit and export SuperLSS models (#581)
- The opt-in SuperLSS Newton endgame keeps its stationary stops, spends one iteration budget and takes cheaper derivative passes; the default stays plain EFS (#582)
- SuperLSS fits run faster: repeated work removed, the dense Gram on the pool, and a default search that finishes its plateaus (#585)
- Editor: SuperLSS cross-validation, live run progress and the distribution panel (#587)
- Editor: a Diagnose tab shows how each fit's smoothing search went (#588)
- SuperLSS prediction: fast quotes, exact standard errors, an instant import, and loading on read-only file systems (#597)
Fixes and speed
- About 2,500 lines of unreachable code go, and design_summary describes factor smooths (#525)
- Each categorical column is coded once per build (#526)
- Shape-constrained terms under an identity link smooth the same whatever the response's units, and one module states the rounding constants (#530)
- Discrete pattern keys past 2**53 decode every row: the codes add as uint64 (#532)
- REML with random effects takes the leverage W-correction route where its design reads row supports (#531)
- Discrete Gram kernels read codes as unsigned, near-nesting notes count in one pass, and a QP refit skips dead rank work (#534)
- A SCOP row-route leaf multiplies each block once for all its columns, so rare-level SCOP fits run 2–3× faster with identical answers (#535)
- The editor reads a held-flat curve's rows as the fit does; tight memory tests read their own call's peak (#540)
- REML pre-solves Tweedie, Gamma/log, binomial and NB2 fits too: about 2× faster on freMTPL2, each closer to its reference optimum (#542)
- A post-fit repair keeps a constrained term's face, a numeric relativity no longer overflows, and tight memory tests read their own peak (#543)
- The direct IRLS solver is a command object with one strategy per route, bit for bit the same fits (#536)
- The W(ρ) correction's derivative Grams take the IRLS Gram's dense split where it pays: about 2 s off a std_mains fit (#546)
- Direct IRLS audit fixes: an intercept-only fit, SCOP's domain refusal and start, a QP face that rounding corrupted, one stagnation rule (#547)
- A nested random-effect chain converges: random effects start at their levels' mean information given the offset (#549)
- Binomial/logit REML reads its W(ρ) leverages from the shared leverage route: the binomial book fits in 19 s instead of 76 s (#555)
- Build exact spline, tensor and spline-by-category blocks on distinct values and occupied cells (#557)
- GLM REML evaluates log|S_λ|₊ block by block: interaction books fit 3.5–11 s faster (#556)
- A generalized Pareto fit whose tail shape is pressed against a wall says so (#558)
- SuperLSS sheds 1,316 lines of self-checks, shims and duplicate helpers, with every output bitwise unchanged (#566)
- Issue 544 follow-on: GG expected shortfall through Q = 0, analytic bounded orders 3-4 for Gaussian, log-normal and GPD, and five fixes (#568)
- SuperLSS: aliased coefficients published at zero, offset columns converge, and EFS steps on row-decided ranks use the matrix they invert (#576 items 4, 5, 6, 8, 9, 10) (#577)
- Binned SuperLSS passes contract their tables once per pass, retire global moments and size chunks by what they build (#576 items 2, 3, 7) (#579)
- Audit defect fixes rebuilt from draft #571 (#583)
- Binned SuperLSS fits run up to twice as fast: row passes contract in long blocks and the finish forms no design rows (#591)
- SuperLSS fits run faster: compiled row kernels for every family that gains from them, a faster generalized gamma, and less peak memory on Linux (#594)
- Four guards against recurring review mistakes, and a work-ledger repeat count that does not depend on memory addresses (#595)
- Binned SuperLSS fits run faster at the same answers: the whole book at 224 coefficients in 7.70 s, from 19.40 s (#598)
- Exact REML fits stop rebuilding the weighted Gram matrix they have just built (#602)
Maintenance
- filelock 4.2.1 and virtualenv 21.14.13 for their advisories (#528)
- Tests: every publication guard is fault-injected, export refusals are covered, and weak assertions check what the code is for (#529)
- tornado 6.6 for its advisories (#539)
- A term the constraint holds flat beside a fitted one adds no degrees of freedom (#538)
- Code outside the fit loop: one weight contract, one repair body, one fit publication (#551)
- Solver code: one structured-kind table, one profiled-adapter base, shared IRLS tails (#550)
- REML engines: one stop and freeze check, one outer-Hessian trace block, one SCOP restart (#552)
- Test-only code out of src: screening oracles, block Gram entry points, unread parameters (#553)
- The batched-moments copy test traces a steady-state call, not the pool's one-off setup (#559)
- Bump the github-actions group across 1 directory with 5 updates (#533)
- Research notes: large-loss modelling sweep and the severity-family framework (C2) (#563)
- Prose pass: docstrings and comments state the rule, not the history (src −7,159) (#574)
- Prose round 2: docstrings and comments state the rule; private-docstring length lint (src −2,878) (#578)
- Binned SuperLSS design rows read from bin tables formed once per pass, for the derivative passes (#582) (#584)
- A debug-only work ledger lists the repeated and rebuildable work in SuperGLM and SuperLSS fits (#590)
