Skip to content

SuperGLM 0.42.0

Choose a tag to compare

@github-actions github-actions released this 07 Nov 13:19
· 100 commits to master since this release

superglm 0.42, the SuperLSS release: censored and truncated distributions for policy limits and deductibles, SuperLSS in the editor, the Tweedie book in 7.7 s, and shape constraints on one QP

0.42 is mostly about SuperLSS: censored and truncated distributions, the editor, and faster fits and predictions.

Highlights

  • Censored and truncated distributions for Gaussian, log-normal, generalized Pareto, Gamma and generalized gamma SuperLSS fits, for claims capped at a policy limit or cut off at a deductible (#545, #565).
  • The editor opens, edits, refits and exports SuperLSS models, with cross-validation and a Diagnose tab (#581, #587, #588).
  • Binned SuperLSS fits: the Tweedie book at 224 coefficients takes 7.70 s, from 19.40 s (#598), after #569, #579, #585, #591 and #594.
  • SuperLSS prediction: 638,007 rows take 0.40 s, from 2.31 s, and a one-row quote about 0.84 ms, from about 3 ms (#597).
  • Shape constraints fit on the QP: one joint Newton step, faster P-splines, and rules joined with & (#527, #537, #573, #580).
  • import superglm takes about 3 ms, from about 1.8 s, and a model loads on a read-only file system (#597).
  • Exact REML fits no longer rebuild the weighted Gram matrix they have just built (#602), and pre-solve more families (#542).

Changed behaviour

  • SuperLSS models save with joblib; to_bytes and from_bytes are gone (#524).
  • The SCOP shape engine is retired: scop specs fit on the QP with a warning, so their held-out results move slightly, and the scalar EFS REML route is deleted (#573).
  • A refit of a scop spec moves by fixture, with no net change: across 84 compared fits, held-out Gini rose on 42 and fell on 42, by more than 0.01 on 6 and 10 of them, all on small or sparse synthetic data. Two NB2 fits with estimated theta stopped converging.
  • superglm.attach is no longer an accidental public name (#597).
  • A SuperLSS Tweedie fit with its power on a wall moves by more than rounding, standard errors of the mean and dispersion coefficients by up to 2.6%, because the power link's derivatives near the wall were inaccurate (#598).

Known

  • Compared with 0.41.0, two non-SCOP fits now report not converged, with answers that barely moved: an NB2 fit with estimated theta and a random effect stops on line_search_failed with its gradient 1.6 times over the tolerance (from #549), and a rich GammaLS fit on outer="efs" stops on objective_rejected (from #583). Tracked in #604.
  • A Tweedie fit with an all-zero category in mean and dispersion does not converge, on this version or 0.41.0, and row order moves its EDF.

Features and behaviour

  • SuperLSS models save with joblib, and to_bytes and from_bytes are gone (#524)
  • Shape-constrained fits take one joint Newton step and need half the inner iterations (#527)
  • Shape-constrained P-splines fit on the QP that cr and bs use: 1.3-17× faster than SCOP where both engines converge (#537)
  • Diagnostics, bands and shape repairs give the same answer in any units (#548)
  • Truncated and censored rows for Gaussian, log-normal and generalized Pareto LSS fits, read the same way by the diagnostics (#545)
  • GammaLS and GeneralizedGammaLSS take truncation= and censored=, with analytic orders 3-4 on bounded rows (#544 stage 2) (#565)
  • Binned SuperLSS fits contract their curvature once per block of chunks, up to five times faster, drop the small-group panels, and certify their terminal rank (#569)
  • Retire the SCOP shape engine: scop specs fit on the QP with a warning; delete the scalar EFS REML route and max_analytical_per_w (#573)
  • Trim stage 1b: dead code and redundant fit re-checks (src −3,717, tests −2,823) (#575)
  • Shape rules joined with &, stretches, flat, relativity caps and band step limits; independent QP working set (#580)
  • Editor: open, edit, refit and export SuperLSS models (#581)
  • The opt-in SuperLSS Newton endgame keeps its stationary stops, spends one iteration budget and takes cheaper derivative passes; the default stays plain EFS (#582)
  • SuperLSS fits run faster: repeated work removed, the dense Gram on the pool, and a default search that finishes its plateaus (#585)
  • Editor: SuperLSS cross-validation, live run progress and the distribution panel (#587)
  • Editor: a Diagnose tab shows how each fit's smoothing search went (#588)
  • SuperLSS prediction: fast quotes, exact standard errors, an instant import, and loading on read-only file systems (#597)

Fixes and speed

  • About 2,500 lines of unreachable code go, and design_summary describes factor smooths (#525)
  • Each categorical column is coded once per build (#526)
  • Shape-constrained terms under an identity link smooth the same whatever the response's units, and one module states the rounding constants (#530)
  • Discrete pattern keys past 2**53 decode every row: the codes add as uint64 (#532)
  • REML with random effects takes the leverage W-correction route where its design reads row supports (#531)
  • Discrete Gram kernels read codes as unsigned, near-nesting notes count in one pass, and a QP refit skips dead rank work (#534)
  • A SCOP row-route leaf multiplies each block once for all its columns, so rare-level SCOP fits run 2–3× faster with identical answers (#535)
  • The editor reads a held-flat curve's rows as the fit does; tight memory tests read their own call's peak (#540)
  • REML pre-solves Tweedie, Gamma/log, binomial and NB2 fits too: about 2× faster on freMTPL2, each closer to its reference optimum (#542)
  • A post-fit repair keeps a constrained term's face, a numeric relativity no longer overflows, and tight memory tests read their own peak (#543)
  • The direct IRLS solver is a command object with one strategy per route, bit for bit the same fits (#536)
  • The W(ρ) correction's derivative Grams take the IRLS Gram's dense split where it pays: about 2 s off a std_mains fit (#546)
  • Direct IRLS audit fixes: an intercept-only fit, SCOP's domain refusal and start, a QP face that rounding corrupted, one stagnation rule (#547)
  • A nested random-effect chain converges: random effects start at their levels' mean information given the offset (#549)
  • Binomial/logit REML reads its W(ρ) leverages from the shared leverage route: the binomial book fits in 19 s instead of 76 s (#555)
  • Build exact spline, tensor and spline-by-category blocks on distinct values and occupied cells (#557)
  • GLM REML evaluates log|S_λ|₊ block by block: interaction books fit 3.5–11 s faster (#556)
  • A generalized Pareto fit whose tail shape is pressed against a wall says so (#558)
  • SuperLSS sheds 1,316 lines of self-checks, shims and duplicate helpers, with every output bitwise unchanged (#566)
  • Issue 544 follow-on: GG expected shortfall through Q = 0, analytic bounded orders 3-4 for Gaussian, log-normal and GPD, and five fixes (#568)
  • SuperLSS: aliased coefficients published at zero, offset columns converge, and EFS steps on row-decided ranks use the matrix they invert (#576 items 4, 5, 6, 8, 9, 10) (#577)
  • Binned SuperLSS passes contract their tables once per pass, retire global moments and size chunks by what they build (#576 items 2, 3, 7) (#579)
  • Audit defect fixes rebuilt from draft #571 (#583)
  • Binned SuperLSS fits run up to twice as fast: row passes contract in long blocks and the finish forms no design rows (#591)
  • SuperLSS fits run faster: compiled row kernels for every family that gains from them, a faster generalized gamma, and less peak memory on Linux (#594)
  • Four guards against recurring review mistakes, and a work-ledger repeat count that does not depend on memory addresses (#595)
  • Binned SuperLSS fits run faster at the same answers: the whole book at 224 coefficients in 7.70 s, from 19.40 s (#598)
  • Exact REML fits stop rebuilding the weighted Gram matrix they have just built (#602)

Maintenance

  • filelock 4.2.1 and virtualenv 21.14.13 for their advisories (#528)
  • Tests: every publication guard is fault-injected, export refusals are covered, and weak assertions check what the code is for (#529)
  • tornado 6.6 for its advisories (#539)
  • A term the constraint holds flat beside a fitted one adds no degrees of freedom (#538)
  • Code outside the fit loop: one weight contract, one repair body, one fit publication (#551)
  • Solver code: one structured-kind table, one profiled-adapter base, shared IRLS tails (#550)
  • REML engines: one stop and freeze check, one outer-Hessian trace block, one SCOP restart (#552)
  • Test-only code out of src: screening oracles, block Gram entry points, unread parameters (#553)
  • The batched-moments copy test traces a steady-state call, not the pool's one-off setup (#559)
  • Bump the github-actions group across 1 directory with 5 updates (#533)
  • Research notes: large-loss modelling sweep and the severity-family framework (C2) (#563)
  • Prose pass: docstrings and comments state the rule, not the history (src −7,159) (#574)
  • Prose round 2: docstrings and comments state the rule; private-docstring length lint (src −2,878) (#578)
  • Binned SuperLSS design rows read from bin tables formed once per pass, for the derivative passes (#582) (#584)
  • A debug-only work ledger lists the repeated and rebuildable work in SuperGLM and SuperLSS fits (#590)