Skip to content

Featurizer 0.7.0

Choose a tag to compare

View all tags
@github-actions github-actions released this 10 Jul 15:35
· 53 commits to master since this release
v0.7.0
9693410

Performance release: the two root causes found by EXPLAIN (ANALYZE) on the
live triage databases (correlated two-window drift → ADR-0012; no-stats
as_of_dates cardinality → ADR-0013) plus conservative planner tuning as an
executor default. Full-aggregator materialization on every live DB dropped from
10–357s to ~6–8s; values proven unchanged by the golden gate throughout.

Known issues

  • The wide variant (all 65 aggregators × 14 transformers) on the widest
    configs can OOM the PostgreSQL backend during query planning.     Diagnosed
    on live donorschoose (2026-07-10): ~14.9k output columns shard into 27 group
    queries of up to ~979 CTEs / 1.8 MB SQL each; planning a single group takes
    30–45s and spikes backend memory until the kernel OOM killer fires (observed
    at a plain EXPLAIN, with a 3000-row cohort — data volume is irrelevant).
    Wide-everything is an extreme, atypical config; mitigation directions
    (CTE-bounded sharding, TEMP-materialized shared pre-passes, per-group
    connections) are recorded in the project TODO.

Changed

  • Conservative PostgreSQL planner/memory tuning is now an executor default.
    Every generated query is a wide multi-way CTE join, which starves under
    PostgreSQL's stock work_mem and collapse limits. The executor now issues
    SET LOCAL work_mem = '64MB', join_collapse_limit = 20,
    from_collapse_limit = 20 (measured ~1.4× on dirtyduck all-agg; a supporting
    lever on top of ADR-0012/0013). geqo deliberately stays ON — the aggressive
    variant (256MB / collapse 30 / geqo off) crashed the backend by exhaustively
    planning a 38-way join. The tuning is applied only to connections featurizer
    opens itself    : a caller's connection= is never touched, because SET LOCAL
    would stay in force for the remainder of the caller's open transaction. On the
    records fast path the SETs share one held connection (and transaction) with
    the query; on the psycopg paths they are savepoint-isolated and best-effort,
    like the ANALYZE. New PLANNER_TUNING / tuning_statements() /
    apply_planner_tuning() in featurizer.executor; covered by
    tests/test_executor_tuning.py.

  • Executor ANALYZEs as_of_dates before running (ADR-0013). The caller's
    freshly-created as_of_dates has no statistics, so PostgreSQL assumed its
    ~2550-row default and planned the lateral-join body for the wrong cardinality —
    a single Merge Join was 99% of donorschoose all-agg's runtime. The executor now
    issues a best-effort, savepoint-isolated ANALYZE as_of_dates on its working
    connection first, in every path (to_dataframe, to_arrow, to_tables).
    donorschoose all-agg 293.6s → 7.5s, dirtyduck 27.6s → 7.0s (~40–50×); values
    unchanged (ANALYZE refreshes stats, not data — golden gate passes).

  • Two-window drift aggregators migrated to set-based pre-aggregation (ADR-0012).
    kl_drift / wasserstein_drift, which ADR-0010 deferred as a non-goal, were the
    entire cost of full-aggregator materialization on real data: live EXPLAIN (ANALYZE) showed 9 correlated SubPlans over the child stream at loops=18909
    (kl_drift firing on ordinary categorical columns × intervals, O(target×children)).
    Rewritten as companion CTEs — recent/baseline counts via count(*) FILTER (KL,
    no self-join) and per-window percentile_cont … FILTER (Wasserstein). dirtyduck
    all-agg 356.8s → 27.6s (~13×)    , all 272 features retained, values proven identical
    by the golden-value gate (now 29 migratable aggregators / 232 frozen cases; P3M
    cases added since drift is degenerate under P1M). Output column names unchanged
    (ADR-0007). Companion-CTE budget guard 132 → 144.

  • ln / log / sqrt transformers are now domain-guarded (ADR-0011). They
    render case when x > 0 then ln(x) end (>= 0 for sqrt) instead of a bare
    ln(x), so an out-of-domain row becomes SQL NULL rather than aborting the
    whole materialization with cannot take logarithm of a negative number. This
    hard-broke any wide/all-transformer config the moment a transformer landed on a
    signed feature (z-score, difference, deviation) — surfaced on the live-DB wide
    variant. Output column names/labels are unchanged (ADR-0007). New
    DomainGuardedTransformer base; guards covered by
    tests/primitives/test_transformations.py.

Fixed

  • Companion pre-aggregation CTE name over 63 bytes emitted an invalid bare
    ~.     A set-based companion CTE (ADR-0010) whose <child>_<family>_<interval>_preaggs_for_<target>
    name exceeded PostgreSQL's 63-byte identifier limit was hash-capped by
    pg_identifier with a ~ separator (safe only inside quotes — output columns
    are always quoted), but _build_preagg_cte strips the quotes to interpolate
    the name bare, leaving a ~ that PostgreSQL parses as an operator
    (syntax error at or near "~"). This hard-broke the full-aggregator config on
    any data with long categorical column names — invisible to the DB-free tests
    and surfaced only by running the integration suite against the live
    food-inspections / dirtyduck data (8 failing realistic tests). The cap
    separator is now folded to _ for the bare CTE identifier; CTE names are
    internal-only, so the ADR-0007 output-column naming contract is untouched.
    Regression guard: tests/test_preagg_shape.py::test_preagg_cte_name_over_63_bytes_is_a_valid_bare_identifier.
Assets 4
Loading