Featurizer 0.8.0
Sharding rework: the donorschoose wide config (~36.8k columns) — a backend
crash in every previous snapshot — now materializes live in ~8 minutes, and
every cell of the 3-DB × 3-variant live matrix is green (all-agg is seconds
everywhere). Full refreshed artifacts: specs/live-db-revalidation-v080/
(+ summary page specs/live-db-revalidation-v080.html); decision record:
ADR-0005 amendment.
Changed
-
Column-group sharding now clusters columns by dependency lineage.
_partition_columnsbuckets the target's output columns by their
source-CTE signature before bin-packing, so same-lineage columns share a
group and each companion pre-aggregation CTE is emitted/executed by the few
groups that need it instead of most of them. Measured on the donorschoose
wideconfig (27 groups, ~14.9k columns): max per-group CTE closure
979 → 287, total closure 11,338 → 2,428, duplicated companion instances
899 → 18, emitted SQL 29.2 MB → 17.4 MB. Group composition changes
(which columns share a<stem>_group_NNNtable); the feature manifest's
feature_groupcolumn remains the supported mapping, and output column
names are unchanged (ADR-0007). -
Groups are additionally bounded by a window-function budget
(max_window_fns_per_group, default 500). PostgreSQL's planning memory
for N same-spec window functions in one select list is superlinear with a
hard cliff: measured live, ~675 window columns plan in ~5s while ~1,350
OOM-killed the backend during a plainEXPLAIN(fresh connection; both
halves of the same list plan fine — count, not content). The packer closes
a group early when adding a column would exceed the budget.Net effect of the two partitioning changes, measured live on the
donorschoosewideconfig (~36.8k output columns, 3,000-row cohort) that
previously OOM-killed the backend: materializes end-to-end in ~8 minutes
(32 groups, render 26.5s + execution 461.6s), max group closure 285 CTEs,
worst per-groupEXPLAINwell under 2s.
Fixed
-
Sharded re-join no longer collides on carried identifier columns. A
target that carries relationship keys beyond its id (donorschoose's
schoolid/teacher_acctid) repeats them in every group query;
to_dataframemerged groups on(as_of_date, id)only, so pandas raised
MergeError: duplicate columnsat the third group. The materialized path
now merges on the fullGroupedQueries.key_columnstuple. -
Sharded group queries no longer carry dead companion CTEs. Per-group
reachability now scans the pruned rendering of each target-level agg CTE
instead of its full-width body, so companion pre-aggregation CTEs whose only
consumer columns landed in other groups are no longer emitted. PostgreSQL 16
discards unreferenced CTEs at negligible planning cost (measured), so this
does not change plan shape — it shrinks the emitted SQL, parse time, and
render time on wide sharded configs.
Added
- Pre-flight plan-size guardrail.
ColumnGroupSharder.plan_size_report()
maps each column group to its live CTE-closure size, andwarn_plan_size()
(wired into every grouped path) logs one loud, actionable warning when any
group's closure predicts a PostgreSQL planner blowup — the failure mode
diagnosed on the donorschoosewideconfig, where ~1000-CTE group queries
took 30–45s of planning each and OOM-killed the backend during a plain
EXPLAIN. The warning names the worst groups and the config levers
(transformers / intervals / entities) instead of letting the run die
minutes later with "server closed the connection unexpectedly".