Releases: ccd-ia/featurizer
Release list
Featurizer 1.1.0
Additive only (semver MINOR under the ADR-0015 freeze). No existing surface
changed, no columns renamed, nothing new persisted — downstream feature
caches are unaffected, so consumers need no coordination beyond an optional
pin bump.
Added
-
Featurizer.columns_matching(pattern)/Featurizer.manifest_matching(pattern)
— select feature columns by globbing the manifest's full, untruncated
label.columns_matchingreturns physical column names in output order
(what goes in aselectlist or a feature-group definition);
manifest_matchingreturns the fullManifestEntryrows for labels,
lineage and intervals.This is the additive answer to a request that kept arriving as "reshape the
63-byte truncation so globs match the readable tail". That reshape was
measured and rejected — it is a regression, see
.out-of-scope/tail-preserving-truncation.md. The problem it was aimed at is
real, though: a glob written against physical names misses every truncated
column. On the sample config*frecuencia_cardiaca*resolves 672 columns
againstlabelbut only 198 againstcolumn. Downstream that is not silent
data loss — triage's explicitfeature_groups.definitionspath raises on the
unmatched columns — but the advice it gives ("add a glob or widen one") is
unactionable against an unpredictable hash. Resolving the group through
columns_matchingis the way out. -
featurizer.manifest.glob_to_like(pattern)— translates a glob into a
SQLLIKEpattern plus its escape character, for querying the persisted
"<schema>"."<stem>_manifest"table with the same syntax. It exists because
_is a literal in a glob but a single-character wildcard inLIKE, and
~95% of real feature labels contain one; a hand-rolled translation
over-matches silently. -
featurizer.manifest.filter_manifest(entries, pattern)— the underlying
free function, for manifests obtained from anywhere.
Behaviour to know about
- A pattern matching nothing raises
LookupError, it does not return[].
A silently empty selection is the failure mode the helper exists to prevent,
so the default is loud; the error carries near-miss suggestions (found by
literal-fragment backoff) and, when the pattern looks aimed at a truncated
physical name, says so explicitly. Passallow_empty=Trueto opt out when
probing for optional features. - Matching is case-sensitive (
fnmatch.fnmatchcase, neverfnmatch, which
is case-insensitive on Windows) and always againstlabel, nevercolumn
ordefinition—definitioninherits the parent's truncation in 234 of the
sample config's 1,217 truncated columns and drops the interval, so windowed
siblings share one.
Featurizer 1.0.1
Pure bug-fix release (semver PATCH under the ADR-0015 freeze). No feature
names change for any config that does not use the empty-list spelling —
the SQL/naming snapshot test is byte-identical.
Fixed
transformations: []/aggregations: []were silently swallowed.
The primitive-selection sites usedconfig.get(...) or DEFAULT, which
treats an explicit empty list as "unset": a config writing
transformations: []to suppress the transform layer silently got all
17 defaults (reported live from triage-pg — ~2,500 unwanted
CUM_SUM/ABS-composition columns). An empty list now suppresses that
layer:transformations: []passes features through unchanged
(byte-identical output to the[identity]workaround spelling, which
keeps working), andaggregations: []builds zero aggregation features
(legal-but-weird by decision — the planner already emits no CTE/join for
a zero-feature relationship). Absent ornullkeys keep applying the
curated defaults. Covered DB-free and with executing integration tests;
documented in the configuration reference.
Downstream note: triage-pg pins the engine version into artifact
identity (its ADR-0016), so its pin bump after this release
intentionally invalidates feature caches — no coordination needed; no
columns are renamed.
Featurizer 1.0.0
The stability release: no new feature families — 1.0.0 is the right to
rely on what 0.9.x already shipped. Every claimed compatibility is now
tested, every known sharp edge is guarded or loudly documented, the 0.9.x
families are validated at realistic scale, and the public surface is frozen
under a written commitment a human actually reviewed
(ADR-0015).
Added
- API freeze (ADR-0015, human-reviewed 2026-07-19). Frozen: the YAML
config schema (incl. thepeer_groups/spatial_relationships/
graph_relationshipsplanner-pass blocks), theFeaturizerpublic
surface and return shapes, the ADR-0007 output-naming contract (incl.
63-byte capping), the opt-in imputation contract, and the ADR-0001/0014
φ-bridge contract. Not frozen: planner/renderer internals, CTE names,
SQL text, module layout. Semver + a ≥-one-minor loguru deprecation policy
in CONTRIBUTING; classifier →Development Status :: 5 - Production/Stable. - CI compatibility matrix. The DB-free tier runs on Python
3.10/3.11/3.12/3.13; the integration tier executes the generated SQL on
PostgreSQL 14/16/17. README + FAQ state the tested matrix. to_tablesheap-row-width pre-flight. A ~1,000+-column group that
SELECTs fine used to fail CTAS withrow is too big(a heap tuple must
fit one 8 KiB page).to_tablesnow estimates each group's row width and
re-partitions with a heap-safe cap — more, narrower tables instead of a
crash; SELECT/fetch paths unchanged.- Docs snapshot per release.
release.ymlbuilds the docs site and
attachesdocs-site-vX.Y.Z.tar.gzto every release — versioned docs with
zero standing infrastructure (the starlight-versions switcher was
evaluated and rejected: it would snapshot generated, gitignored content). - Committed benchmark harnesses.
benchmarks/final_matrix.py(the
3-DB × 3-variant live matrix; earlier snapshots came from uncommitted
scripts),benchmarks/bridge_workloads.py(graph/centrality/text at
scale),benchmarks/render_v100_pages.py(artifact pages as a pure
function of the committed raw JSON).
Fixed
- Manifest-under-sharding guard. The silent
group_000fallback in the
persisted<stem>_manifestis gone: the manifest writer now receives the
exact partition the group tables were written from, and an orphaned
column raises instead of silently mis-tagging lineage (triage joins on
feature_group). Also fixes the latent case where a config that fits one
query but partitions into >1 groups wrote tables inconsistent with the
manifest. - Imputation × materialization, proven together.
to_arrow(impute=True)
andto_dataframe(impute=True)over the oversized-child TEMP-table path
are now integration-tested end to end: counts fill 0, measures keep NULL
plus__missingindicators, and the Arrow and pandas paths agree
value-for-value.
Validated
- Live 3-DB revalidation (committed:
specs/live-db-revalidation-v100/): no regression vs v0.8.0 — dirtyduck
all-agg/wide 7.0/60.2s (was 7.5/63.2), chicago311 5.7/47.5s (was
6.0/49.2), donorschoose 8.3/501.1s (was 7.6/470.1; wide's +6.6% tracks
its +6% feature growth at identical per-column throughput; 39,022
features / 33 shards / 0 duplicate names). - The 0.9.x families measured at scale for the first time: the native
graph_relationshipspass over 13,950 live chain edges × the full
22,169-facility cohort × 3 as-of dates in 7.8s (DEGREE hand-SQL-verified);
CentralityBridge.materialize_snapshotscheap tier 0.5s vs
include_heavy17.2s over 3 windows (the measured 34× reason heavy
metrics are opt-in);SentimentBridgeover 50,000 real inspector
comments materialized and spine-aggregated end to end (hand-SQL-verified).
Documented
- The as-of-LATERAL materialization residual stays a loud, deliberate
boundary (NotImplementedErrorwith both workarounds), now pinned by
tests and documented in the FAQ and the configuration reference. - The pyright/coverage carve-outs on the two dynamic primitive modules and
the 70% coverage floor are documented in CONTRIBUTING as intentional,
with the reason (the execution tiers are their real coverage). docs/featurizer-overview.orgretired (stale counts, DSaPP-era branding;
superseded by the docs hub's concepts/walkthrough/internals pages).
Decisions (recorded so they stop recurring)
- No PyPI, reaffirmed. Derived from dssg/featurizer and the name is
generic; GitHub releases + git-tag pins are the working distribution
channel (triage consumes them today). Revisit only on real external
demand. - No upstream dssg PR. The tree has diverged by essentially everything
(engine rewrite, 150 registry primitives, planner passes, φ-bridges, docs
hub); a PR is unreviewable. The public ccd-ia repo is the continuation.
Featurizer 0.9.1
Phase 5 of the text/graph plan: trajectory, sequence extensions, and the
Path-2 move where text induces the graph. Everything here is a φ-bridge or
edge builder — zero engine change.
Added
EmbeddingTrajectoryBridge(featurizer/bridge/trajectory.py, numpy
only): per-eventnovelty(1 − max cosine to the entity's own
strictly-prior embeddings — "out of character?"),drift(distance to the
prior-history centroid), andvolatility(step distance to the previous
event). First events are NULL (no history ≠ maximal novelty); accepts
Python sequences, PostgreSQL arrays, or pgvector text — a materialized
SentenceEmbeddingBridgetable reads back directly.- Sequence extensions (
featurizer/bridge/changepoint.py, numpy only,
snapshot-aware per ADR-0014):ChangePointBridge— strongest mean shift in
an entity's pre-t₀ measure series (binary-segmentation score + 0–1
position);PeriodicityBridge— FFT-peak strength and dominant period of
the binned event-count series (7 with daily bins and a weekly rhythm). - Text-induced edges (
featurizer/bridge/edges.py, Path 2): a small
EdgeBridgebase whosematerialize_edgeswrites an(src, dst, ts)
table — exactly what the graph bridges and the native
graph_relationshipsstage consume.NearDuplicateEdgeBridge(MinHash/LSH
via datasketch; an edge between the entities of near-duplicate documents,
knowable at the later document's timestamp; self-copies excluded) and
CoMentionEdgeBridge(names mentioned together per document; naive
built-in extractor,extract=pluggable). The two-stage
text→edges→centrality→spine wiring is integration-tested end to end and
documented in the bridge cookbook. - Shared bridge plumbing (
load_rows,fit_slice,create_table_sql,
value_sql_type) promoted to public module functions in
featurizer/bridge/base.py;word_tokenspublic in.nlp. Behaviour
unchanged. - Deps:
datasketchjoins the[bridge]extra and the dev group (the
near-duplicate tests execute under plainuv sync). - Tests: 24 new DB-free (planted outlier / step / weekly-rhythm / copy-paste
signals all recovered; per-entity and strictly-prior history isolation) +
the live-PG two-stage pipeline test.
Featurizer 0.9.0
The text/graph feature-family release (plan:
specs/incorporating-text-graph-feature-families.html): the taxonomy's
[GAP] substrates become shipped φ-bridge families, enabled by an additive
bridge-contract extension (ADR-0014), plus one deliberate engine addition —
the native 1-hop graph_relationships planner pass. Trajectory / sequence /
text-induced-edge families are the 0.9.1 line.
Added
- Bridge contract extensions (ADR-0014, all additive) —
MultiColumnBridge
(compute() → {pk: {col: val}}: one expensive pass emits N declared value
columns, with per-column variable types incl. categorical);
temporal snapshot sequences (compute_snapshots/
materialize_snapshots: rebuild the model per as-of window on the pre-t₀
slice, asserted per window, output keyed(entity, as_of_date)as an
ordinary event stream — O(windows × build) by design);materialize_nodes
(per-entity output for bridges whose compute keys by node);persist=
(real table for orchestrated assets vs the default session-temporary);
andmodel_vintage+assert_model_vintage(pretrained-model training
cutoff as declarable, assertable metadata —assert_pre_t0guards fitted
models only). The single-column contract is regression-proven byte-identical. - Text Path-1 bridges (
featurizer/bridge/nlp.py, multilingual by
default — Spanish register, never silent English):SentimentBridge
(lexicon valence, built-in es/en/xx starter lexicons, pluggablelexicon=),
ReadabilityBridge(Fernández-Huerta / Flesch),LanguageIdBridge
(stopword-profile detection, categorical output) — all three
dependency-free — andNERCountsBridge(one spaCy parse → persons / orgs /
locations / money / dates via the multi-column contract; carries
model_vintage). - Graph bridges:
CentralityBridge(one networkx build → degree / in /
out / weighted, coreness, clustering by default; betweenness, eigenvector,
closeness opt-in viainclude_heavy=so configs never get silently
slower; snapshot-aware) andCommunityBridge(Louvain membership as a
categorical column + modularity; SBM/MDL-surprise deferred — graph-tool is
not pip-installable). - Native 1-hop graph pass (the one engine change): a top-level
graph_relationshipsconfig block — edge table with requiredtimestamp,
optional neighbour-state entity,measures/sharesdefaults from
declared variable types — generatingDEGREE(<name>)(+ one windowed
variant per configured interval) andNEIGHBOUR_MEAN/NEIGHBOUR_SHARE
columns in pure SQL, bounded by both the edge timestamp and the
neighbour state'stemporal_ix. Strictly 1-hop: 2-hop aggregation (the
canonical temporal-GNN leakage) is not offered, and validation says why.
Validation quality matches the spatial block (required keys, entity refs,
family/column typo suggestions). - Docs: bridge cookbook page (worked example per modality, the native
alternative, dependency matrix), ADR-0014 in the themed index,[GAP]→
shipped 0.9.0markers in the taxonomy doc, FAQ answer updated. - Deps:
spacyandpython-louvainjoin the[bridge]extra (spaCy models
remain separate downloads);networkx+python-louvainjoin the dev
group so the hand-computed graph tests execute under plainuv sync. - Tests: 60 new DB-free (contract shapes, hand-computed NLP and graph
values, SQL-shape guards for the native pass) and 10 new live-PG
integration tests (materialize → spine handoff per family, snapshot stream
through the spine, planted future edge and future neighbour state both
excluded).
Added — docs hub (shipped to master between 0.8.0 and this release)
-
The docs site is now a full documentation hub on Astro Starlight
(aligned with triage's docs stack; plan:specs/github-pages-docs-hub.html):
a 10-section walkthrough tutorial (every command executed during authoring),
the five tutorial notebooks rendered in-theme from their committed executed
outputs (never executed in CI), a primitives reference generated from the
live registry (count-parity tested — it cannot drift), an authored
configuration reference, the 13 ADRs with a themed index, and the changelog.
Python pre-build seamsite/gen.py(uv,docsgroup) +astro build;
site/check_links.pygates every deploy. Validation artifacts stay
pass-through, untouched, under/specs/. -
Project site on GitHub Pages (
https://ccd-ia.github.io/featurizer/):
landing page, the live-DB validation artifacts (v0.6.0 / v0.8.0), and a
FeaturizerVizgallery rendered from a live 177k-row × 272-feature
dirtyduck matrix. Deployed by.github/workflows/pages.ymlon pushes that
touchsite/,specs/, ordocs/images/. -
README: visualization gallery (6 real plots), latest-release and docs
badges; the exported Table of Contents block removed (GitHub renders its
own outline).
Fixed
plot_correlation_clustermapno longer crashes on matrices containing
constant or (near-)all-NULL features (undefined correlations made scipy's
linkage reject the distance matrix); such features are dropped with a
notice.
Featurizer 0.8.0
Sharding rework: the donorschoose wide config (~36.8k columns) — a backend
crash in every previous snapshot — now materializes live in ~8 minutes, and
every cell of the 3-DB × 3-variant live matrix is green (all-agg is seconds
everywhere). Full refreshed artifacts: specs/live-db-revalidation-v080/
(+ summary page specs/live-db-revalidation-v080.html); decision record:
ADR-0005 amendment.
Changed
-
Column-group sharding now clusters columns by dependency lineage.
_partition_columnsbuckets the target's output columns by their
source-CTE signature before bin-packing, so same-lineage columns share a
group and each companion pre-aggregation CTE is emitted/executed by the few
groups that need it instead of most of them. Measured on the donorschoose
wideconfig (27 groups, ~14.9k columns): max per-group CTE closure
979 → 287, total closure 11,338 → 2,428, duplicated companion instances
899 → 18, emitted SQL 29.2 MB → 17.4 MB. Group composition changes
(which columns share a<stem>_group_NNNtable); the feature manifest's
feature_groupcolumn remains the supported mapping, and output column
names are unchanged (ADR-0007). -
Groups are additionally bounded by a window-function budget
(max_window_fns_per_group, default 500). PostgreSQL's planning memory
for N same-spec window functions in one select list is superlinear with a
hard cliff: measured live, ~675 window columns plan in ~5s while ~1,350
OOM-killed the backend during a plainEXPLAIN(fresh connection; both
halves of the same list plan fine — count, not content). The packer closes
a group early when adding a column would exceed the budget.Net effect of the two partitioning changes, measured live on the
donorschoosewideconfig (~36.8k output columns, 3,000-row cohort) that
previously OOM-killed the backend: materializes end-to-end in ~8 minutes
(32 groups, render 26.5s + execution 461.6s), max group closure 285 CTEs,
worst per-groupEXPLAINwell under 2s.
Fixed
-
Sharded re-join no longer collides on carried identifier columns. A
target that carries relationship keys beyond its id (donorschoose's
schoolid/teacher_acctid) repeats them in every group query;
to_dataframemerged groups on(as_of_date, id)only, so pandas raised
MergeError: duplicate columnsat the third group. The materialized path
now merges on the fullGroupedQueries.key_columnstuple. -
Sharded group queries no longer carry dead companion CTEs. Per-group
reachability now scans the pruned rendering of each target-level agg CTE
instead of its full-width body, so companion pre-aggregation CTEs whose only
consumer columns landed in other groups are no longer emitted. PostgreSQL 16
discards unreferenced CTEs at negligible planning cost (measured), so this
does not change plan shape — it shrinks the emitted SQL, parse time, and
render time on wide sharded configs.
Added
- Pre-flight plan-size guardrail.
ColumnGroupSharder.plan_size_report()
maps each column group to its live CTE-closure size, andwarn_plan_size()
(wired into every grouped path) logs one loud, actionable warning when any
group's closure predicts a PostgreSQL planner blowup — the failure mode
diagnosed on the donorschoosewideconfig, where ~1000-CTE group queries
took 30–45s of planning each and OOM-killed the backend during a plain
EXPLAIN. The warning names the worst groups and the config levers
(transformers / intervals / entities) instead of letting the run die
minutes later with "server closed the connection unexpectedly".
Featurizer 0.7.0
Performance release: the two root causes found by EXPLAIN (ANALYZE) on the
live triage databases (correlated two-window drift → ADR-0012; no-stats
as_of_dates cardinality → ADR-0013) plus conservative planner tuning as an
executor default. Full-aggregator materialization on every live DB dropped from
10–357s to ~6–8s; values proven unchanged by the golden gate throughout.
Known issues
- The
widevariant (all 65 aggregators × 14 transformers) on the widest
configs can OOM the PostgreSQL backend during query planning. Diagnosed
on live donorschoose (2026-07-10): ~14.9k output columns shard into 27 group
queries of up to ~979 CTEs / 1.8 MB SQL each; planning a single group takes
30–45s and spikes backend memory until the kernel OOM killer fires (observed
at a plainEXPLAIN, with a 3000-row cohort — data volume is irrelevant).
Wide-everything is an extreme, atypical config; mitigation directions
(CTE-bounded sharding, TEMP-materialized shared pre-passes, per-group
connections) are recorded in the project TODO.
Changed
-
Conservative PostgreSQL planner/memory tuning is now an executor default.
Every generated query is a wide multi-way CTE join, which starves under
PostgreSQL's stockwork_memand collapse limits. The executor now issues
SET LOCAL work_mem = '64MB',join_collapse_limit = 20,
from_collapse_limit = 20(measured ~1.4× on dirtyduck all-agg; a supporting
lever on top of ADR-0012/0013).geqodeliberately stays ON — the aggressive
variant (256MB / collapse 30 / geqo off) crashed the backend by exhaustively
planning a 38-way join. The tuning is applied only to connections featurizer
opens itself: a caller'sconnection=is never touched, becauseSET LOCAL
would stay in force for the remainder of the caller's open transaction. On the
records fast path the SETs share one held connection (and transaction) with
the query; on the psycopg paths they are savepoint-isolated and best-effort,
like the ANALYZE. NewPLANNER_TUNING/tuning_statements()/
apply_planner_tuning()infeaturizer.executor; covered by
tests/test_executor_tuning.py. -
Executor ANALYZEs
as_of_datesbefore running (ADR-0013). The caller's
freshly-createdas_of_dateshas no statistics, so PostgreSQL assumed its
~2550-row default and planned the lateral-join body for the wrong cardinality —
a single Merge Join was 99% of donorschoose all-agg's runtime. The executor now
issues a best-effort, savepoint-isolatedANALYZE as_of_dateson its working
connection first, in every path (to_dataframe,to_arrow,to_tables).
donorschoose all-agg 293.6s → 7.5s, dirtyduck 27.6s → 7.0s (~40–50×); values
unchanged (ANALYZErefreshes stats, not data — golden gate passes). -
Two-window drift aggregators migrated to set-based pre-aggregation (ADR-0012).
kl_drift/wasserstein_drift, which ADR-0010 deferred as a non-goal, were the
entire cost of full-aggregator materialization on real data: liveEXPLAIN (ANALYZE)showed 9 correlatedSubPlans over the child stream atloops=18909
(kl_driftfiring on ordinary categorical columns × intervals, O(target×children)).
Rewritten as companion CTEs — recent/baseline counts viacount(*) FILTER(KL,
no self-join) and per-windowpercentile_cont … FILTER(Wasserstein). dirtyduck
all-agg 356.8s → 27.6s (~13×), all 272 features retained, values proven identical
by the golden-value gate (now 29 migratable aggregators / 232 frozen cases; P3M
cases added since drift is degenerate under P1M). Output column names unchanged
(ADR-0007). Companion-CTE budget guard 132 → 144. -
ln/log/sqrttransformers are now domain-guarded (ADR-0011). They
rendercase when x > 0 then ln(x) end(>= 0for sqrt) instead of a bare
ln(x), so an out-of-domain row becomes SQLNULLrather than aborting the
whole materialization withcannot take logarithm of a negative number. This
hard-broke any wide/all-transformer config the moment a transformer landed on a
signed feature (z-score, difference, deviation) — surfaced on the live-DBwide
variant. Output column names/labels are unchanged (ADR-0007). New
DomainGuardedTransformerbase; guards covered by
tests/primitives/test_transformations.py.
Fixed
- Companion pre-aggregation CTE name over 63 bytes emitted an invalid bare
~. A set-based companion CTE (ADR-0010) whose<child>_<family>_<interval>_preaggs_for_<target>
name exceeded PostgreSQL's 63-byte identifier limit was hash-capped by
pg_identifierwith a~separator (safe only inside quotes — output columns
are always quoted), but_build_preagg_ctestrips the quotes to interpolate
the name bare, leaving a~that PostgreSQL parses as an operator
(syntax error at or near "~"). This hard-broke the full-aggregator config on
any data with long categorical column names — invisible to the DB-free tests
and surfaced only by running the integration suite against the live
food-inspections / dirtyduck data (8 failing realistic tests). The cap
separator is now folded to_for the bare CTE identifier; CTE names are
internal-only, so the ADR-0007 output-column naming contract is untouched.
Regression guard:tests/test_preagg_shape.py::test_preagg_cte_name_over_63_bytes_is_a_valid_bare_identifier.
Featurizer 0.6.0
Set-based pre-aggregation for the correlated-subquery aggregator tier — the
performance follow-up ADR-0009 deferred. Removes the full-cohort scaling cliff
while preserving output column names (ADR-0007) and values exactly.
Added
- Set-based pre-aggregation path (ADR-0010). Each of the 27 migratable
subquery aggregators now emits one companion CTE — a single window (or
grouped-join) pre-pass over the child stream reduced by a plainGROUP BY—
instead of a scalar correlated subquery evaluated once per target row. Cost
drops fromO(target_rows × subqueries × child_scan)to oneO(N log N)pass
per family. Opt-in per aggregator viaSubqueryAggregator._build_preagg; the
companion CTE reuses the existing join / synth-pruning / sharding /
materialization machinery unchanged. - Golden-value regression harness.
tests/integration/test_preagg_value_equality.py+
tests/fixtures/preagg_golden_values.jsonfreeze the v0.5.2 correlated values
(162 cases) and assert every migrated aggregator reproduces them exactly.
tests/test_preagg_shape.pyadds DB-free companion-CTE shape guards. A
benchmarks/package (outside the wheel) measures the scaling curve.
Changed
- Advanced-aggregator full-cohort materialization is now practical. Measured
on a synthetic 10k-parent cohort, the all-aggregator matrix went from >300 s
(timeout, censored) to 2.6 s; the worst individual families improved
~150–390× (mean_deviation93.9 s → 0.24 s,trimmed_mean_1094.6 s →
0.27 s,theil70.2 s → 0.45 s). The default-active tier is unchanged. Output
column names and values are byte-/value-identical to v0.5.2 (proven by the
golden harness + the ADR-0007 name-stability snapshot). - Families migrated: gap (
gap_mean/stddev/min/max,gap_cv,burstiness),
categorical (entropy,hhi), numeric-stream (gini,mean_deviation,
theil,acf_1,variance_ratio,cosinor_amplitude_weekly,
trimmed_mean_10,median_absolute_deviation), and sequence/transition
(ngram_2_freq,ngram_3_freq,sequence_entropy,longest_streak,
state_volatility,transition_matrix_summary,rework_count,
recurrence_interval,markov_conditional_entropy,max_transition_prob,
time_in_current_state).
Not migrated (intentional)
- The special-config families keep the correlated path: predicate-driven
(first_passage_time,cross_type_latency,right_censoring_indicator),
two-window drift (kl_drift,wasserstein_drift), and spatial
(distance_travelled,radius_of_gyration,spatial_std,bbox_area). They
fire only under special config and are out of the full-cohort scope; they
migrate later only if a real workload demands it.
Featurizer 0.5.2
Advanced-aggregator hardening: full-registry execution coverage (closing the
string-shape-only blind spot), plus the runtime fixes it surfaced.
Added
- Full-registry aggregator execution coverage.
tests/integration/test_all_aggregators_execution.pynow executes every
registered aggregator on real PostgreSQL over edge-case fixtures (single-row,
constant, zero/negative, avg-zero, single-category groups; date and
timestamp temporal columns). Previously only the default-active set had
execution coverage — the advanced tier was string-shape tested only, which is
how the v0.5.1 cluster of runtime bugs slipped through. "Every registered
aggregator executes without error" is now a tested invariant.
Fixed
harmonic_meandivision-by-zero.count(x)/sum(1/x)raised on a zero
value (1/0) and on a zero denominator. Now positive-domain and guarded:
case when min(x) > 0 then count(x)/NULLIF(sum(1.0/NULLIF(x,0)),0) else null end
(NULL on the undefined non-positive domain, mirroringgeometric_mean).mean_deviationrestored as a correct two-passSubqueryAggregator
(avg(abs(x - mean))via a correlated subquery for the mean) and re-added to
the default set — it had been removed in v0.5.1 because the single-pass form
nested aggregates. Verified: MAD of[1,4,9,16]= 5.0.- Planner empty-CTE bug. A single-type aggregation set over a mixed-type
entity graph (e.g.[entropy]over a numeric-only child) emitted
select <key>, from …— a dangling comma. The planner now skips emitting the
aggs CTE (and its join) when an aggregation yields no features for a child.
Removed
z_scoreandmin_max_scaledropped from the registry. They are per-row
normalizations, not reductions — their SQL references a bare, un-grouped
column, invalid in aGROUP BYaggregate. Use thecross_entity_zscore/
cross_entity_percentiletransformers instead. (v0.5.1 had excluded them from
the default set but kept them registered; they are now fully removed.)
Featurizer 0.5.1
Transformer-family label truncation + a cluster of never-executed advanced
aggregator bugs found by stress-testing against three live datasets, plus a
one-hot cardinality guard and CI action bumps.
Added
- High-cardinality one-hot warning. Resolving a
role: categorical
vocabulary (declared list or introspectedENUM) larger than 25 values now
logs a warning: one-hot encoding emits one sparse 0/1 column per value, which
is wide and weak. The nudge is to declare a top-Nvocabulary:and let the
long tail fall into the all-zero "other". featurizer stays split-blind (it
cannot frequency/target-encode — those are fitted, train-only transforms), so
a warning on the declared/ENUM size is the right lever. Every value is still
encoded (no silent data loss).
Fixed
-
Transformer-family names now survive PostgreSQL's 63-byte identifier cap.
Every transformer (the base unary path plus the window / rolling / lag / EMA /
Holt-Winters / diff / cumulative-product / cyclical / binary / population /
CUSUM / mean-shift families) now routes its output name through
pg_identifier— a deterministic hash suffix past 63 bytes — and carries a
full untruncatedlabel. Previously these names were emitted verbatim and
silently truncated by PostgreSQL at runtime, so a long transformer-wrapped
name (e.g.ABS(patients.MEAN(visits.ABS(visits.duration_minutes)|interval=P1D))
at 68 bytes) risked collapsing into an ambiguous column and carried no
intended name for the manifest. This completes the v0.5.0 manifest-label
work, which had wired aggregations only; the manifest now maps capped
transformer columns back to their full names and populates their lineage and
descriptions. Short names stay byte-identical (the ADR-0007 name-stability
contract). -
Temporal aggregators are now type-agnostic (date and timestamp columns).
Stress-testing against three live datasets surfaced dialect bugs that only
appear when a temporal aggregation runs on a real column:event_rate/
time_spanemittedEXTRACT(EPOCH FROM max - min), invalid on adate
column (date - dateis an integer); thegap_*family /burstiness/
cross_type_latencydifferenced raw temporal values, andSTDDEV(interval)
is undefined ontimestampcolumns. All now extract epoch seconds per side
and express the result in days (EXTRACT(EPOCH FROM col)/86400.0), which
is numeric for both types and preserves the original integer-day output on
datecolumns. Verified executing on both adateand atimestampfixture. -
geometric_meanproduced invalid SQL — unbalanced parentheses (syntax
error atelse) and base-10logwhere the geometric mean needsln. Now
case when min(x) > 0 then exp(avg(ln(x))) else null end(NULL on the
undefined non-positive domain; thelnargument is guarded so the aggregate
never raises before the outer guard nulls it). -
skewness/kurtosisrewritten as pure-aggregate raw moments. They
referenced a bare, un-grouped column (invalid in theGROUP BYaggregation
CTE) and used the**operator PostgreSQL lacks. Now computed from
avg(power(x,k))andvar_pop(x)— valid SQL and statistically correct
(a normal distribution gives kurtosis 3).
Changed
-
z_score,min_max_scale,mean_deviationremoved from the default
aggregation set (still registered / requestable). The first two are per-row
normalizations, not reductions — their SQL references a bare column that is
invalid inside aGROUP BYaggregate — and are redundant with the
cross_entity_zscore/cross_entity_percentiletransformers.
mean_deviationnests aggregates (sum(abs(x - avg(x)))), forbidden by
PostgreSQL; it awaits a SubqueryAggregator rewrite. Removing them keeps a
wholesale default/wide aggregation sweep valid on real schemas. -
in_arrayremoved from the default transformer set (still registered).
Its__call__requires anan_arrayargument the planner cannot supply, so
it crashed any wholesale default/wide transform set.