Skip to content

Releases: ccd-ia/featurizer

Featurizer 1.1.0

Choose a tag to compare

@github-actions github-actions released this 16 Aug 04:45

Additive only (semver MINOR under the ADR-0015 freeze). No existing surface
changed, no columns renamed, nothing new persisted — downstream feature
caches are unaffected, so consumers need no coordination beyond an optional
pin bump.

Added

  • Featurizer.columns_matching(pattern) / Featurizer.manifest_matching(pattern)
    — select feature columns by globbing the manifest's full, untruncated
    label. columns_matching returns physical column names in output order
    (what goes in a select list or a feature-group definition);
    manifest_matching returns the full ManifestEntry rows for labels,
    lineage and intervals.

    This is the additive answer to a request that kept arriving as "reshape the
    63-byte truncation so globs match the readable tail". That reshape was
    measured and rejected — it is a regression, see
    .out-of-scope/tail-preserving-truncation.md. The problem it was aimed at is
    real, though: a glob written against physical names misses every truncated
    column. On the sample config *frecuencia_cardiaca* resolves 672 columns
    against label but only 198 against column. Downstream that is not silent
    data loss — triage's explicit feature_groups.definitions path raises on the
    unmatched columns — but the advice it gives ("add a glob or widen one") is
    unactionable against an unpredictable hash. Resolving the group through
    columns_matching is the way out.

  • featurizer.manifest.glob_to_like(pattern) — translates a glob into a
    SQL LIKE pattern plus its escape character, for querying the persisted
    "<schema>"."<stem>_manifest" table with the same syntax. It exists because
    _ is a literal in a glob but a single-character wildcard in LIKE, and
    ~95% of real feature labels contain one; a hand-rolled translation
    over-matches silently.

  • featurizer.manifest.filter_manifest(entries, pattern) — the underlying
    free function, for manifests obtained from anywhere.

Behaviour to know about

  • A pattern matching nothing raises LookupError, it does not return [].
    A silently empty selection is the failure mode the helper exists to prevent,
    so the default is loud; the error carries near-miss suggestions (found by
    literal-fragment backoff) and, when the pattern looks aimed at a truncated
    physical name, says so explicitly. Pass allow_empty=True to opt out when
    probing for optional features.
  • Matching is case-sensitive (fnmatch.fnmatchcase, never fnmatch, which
    is case-insensitive on Windows) and always against label, never column
    or definitiondefinition inherits the parent's truncation in 234 of the
    sample config's 1,217 truncated columns and drops the interval, so windowed
    siblings share one.

Featurizer 1.0.1

Choose a tag to compare

@github-actions github-actions released this 08 Aug 03:17

Pure bug-fix release (semver PATCH under the ADR-0015 freeze). No feature
names change for any config that does not use the empty-list spelling —
the SQL/naming snapshot test is byte-identical.

Fixed

  • transformations: [] / aggregations: [] were silently swallowed.
    The primitive-selection sites used config.get(...) or DEFAULT, which
    treats an explicit empty list as "unset": a config writing
    transformations: [] to suppress the transform layer silently got all
    17 defaults (reported live from triage-pg — ~2,500 unwanted
    CUM_SUM/ABS-composition columns). An empty list now suppresses that
    layer: transformations: [] passes features through unchanged
    (byte-identical output to the [identity] workaround spelling, which
    keeps working), and aggregations: [] builds zero aggregation features
    (legal-but-weird by decision — the planner already emits no CTE/join for
    a zero-feature relationship). Absent or null keys keep applying the
    curated defaults. Covered DB-free and with executing integration tests;
    documented in the configuration reference.
    Downstream note: triage-pg pins the engine version into artifact
    identity (its ADR-0016), so its pin bump after this release
    intentionally invalidates feature caches — no coordination needed; no
    columns are renamed.

Featurizer 1.0.0

Choose a tag to compare

@github-actions github-actions released this 19 Jul 07:36

The stability release: no new feature families — 1.0.0 is the right to
rely
on what 0.9.x already shipped. Every claimed compatibility is now
tested, every known sharp edge is guarded or loudly documented, the 0.9.x
families are validated at realistic scale, and the public surface is frozen
under a written commitment a human actually reviewed
(ADR-0015).

Added

  • API freeze (ADR-0015, human-reviewed 2026-07-19). Frozen: the YAML
    config schema (incl. the peer_groups / spatial_relationships /
    graph_relationships planner-pass blocks), the Featurizer public
    surface and return shapes, the ADR-0007 output-naming contract (incl.
    63-byte capping), the opt-in imputation contract, and the ADR-0001/0014
    φ-bridge contract. Not frozen: planner/renderer internals, CTE names,
    SQL text, module layout. Semver + a ≥-one-minor loguru deprecation policy
    in CONTRIBUTING; classifier → Development Status :: 5 - Production/Stable.
  • CI compatibility matrix. The DB-free tier runs on Python
    3.10/3.11/3.12/3.13; the integration tier executes the generated SQL on
    PostgreSQL 14/16/17. README + FAQ state the tested matrix.
  • to_tables heap-row-width pre-flight. A ~1,000+-column group that
    SELECTs fine used to fail CTAS with row is too big (a heap tuple must
    fit one 8 KiB page). to_tables now estimates each group's row width and
    re-partitions with a heap-safe cap — more, narrower tables instead of a
    crash; SELECT/fetch paths unchanged.
  • Docs snapshot per release. release.yml builds the docs site and
    attaches docs-site-vX.Y.Z.tar.gz to every release — versioned docs with
    zero standing infrastructure (the starlight-versions switcher was
    evaluated and rejected: it would snapshot generated, gitignored content).
  • Committed benchmark harnesses. benchmarks/final_matrix.py (the
    3-DB × 3-variant live matrix; earlier snapshots came from uncommitted
    scripts), benchmarks/bridge_workloads.py (graph/centrality/text at
    scale), benchmarks/render_v100_pages.py (artifact pages as a pure
    function of the committed raw JSON).

Fixed

  • Manifest-under-sharding guard. The silent group_000 fallback in the
    persisted <stem>_manifest is gone: the manifest writer now receives the
    exact partition the group tables were written from, and an orphaned
    column raises instead of silently mis-tagging lineage (triage joins on
    feature_group). Also fixes the latent case where a config that fits one
    query but partitions into >1 groups wrote tables inconsistent with the
    manifest.
  • Imputation × materialization, proven together. to_arrow(impute=True)
    and to_dataframe(impute=True) over the oversized-child TEMP-table path
    are now integration-tested end to end: counts fill 0, measures keep NULL
    plus __missing indicators, and the Arrow and pandas paths agree
    value-for-value.

Validated

  • Live 3-DB revalidation (committed:
    specs/live-db-revalidation-v100/): no regression vs v0.8.0 — dirtyduck
    all-agg/wide 7.0/60.2s (was 7.5/63.2), chicago311 5.7/47.5s (was
    6.0/49.2), donorschoose 8.3/501.1s (was 7.6/470.1; wide's +6.6% tracks
    its +6% feature growth at identical per-column throughput; 39,022
    features / 33 shards / 0 duplicate names).
  • The 0.9.x families measured at scale for the first time: the native
    graph_relationships pass over 13,950 live chain edges × the full
    22,169-facility cohort × 3 as-of dates in 7.8s (DEGREE hand-SQL-verified);
    CentralityBridge.materialize_snapshots cheap tier 0.5s vs
    include_heavy 17.2s over 3 windows (the measured 34× reason heavy
    metrics are opt-in); SentimentBridge over 50,000 real inspector
    comments materialized and spine-aggregated end to end (hand-SQL-verified).

Documented

  • The as-of-LATERAL materialization residual stays a loud, deliberate
    boundary
    (NotImplementedError with both workarounds), now pinned by
    tests and documented in the FAQ and the configuration reference.
  • The pyright/coverage carve-outs on the two dynamic primitive modules and
    the 70% coverage floor are documented in CONTRIBUTING as intentional,
    with the reason (the execution tiers are their real coverage).
  • docs/featurizer-overview.org retired (stale counts, DSaPP-era branding;
    superseded by the docs hub's concepts/walkthrough/internals pages).

Decisions (recorded so they stop recurring)

  • No PyPI, reaffirmed. Derived from dssg/featurizer and the name is
    generic; GitHub releases + git-tag pins are the working distribution
    channel (triage consumes them today). Revisit only on real external
    demand.
  • No upstream dssg PR. The tree has diverged by essentially everything
    (engine rewrite, 150 registry primitives, planner passes, φ-bridges, docs
    hub); a PR is unreviewable. The public ccd-ia repo is the continuation.

Featurizer 0.9.1

Choose a tag to compare

@github-actions github-actions released this 18 Jul 07:15

Phase 5 of the text/graph plan: trajectory, sequence extensions, and the
Path-2 move where text induces the graph. Everything here is a φ-bridge or
edge builder — zero engine change.

Added

  • EmbeddingTrajectoryBridge (featurizer/bridge/trajectory.py, numpy
    only): per-event novelty (1 − max cosine to the entity's own
    strictly-prior embeddings — "out of character?"), drift (distance to the
    prior-history centroid), and volatility (step distance to the previous
    event). First events are NULL (no history ≠ maximal novelty); accepts
    Python sequences, PostgreSQL arrays, or pgvector text — a materialized
    SentenceEmbeddingBridge table reads back directly.
  • Sequence extensions (featurizer/bridge/changepoint.py, numpy only,
    snapshot-aware per ADR-0014): ChangePointBridge — strongest mean shift in
    an entity's pre-t₀ measure series (binary-segmentation score + 0–1
    position); PeriodicityBridge — FFT-peak strength and dominant period of
    the binned event-count series (7 with daily bins and a weekly rhythm).
  • Text-induced edges (featurizer/bridge/edges.py, Path 2): a small
    EdgeBridge base whose materialize_edges writes an (src, dst, ts)
    table — exactly what the graph bridges and the native
    graph_relationships stage consume. NearDuplicateEdgeBridge (MinHash/LSH
    via datasketch; an edge between the entities of near-duplicate documents,
    knowable at the later document's timestamp; self-copies excluded) and
    CoMentionEdgeBridge (names mentioned together per document; naive
    built-in extractor, extract= pluggable). The two-stage
    text→edges→centrality→spine wiring is integration-tested end to end and
    documented in the bridge cookbook.
  • Shared bridge plumbing (load_rows, fit_slice, create_table_sql,
    value_sql_type) promoted to public module functions in
    featurizer/bridge/base.py; word_tokens public in .nlp. Behaviour
    unchanged.
  • Deps: datasketch joins the [bridge] extra and the dev group (the
    near-duplicate tests execute under plain uv sync).
  • Tests: 24 new DB-free (planted outlier / step / weekly-rhythm / copy-paste
    signals all recovered; per-entity and strictly-prior history isolation) +
    the live-PG two-stage pipeline test.

Featurizer 0.9.0

Choose a tag to compare

@github-actions github-actions released this 17 Jul 21:44

The text/graph feature-family release (plan:
specs/incorporating-text-graph-feature-families.html): the taxonomy's
[GAP] substrates become shipped φ-bridge families, enabled by an additive
bridge-contract extension (ADR-0014), plus one deliberate engine addition —
the native 1-hop graph_relationships planner pass. Trajectory / sequence /
text-induced-edge families are the 0.9.1 line.

Added

  • Bridge contract extensions (ADR-0014, all additive)MultiColumnBridge
    (compute() → {pk: {col: val}}: one expensive pass emits N declared value
    columns, with per-column variable types incl. categorical);
    temporal snapshot sequences (compute_snapshots /
    materialize_snapshots: rebuild the model per as-of window on the pre-t₀
    slice, asserted per window, output keyed (entity, as_of_date) as an
    ordinary event stream — O(windows × build) by design); materialize_nodes
    (per-entity output for bridges whose compute keys by node); persist=
    (real table for orchestrated assets vs the default session-temporary);
    and model_vintage + assert_model_vintage (pretrained-model training
    cutoff as declarable, assertable metadata — assert_pre_t0 guards fitted
    models only). The single-column contract is regression-proven byte-identical.
  • Text Path-1 bridges (featurizer/bridge/nlp.py, multilingual by
    default — Spanish register, never silent English): SentimentBridge
    (lexicon valence, built-in es/en/xx starter lexicons, pluggable lexicon=),
    ReadabilityBridge (Fernández-Huerta / Flesch), LanguageIdBridge
    (stopword-profile detection, categorical output) — all three
    dependency-free — and NERCountsBridge (one spaCy parse → persons / orgs /
    locations / money / dates via the multi-column contract; carries
    model_vintage).
  • Graph bridges: CentralityBridge (one networkx build → degree / in /
    out / weighted, coreness, clustering by default; betweenness, eigenvector,
    closeness opt-in via include_heavy= so configs never get silently
    slower; snapshot-aware) and CommunityBridge (Louvain membership as a
    categorical column + modularity; SBM/MDL-surprise deferred — graph-tool is
    not pip-installable).
  • Native 1-hop graph pass (the one engine change): a top-level
    graph_relationships config block — edge table with required timestamp,
    optional neighbour-state entity, measures / shares defaults from
    declared variable types — generating DEGREE(<name>) (+ one windowed
    variant per configured interval) and NEIGHBOUR_MEAN / NEIGHBOUR_SHARE
    columns in pure SQL, bounded by both the edge timestamp and the
    neighbour state's temporal_ix. Strictly 1-hop: 2-hop aggregation (the
    canonical temporal-GNN leakage) is not offered, and validation says why.
    Validation quality matches the spatial block (required keys, entity refs,
    family/column typo suggestions).
  • Docs: bridge cookbook page (worked example per modality, the native
    alternative, dependency matrix), ADR-0014 in the themed index, [GAP]
    shipped 0.9.0 markers in the taxonomy doc, FAQ answer updated.
  • Deps: spacy and python-louvain join the [bridge] extra (spaCy models
    remain separate downloads); networkx + python-louvain join the dev
    group so the hand-computed graph tests execute under plain uv sync.
  • Tests: 60 new DB-free (contract shapes, hand-computed NLP and graph
    values, SQL-shape guards for the native pass) and 10 new live-PG
    integration tests (materialize → spine handoff per family, snapshot stream
    through the spine, planted future edge and future neighbour state both
    excluded).

Added — docs hub (shipped to master between 0.8.0 and this release)

  • The docs site is now a full documentation hub on Astro Starlight
    (aligned with triage's docs stack; plan: specs/github-pages-docs-hub.html):
    a 10-section walkthrough tutorial (every command executed during authoring),
    the five tutorial notebooks rendered in-theme from their committed executed
    outputs (never executed in CI), a primitives reference generated from the
    live registry (count-parity tested — it cannot drift), an authored
    configuration reference, the 13 ADRs with a themed index, and the changelog.
    Python pre-build seam site/gen.py (uv, docs group) + astro build;
    site/check_links.py gates every deploy. Validation artifacts stay
    pass-through, untouched, under /specs/.

  • Project site on GitHub Pages (https://ccd-ia.github.io/featurizer/):
    landing page, the live-DB validation artifacts (v0.6.0 / v0.8.0), and a
    FeaturizerViz gallery rendered from a live 177k-row × 272-feature
    dirtyduck matrix. Deployed by .github/workflows/pages.yml on pushes that
    touch site/, specs/, or docs/images/.

  • README: visualization gallery (6 real plots), latest-release and docs
    badges; the exported Table of Contents block removed (GitHub renders its
    own outline).

Fixed

  • plot_correlation_clustermap no longer crashes on matrices containing
    constant or (near-)all-NULL features (undefined correlations made scipy's
    linkage reject the distance matrix); such features are dropped with a
    notice.

Featurizer 0.8.0

Choose a tag to compare

@github-actions github-actions released this 12 Jul 06:19

Sharding rework: the donorschoose wide config (~36.8k columns) — a backend
crash in every previous snapshot — now materializes live in ~8 minutes, and
every cell of the 3-DB × 3-variant live matrix is green (all-agg is seconds
everywhere). Full refreshed artifacts: specs/live-db-revalidation-v080/
(+ summary page specs/live-db-revalidation-v080.html); decision record:
ADR-0005 amendment.

Changed

  • Column-group sharding now clusters columns by dependency lineage.
    _partition_columns buckets the target's output columns by their
    source-CTE signature before bin-packing, so same-lineage columns share a
    group and each companion pre-aggregation CTE is emitted/executed by the few
    groups that need it instead of most of them. Measured on the donorschoose
    wide config (27 groups, ~14.9k columns): max per-group CTE closure
    979 → 287, total closure 11,338 → 2,428, duplicated companion instances
    899 → 18, emitted SQL 29.2 MB → 17.4 MB. Group composition changes
    (which columns share a <stem>_group_NNN table); the feature manifest's
    feature_group column remains the supported mapping, and output column
    names are unchanged (ADR-0007).

  • Groups are additionally bounded by a window-function budget
    (max_window_fns_per_group, default 500). PostgreSQL's planning memory
    for N same-spec window functions in one select list is superlinear with a
    hard cliff: measured live, ~675 window columns plan in ~5s while ~1,350
    OOM-killed the backend during a plain EXPLAIN (fresh connection; both
    halves of the same list plan fine — count, not content). The packer closes
    a group early when adding a column would exceed the budget.

    Net effect of the two partitioning changes, measured live on the
    donorschoose wide config (~36.8k output columns, 3,000-row cohort) that
    previously OOM-killed the backend: materializes end-to-end in ~8 minutes
    (32 groups, render 26.5s + execution 461.6s), max group closure 285 CTEs,
    worst per-group EXPLAIN well under 2s.

Fixed

  • Sharded re-join no longer collides on carried identifier columns. A
    target that carries relationship keys beyond its id (donorschoose's
    schoolid / teacher_acctid) repeats them in every group query;
    to_dataframe merged groups on (as_of_date, id) only, so pandas raised
    MergeError: duplicate columns at the third group. The materialized path
    now merges on the full GroupedQueries.key_columns tuple.

  • Sharded group queries no longer carry dead companion CTEs. Per-group
    reachability now scans the pruned rendering of each target-level agg CTE
    instead of its full-width body, so companion pre-aggregation CTEs whose only
    consumer columns landed in other groups are no longer emitted. PostgreSQL 16
    discards unreferenced CTEs at negligible planning cost (measured), so this
    does not change plan shape — it shrinks the emitted SQL, parse time, and
    render time on wide sharded configs.

Added

  • Pre-flight plan-size guardrail. ColumnGroupSharder.plan_size_report()
    maps each column group to its live CTE-closure size, and warn_plan_size()
    (wired into every grouped path) logs one loud, actionable warning when any
    group's closure predicts a PostgreSQL planner blowup — the failure mode
    diagnosed on the donorschoose wide config, where ~1000-CTE group queries
    took 30–45s of planning each and OOM-killed the backend during a plain
    EXPLAIN. The warning names the worst groups and the config levers
    (transformers / intervals / entities) instead of letting the run die
    minutes later with "server closed the connection unexpectedly".

Featurizer 0.7.0

Choose a tag to compare

@github-actions github-actions released this 10 Jul 15:35

Performance release: the two root causes found by EXPLAIN (ANALYZE) on the
live triage databases (correlated two-window drift → ADR-0012; no-stats
as_of_dates cardinality → ADR-0013) plus conservative planner tuning as an
executor default. Full-aggregator materialization on every live DB dropped from
10–357s to ~6–8s; values proven unchanged by the golden gate throughout.

Known issues

  • The wide variant (all 65 aggregators × 14 transformers) on the widest
    configs can OOM the PostgreSQL backend during query planning.
    Diagnosed
    on live donorschoose (2026-07-10): ~14.9k output columns shard into 27 group
    queries of up to ~979 CTEs / 1.8 MB SQL each; planning a single group takes
    30–45s and spikes backend memory until the kernel OOM killer fires (observed
    at a plain EXPLAIN, with a 3000-row cohort — data volume is irrelevant).
    Wide-everything is an extreme, atypical config; mitigation directions
    (CTE-bounded sharding, TEMP-materialized shared pre-passes, per-group
    connections) are recorded in the project TODO.

Changed

  • Conservative PostgreSQL planner/memory tuning is now an executor default.
    Every generated query is a wide multi-way CTE join, which starves under
    PostgreSQL's stock work_mem and collapse limits. The executor now issues
    SET LOCAL work_mem = '64MB', join_collapse_limit = 20,
    from_collapse_limit = 20 (measured ~1.4× on dirtyduck all-agg; a supporting
    lever on top of ADR-0012/0013). geqo deliberately stays ON — the aggressive
    variant (256MB / collapse 30 / geqo off) crashed the backend by exhaustively
    planning a 38-way join. The tuning is applied only to connections featurizer
    opens itself
    : a caller's connection= is never touched, because SET LOCAL
    would stay in force for the remainder of the caller's open transaction. On the
    records fast path the SETs share one held connection (and transaction) with
    the query; on the psycopg paths they are savepoint-isolated and best-effort,
    like the ANALYZE. New PLANNER_TUNING / tuning_statements() /
    apply_planner_tuning() in featurizer.executor; covered by
    tests/test_executor_tuning.py.

  • Executor ANALYZEs as_of_dates before running (ADR-0013). The caller's
    freshly-created as_of_dates has no statistics, so PostgreSQL assumed its
    ~2550-row default and planned the lateral-join body for the wrong cardinality —
    a single Merge Join was 99% of donorschoose all-agg's runtime. The executor now
    issues a best-effort, savepoint-isolated ANALYZE as_of_dates on its working
    connection first, in every path (to_dataframe, to_arrow, to_tables).
    donorschoose all-agg 293.6s → 7.5s, dirtyduck 27.6s → 7.0s (~40–50×); values
    unchanged (ANALYZE refreshes stats, not data — golden gate passes).

  • Two-window drift aggregators migrated to set-based pre-aggregation (ADR-0012).
    kl_drift / wasserstein_drift, which ADR-0010 deferred as a non-goal, were the
    entire cost of full-aggregator materialization on real data: live EXPLAIN (ANALYZE) showed 9 correlated SubPlans over the child stream at loops=18909
    (kl_drift firing on ordinary categorical columns × intervals, O(target×children)).
    Rewritten as companion CTEs — recent/baseline counts via count(*) FILTER (KL,
    no self-join) and per-window percentile_cont … FILTER (Wasserstein). dirtyduck
    all-agg 356.8s → 27.6s (~13×)
    , all 272 features retained, values proven identical
    by the golden-value gate (now 29 migratable aggregators / 232 frozen cases; P3M
    cases added since drift is degenerate under P1M). Output column names unchanged
    (ADR-0007). Companion-CTE budget guard 132 → 144.

  • ln / log / sqrt transformers are now domain-guarded (ADR-0011). They
    render case when x > 0 then ln(x) end (>= 0 for sqrt) instead of a bare
    ln(x), so an out-of-domain row becomes SQL NULL rather than aborting the
    whole materialization with cannot take logarithm of a negative number. This
    hard-broke any wide/all-transformer config the moment a transformer landed on a
    signed feature (z-score, difference, deviation) — surfaced on the live-DB wide
    variant. Output column names/labels are unchanged (ADR-0007). New
    DomainGuardedTransformer base; guards covered by
    tests/primitives/test_transformations.py.

Fixed

  • Companion pre-aggregation CTE name over 63 bytes emitted an invalid bare
    ~.
    A set-based companion CTE (ADR-0010) whose <child>_<family>_<interval>_preaggs_for_<target>
    name exceeded PostgreSQL's 63-byte identifier limit was hash-capped by
    pg_identifier with a ~ separator (safe only inside quotes — output columns
    are always quoted), but _build_preagg_cte strips the quotes to interpolate
    the name bare, leaving a ~ that PostgreSQL parses as an operator
    (syntax error at or near "~"). This hard-broke the full-aggregator config on
    any data with long categorical column names — invisible to the DB-free tests
    and surfaced only by running the integration suite against the live
    food-inspections / dirtyduck data (8 failing realistic tests). The cap
    separator is now folded to _ for the bare CTE identifier; CTE names are
    internal-only, so the ADR-0007 output-column naming contract is untouched.
    Regression guard: tests/test_preagg_shape.py::test_preagg_cte_name_over_63_bytes_is_a_valid_bare_identifier.

Featurizer 0.6.0

Choose a tag to compare

@nanounanue nanounanue released this 10 Jul 19:04

Set-based pre-aggregation for the correlated-subquery aggregator tier — the
performance follow-up ADR-0009 deferred. Removes the full-cohort scaling cliff
while preserving output column names (ADR-0007) and values exactly.

Added

  • Set-based pre-aggregation path (ADR-0010). Each of the 27 migratable
    subquery aggregators now emits one companion CTE — a single window (or
    grouped-join) pre-pass over the child stream reduced by a plain GROUP BY
    instead of a scalar correlated subquery evaluated once per target row. Cost
    drops from O(target_rows × subqueries × child_scan) to one O(N log N) pass
    per family. Opt-in per aggregator via SubqueryAggregator._build_preagg; the
    companion CTE reuses the existing join / synth-pruning / sharding /
    materialization machinery unchanged.
  • Golden-value regression harness.
    tests/integration/test_preagg_value_equality.py +
    tests/fixtures/preagg_golden_values.json freeze the v0.5.2 correlated values
    (162 cases) and assert every migrated aggregator reproduces them exactly.
    tests/test_preagg_shape.py adds DB-free companion-CTE shape guards. A
    benchmarks/ package (outside the wheel) measures the scaling curve.

Changed

  • Advanced-aggregator full-cohort materialization is now practical. Measured
    on a synthetic 10k-parent cohort, the all-aggregator matrix went from >300 s
    (timeout, censored)
    to 2.6 s; the worst individual families improved
    ~150–390× (mean_deviation 93.9 s → 0.24 s, trimmed_mean_10 94.6 s →
    0.27 s, theil 70.2 s → 0.45 s). The default-active tier is unchanged. Output
    column names and values are byte-/value-identical to v0.5.2 (proven by the
    golden harness + the ADR-0007 name-stability snapshot).
  • Families migrated: gap (gap_mean/stddev/min/max, gap_cv, burstiness),
    categorical (entropy, hhi), numeric-stream (gini, mean_deviation,
    theil, acf_1, variance_ratio, cosinor_amplitude_weekly,
    trimmed_mean_10, median_absolute_deviation), and sequence/transition
    (ngram_2_freq, ngram_3_freq, sequence_entropy, longest_streak,
    state_volatility, transition_matrix_summary, rework_count,
    recurrence_interval, markov_conditional_entropy, max_transition_prob,
    time_in_current_state).

Not migrated (intentional)

  • The special-config families keep the correlated path: predicate-driven
    (first_passage_time, cross_type_latency, right_censoring_indicator),
    two-window drift (kl_drift, wasserstein_drift), and spatial
    (distance_travelled, radius_of_gyration, spatial_std, bbox_area). They
    fire only under special config and are out of the full-cohort scope; they
    migrate later only if a real workload demands it.

Featurizer 0.5.2

Choose a tag to compare

@nanounanue nanounanue released this 10 Jul 19:04

Advanced-aggregator hardening: full-registry execution coverage (closing the
string-shape-only blind spot), plus the runtime fixes it surfaced.

Added

  • Full-registry aggregator execution coverage.
    tests/integration/test_all_aggregators_execution.py now executes every
    registered aggregator on real PostgreSQL over edge-case fixtures (single-row,
    constant, zero/negative, avg-zero, single-category groups; date and
    timestamp temporal columns). Previously only the default-active set had
    execution coverage — the advanced tier was string-shape tested only, which is
    how the v0.5.1 cluster of runtime bugs slipped through. "Every registered
    aggregator executes without error" is now a tested invariant.

Fixed

  • harmonic_mean division-by-zero. count(x)/sum(1/x) raised on a zero
    value (1/0) and on a zero denominator. Now positive-domain and guarded:
    case when min(x) > 0 then count(x)/NULLIF(sum(1.0/NULLIF(x,0)),0) else null end
    (NULL on the undefined non-positive domain, mirroring geometric_mean).
  • mean_deviation restored as a correct two-pass SubqueryAggregator
    (avg(abs(x - mean)) via a correlated subquery for the mean) and re-added to
    the default set — it had been removed in v0.5.1 because the single-pass form
    nested aggregates. Verified: MAD of [1,4,9,16] = 5.0.
  • Planner empty-CTE bug. A single-type aggregation set over a mixed-type
    entity graph (e.g. [entropy] over a numeric-only child) emitted
    select <key>, from … — a dangling comma. The planner now skips emitting the
    aggs CTE (and its join) when an aggregation yields no features for a child.

Removed

  • z_score and min_max_scale dropped from the registry. They are per-row
    normalizations, not reductions — their SQL references a bare, un-grouped
    column, invalid in a GROUP BY aggregate. Use the cross_entity_zscore /
    cross_entity_percentile transformers instead. (v0.5.1 had excluded them from
    the default set but kept them registered; they are now fully removed.)

Featurizer 0.5.1

Choose a tag to compare

@nanounanue nanounanue released this 10 Jul 19:04

Transformer-family label truncation + a cluster of never-executed advanced
aggregator bugs found by stress-testing against three live datasets, plus a
one-hot cardinality guard and CI action bumps.

Added

  • High-cardinality one-hot warning. Resolving a role: categorical
    vocabulary (declared list or introspected ENUM) larger than 25 values now
    logs a warning: one-hot encoding emits one sparse 0/1 column per value, which
    is wide and weak. The nudge is to declare a top-N vocabulary: and let the
    long tail fall into the all-zero "other". featurizer stays split-blind (it
    cannot frequency/target-encode — those are fitted, train-only transforms), so
    a warning on the declared/ENUM size is the right lever. Every value is still
    encoded (no silent data loss).

Fixed

  • Transformer-family names now survive PostgreSQL's 63-byte identifier cap.
    Every transformer (the base unary path plus the window / rolling / lag / EMA /
    Holt-Winters / diff / cumulative-product / cyclical / binary / population /
    CUSUM / mean-shift families) now routes its output name through
    pg_identifier — a deterministic hash suffix past 63 bytes — and carries a
    full untruncated label. Previously these names were emitted verbatim and
    silently truncated by PostgreSQL at runtime, so a long transformer-wrapped
    name (e.g. ABS(patients.MEAN(visits.ABS(visits.duration_minutes)|interval=P1D))
    at 68 bytes) risked collapsing into an ambiguous column and carried no
    intended name for the manifest. This completes the v0.5.0 manifest-label
    work, which had wired aggregations only; the manifest now maps capped
    transformer columns back to their full names and populates their lineage and
    descriptions. Short names stay byte-identical (the ADR-0007 name-stability
    contract).

  • Temporal aggregators are now type-agnostic (date and timestamp columns).
    Stress-testing against three live datasets surfaced dialect bugs that only
    appear when a temporal aggregation runs on a real column: event_rate /
    time_span emitted EXTRACT(EPOCH FROM max - min), invalid on a date
    column (date - date is an integer); the gap_* family / burstiness /
    cross_type_latency differenced raw temporal values, and STDDEV(interval)
    is undefined on timestamp columns. All now extract epoch seconds per side
    and express the result in days (EXTRACT(EPOCH FROM col)/86400.0), which
    is numeric for both types and preserves the original integer-day output on
    date columns. Verified executing on both a date and a timestamp fixture.

  • geometric_mean produced invalid SQL — unbalanced parentheses (syntax
    error at else) and base-10 log where the geometric mean needs ln. Now
    case when min(x) > 0 then exp(avg(ln(x))) else null end (NULL on the
    undefined non-positive domain; the ln argument is guarded so the aggregate
    never raises before the outer guard nulls it).

  • skewness / kurtosis rewritten as pure-aggregate raw moments. They
    referenced a bare, un-grouped column (invalid in the GROUP BY aggregation
    CTE) and used the ** operator PostgreSQL lacks. Now computed from
    avg(power(x,k)) and var_pop(x) — valid SQL and statistically correct
    (a normal distribution gives kurtosis 3).

Changed

  • z_score, min_max_scale, mean_deviation removed from the default
    aggregation set
    (still registered / requestable). The first two are per-row
    normalizations, not reductions — their SQL references a bare column that is
    invalid inside a GROUP BY aggregate — and are redundant with the
    cross_entity_zscore / cross_entity_percentile transformers.
    mean_deviation nests aggregates (sum(abs(x - avg(x)))), forbidden by
    PostgreSQL; it awaits a SubqueryAggregator rewrite. Removing them keeps a
    wholesale default/wide aggregation sweep valid on real schemas.

  • in_array removed from the default transformer set (still registered).
    Its __call__ requires an an_array argument the planner cannot supply, so
    it crashed any wholesale default/wide transform set.