Skip to content

v0.6.0

Choose a tag to compare

@github-actions github-actions released this 03 Oct 12:34
· 113 commits to main since this release
d4570dc

Changelog

Added

  • A late row reaches SQLMesh. An incremental_by_partition entity at scd: type1 may
    declare arrival_lag: (30h, 2d, 1w). Its SQLMesh model pins interval_unit 'day'
    and sets lookback to the lag in whole days, rounded up, so a row that lands after its
    interval ran is loaded on a later run. A cron more frequent than daily on such a model is
    refused by SQLMesh unless the caller also sets allow_partials. Declared on any other
    entity, a type 2 entity included, it is refused. dbt emits nothing for it. A project that
    declares no lag compiles as before, apart from the fingerprint header.

  • A declared key is checked, not only trusted. Every entity with a key: and no
    dedupe: gets a generated blocking audit, <entity>_key_unique: no two rows share the
    key, among the current versions on an scd: type2 entity and among all rows otherwise.
    dedupe: enforces the key by construction and removes the audit; a unique rule or an
    incremental merge does not. On SQLMesh an incremental_by_partition entity checks only
    the run's own range of keys against the whole model; every other entity, and every
    entity on dbt, audits the whole model.

    Upgrade note. A source that repeats keys now stops the build: declare dedupe: on
    the entity or fix the source. History written before the audit existed is not re-checked
    by a scheduled run: sqlmesh run audits only the range it builds, and adding an audit is
    a metadata-only change that sqlmesh plan checks over the plan's own start..end range. To
    cover it, run the audit's query once over the whole model after upgrading, or deploy the
    upgrade with a plan whose range spans the history.

Changed

  • The MetricFlow manifest carries owners. A mart's owner: reaches its semantic model
    and a metric's owner: its metric, both as config.meta.owner; with no owner, config
    stays null. An entity's owner is not carried: an entity reaches MetricFlow only as entity
    elements inside a mart's semantic model, which carries the mart's owner. An exposure
    still emits no saved_query — a saved query is a request someone wrote, an exposure a
    consumer — and the manifest's saved_queries stays empty.

    Upgrade note. A project that declares an owner on a mart, or on a metric the
    manifest emits, compiles a different semantic_manifest.json, so its manifest hash
    moves; a metric MetricFlow cannot serve is omitted, owner and all.

  • bloomery_ir_version is 26 (it was 25 in 0.5.0). EntityIR carries the declared
    arrival lag, None where none is declared, so every project's fingerprint moves, and with
    it the header of every artifact that carries one. An IR written by 0.5.0 is refused by
    plan() and ir_from_json with the version mismatch message; recompile the project.

Fixed

  • A replayed parent brings its children back. When a change lets a quarantined parent's
    rows back in, bloomery plan also names in replay_scope every child that quarantined
    on it before the change, parents first, with a cycle of referential rules broken at its first
    member by name; a child that rewrote or flagged its orphans
    (on_missing: unknown_member or flag) is named in backfill_scope. Before, the child
    stayed orphaned and a blocking coverage: check on the parent failed every run. Rows
    bronze no longer holds keep their rewrite; the quality docs name the remedy.

  • A fact from an upstream project grades as imported. The evidence guard now reads the
    composed project, so a strict (requires_evidence: locked) mart over an imported base,
    a strict mart listing an imported metric, and a strict exposure naming an imported mart
    are refused, naming the upstream alias and fingerprint. The fix is
    requires_evidence: assumed on the consumer, or carrying the requirement on the
    upstream's mart.

    Upgrade note. A strict consumer resting on an imported fact compiled before and is
    refused now.

  • A dbt replay runs only against the project it was compiled with. dbt_project.yml
    carries vars: {bloomery_fingerprint: ...}, and each replay macro refuses before its
    first statement when that var differs from the fingerprint it was emitted under, or when
    the entity or its reject table lacks a column the statements write. The refusal names
    the missing columns and the rebuild that adds them: dbt run --full-refresh for an
    incremental model, which a plain run never widens. A fingerprint mismatch is fixed by
    compiling again with bloomery into a clean directory, not by a rebuild.

  • A schema change fails loudly on dbt. Every incremental dbt model, an entity's and its
    reject table's, carries on_schema_change='fail', so a field added, dropped or renamed
    stops the build instead of leaving the table silently without the column. bloomery plan names the fix beside each such field change: dbt build --full-refresh -s over the
    relations that fail, which are the entity when it is an incremental type 1 model and its
    <entity>__reject table whenever it quarantines. A full table rebuilds whole and a
    type 2 snapshot widens itself, so neither is named. The refresh recomputes history as
    SQLMesh's rebuild does; a refreshed reject table loses its resolved rows, as the dbt
    how-to says.

    Upgrade note. On dbt, a field change on an incremental entity, or on any entity that
    quarantines, now needs the full refresh the plan names.

  • A partition transform builds on Trino. The SQLMesh model spells days(...),
    months(...), years(...) and hours(...) in Trino's singular form (day, month,
    year, hour), which Trino's Iceberg connector requires; it refused the plural. A
    Hive catalog takes no transform at all: partition by a date column there. Measured on
    Trino 483: a daily interval over a coarse partition overwrites nothing outside its own
    range, so no interval unit is derived.

  • A mapped vector is refused at the field. A mapping key or field, recipe or macro aimed
    at a vector(...) column is refused at type-check with its path. It reached emit and
    failed there with no address; a vector comes from a step's declared output.

  • The fuzz lane's first findings are closed. A refused spec no longer keeps its whole
    validation graph alive, which grew a long-running caller's memory with every refusal;
    WHILE*2 and IF a, which sqlglot parses as a statement or gives up on as a Command,
    are refused as SQL text; and authored SQL text nests at most 32 levels, refused at load
    for a spec field or a quality rule's expression, and at compile for a step body, a
    registered macro body or the expression a chain of macros composes, instead of
    overflowing the stack at a later re-parse.

    Upgrade note. SQL text nested deeper than 32 levels, or that only parses as a
    statement, is refused now.