Skip to content

Releases

v0.12.0

Choose a tag to compare

@github-actions github-actions released this 06 Nov 11:34
4648f48

Changelog

Added

  • A string step parameter can list the values it takes. A manifest declares
    allocate: {type: string, values: [none, revenue, units], default: none}, and a value
    outside the list, such as allocate: revenu, is refused with a StepError that names
    the allowed values, whether a step wiring sets it or a field mapping or repair recipe
    calling a sql_macro does. Before, a misspelled value compiled and the
    step ran whichever branch its body fell through to. A manifest is refused if its
    default is not one of its values, or if a parameter that is not a string declares
    them. (#403)
  • A step input can be optional, so one step serves projects with and without a
    source.
    A sql_model declares it optional: true with typed columns:,
    and a wiring may leave it out. The body reads it by name, as the bound relation,
    which must carry those columns, or unbound as an empty relation of them. (#402)

Fixed

  • An entity field no mapping produces is a NULL column of its silver model. It was
    left out, so a step whose input requires it compiled and then failed on the engine
    (Binder Error: … does not have a column named …). The column is now typed NULL, as a
    field only some of a merged entity's sources map already was, so marts and the semantic
    layers carry it too. Three kinds stay out: a required field the one mapping omits, a
    vector field, and a field spelled as a canonical that a mapped field links, which a
    metric's filter would otherwise read as written. A step that requires one of them is
    refused at compile. (#404)

v0.11.2

Choose a tag to compare

@github-actions github-actions released this 04 Nov 21:28
8a02aa8

Changelog

Fixed

  • A spec value YAML cannot build is a SpecParseError naming its line. An unquoted
    date the calendar does not have, such as 2026-13-01, a type tag over text it cannot
    read, such as !!int "" or !!bool maybe, or a \U escape past U+10FFFF raised a
    bare Python exception out of load_project and the CLI. The refusal now names the
    document and the line, and says to quote the value and drop any !! tag if it is
    meant as text. (#398)

v0.11.1

Choose a tag to compare

@github-actions github-actions released this 02 Nov 14:01
68c5678

Changelog

Fixed

  • A transform applied to another transform's output renders for its engine. On
    PostgreSQL, a strip_suffix or regex_extract over a strip_suffix or
    regex_extract failed at run time (function ends_with(text, unknown) does not exist, and likewise regexp_extract), and Redshift failed the same way on a
    strip_suffix of a strip_suffix. On Redshift a strip_prefix of a strip_prefix
    read the inner prefix through LIKE, where _ and % are wildcards, and a
    regex_extract of a regex_extract took the inner one's whole match rather than its
    group. On Databricks the inner regex_extract lost its group index and read the
    engine's default, group 1. Each step now renders as it does alone, and so a nested
    regex_extract naming a group above 1 is refused on Redshift at compile, as a single
    one is, where it used to compile to a call that read the group as a start position.
  • regex_extract is NULL on DuckDB where the pattern does not match, as it already
    was on PostgreSQL and Trino, rather than ''. A coalesce or enum_map after the
    extract now sees the miss on DuckDB too, so a mapping that falls back to a default
    lands the same silver on both engines, and a missing reference is NULL rather than an
    id spelled ''. A capture group that takes no part in the match is NULL too, as on
    PostgreSQL and Trino, while a group that matches the empty text is still ''. Only
    the emitted SQL changes, so IR fingerprints stay the same, but DuckDB silver rebuilt
    from the same bronze moves from '' to NULL on those rows. (#392)

v0.11.0

Choose a tag to compare

@github-actions github-actions released this 01 Nov 11:03
644c921

Changelog

Added

  • A naming policy can name a relation's catalog. relation() may return
    (catalog, namespace, relation) as well as (namespace, relation), and every target
    renders the catalog as a real third part: lake.silver.order in the SQL and the SQLMesh
    model name, +database and a source's database on dbt, node_relation.database on
    MetricFlow, and Cube's sql_table. One DuckDB project can read bronze from an attached
    Postgres and write silver and gold to a DuckLake without rewriting the emitted files,
    and plans read the same relations. A policy that returns a pair compiles exactly as
    before. (#389)

v0.10.0

Choose a tag to compare

@github-actions github-actions released this 30 Oct 14:03
4b8863c

Changelog

Added

  • A key part can be a constant. key: {marketplace: {const: Ozon}, order_id: {from: …}}
    keys each mapping's rows by a value fixed for that mapping beside the source's own id,
    so two sources feeding one merged entity land as distinct rows where their ids overlap,
    and the collision audit stays silent on them. It is typed and lowered as a field's
    constant is, and a quarantined row replays with it. A mapping that sets every key
    column of the entity as a constant is refused, since every row of it would carry one
    key. (#380)

  • A plan change says what happened, and to which column, as values. Change gains
    action (added, removed, renamed, retyped or changed, the new ChangeAction
    enum), member (the column, measure, dimension column or judged field, where subject
    names the mart or entity holding it) and advice (per-target advice as
    (target, sentence) pairs). A program showing a plan in another language, or grouped by
    column, no longer reads them off detail's English, and a new mart column reported as a
    flattened column and as a dimension carries one member in both. bloomery plan --format json carries the three fields. (#383)

Changed

  • A per_unit field can be read through a chain of many-to-one relationships, not
    only one: a marketplace sale reading the product's unit cost through its item compiles,
    and each mart reads the column it flattened the cost into. A hop read one-to-many, from
    a parent to its children, anywhere in the chain still refuses it, and a refusal now
    names the chain it was reached by rather than "no declared relationship". (#382)

  • Change.detail no longer carries the dbt full-refresh advice. A field added to,
    dropped from or renamed on an incremental entity used to end its detail with ; on dbt, run …; that sentence is now in advice under dbt, so a project shipped to SQLMesh is
    not told to run a dbt command. A reader matching the old detail text must read
    advice. bloomery plan's text output still prints it after the sentence. A flattened
    mart column whose type alone moved now reads flattened column '…' retyped, with the two
    types in old/new, and a new quarantine: block reads quarantine block added;
    both read changed before. (#383)

Fixed

  • Lineage through a step no longer shows a column named after the step's output.
    Given the step registry, a step feeds only the columns its manifest produces, so a
    walk or a rendered graph no longer carries customer_profile.profile beside
    customer_profile.segment. Without a registry, as in bloomery lineage, that node
    is still the step's one out-edge, standing for the whole relation it writes. In
    timeline(), a change to a quality: rule a wiring declares on its output is now
    reported on the step's node. (#381)

  • A field one mapping reads and another sets as a constant no longer refuses the merge.
    On a merged entity in the quality system, the reading mapping's implicit coercible
    rule was taken for a quality: disagreement with the constant, which has no rule to
    agree with, and nothing either mapping declared could fix it. The rule now applies to
    the reading source's rows only. (#386)

v0.9.0

Choose a tag to compare

@github-actions github-actions released this 29 Oct 15:56
d46669d

Changelog

Added

  • Entities, marts and step-produced columns take a label:. An entity's label reaches
    the MetricFlow entity that stands for it, a mart's becomes MetricFlow's semantic model
    label and Cube's cube and view title, and a manifest's produces: column label
    reaches the output entity's column as a mapped field's does. No SQL changes, and
    changing one is metadata only in plan() and timeline(); a step whose column label
    changed is not restated. (#373)

  • A field mapping can set a constant. marketplace: {const: Wildberries} sets the field to
    one literal for every row the mapping lands, typed against the field's declaration when
    the project compiles and reading no source path. Two marketplaces feeding one merged
    entity each name their rows, and a single mapping can too. Lineage shows no source
    column for the field, and a value outside the field's type is refused with its path.
    (#375)

  • unit: ratio for a dimensionless factor. A canonical field holding a percentage, a
    VAT rate, a ±1 sign or a 0/1 flag declares unit: ratio, and like a count it carries no
    tax basis. So price * (100 - discount) / 100 - cost compiles instead of being refused
    as an unknown basis beside money, and price + discount is still a UnitMismatch.
    (#367)

  • A metric template can read a per-unit field of a many-to-one parent. A catalog field
    that is a rate rather than an amount, such as a product's unit cost, declares
    per_unit: true. A template at order-line grain can then compute
    product_cost * line_quantity: the grain guard admits the field read down one
    many-to-one relationship, and each mart reads the column it flattened the field into,
    under its own prefix. An amount of the parent, such as order shipping read per line, is
    refused as before. (#366)

  • A metric says which direction of change is good news. better: higher, lower or
    neither on a metric or its catalog template reaches MetricFlow as config.meta.better
    and Cube as the measure's meta.better, so a dashboard colouring a delta or reading a
    target on a cost as a ceiling takes it from the spec. A metric that does not say emits
    nothing, and a ratio or derived metric does not take it from its parts. No SQL changes.
    (#364)

Changed

  • bloomery_ir_version is 31. The IR gains MetricIR.better (#364), and EntityIR.label,
    MartIR.label and StepColumnIR.label (#373). Every project re-fingerprints once on
    upgrade, and plan() refuses to diff an artifact set compiled by 0.8 against one compiled
    by this version: re-baseline after upgrading.

Fixed

  • A relationship may name a step output, so a mart can flatten what a step computes,
    such as revenue by the buyer's segment. Its via columns are checked against the step's
    produces, and a referential rule or coverage: check through one is refused when
    the step reads the dependent entity, a cycle. (#362)

  • Two entities whose referential rules join each other are refused. Each rule is a
    LEFT JOIN inside its entity's model, so the pair compiled to two models that each read
    the other, which no engine can build. The refusal names the chain of reads. (#363)

  • Lineage runs through a sql_model step. resolve() takes the step registry as
    steps=, and with it the graph draws the step to each column it produces, a metric over
    a step output to the columns its expr reads, and a step reading another step's output
    from each of that output's columns. A walk from a step's input now reaches the metrics
    over its output, and back. Without a registry, as in bloomery lineage, the graph is as
    before. (#365)

  • A metric template's filter and semi_additive.over read the project's columns. Both
    were read as one project's literal column names, so a template filtering on
    deal_outcome or reading a snapshot over stock_at could not be used by a project
    that spells those columns otherwise. A filter name that no mart listing the metric
    carries, or an over the grain entity has no column for, now reads the one grain column
    that links it as its canonical, as the template's expr already did. A name that is
    already a column keeps it, so every spec that compiled reads what it read. (#368)

v0.8.0

Choose a tag to compare

@github-actions github-actions released this 28 Oct 15:55
8e59848

Changelog

Added

  • Epoch timestamps: parse_ts: UNIX_SECONDS and UNIX_MILLIS. A count since 1970,
    as amoCRM and many other APIs send times, maps to a timestamp in one step. It is an
    instant, so to_utc after it is refused and zone_in: may only be UTC. The count must
    be an integer of at most fifteen digits in year 1 to 9999; anything else is NULL, which
    a quality-carrying entity quarantines. It lowers to integer arithmetic from 1970 and
    reads the same under any session zone on DuckDB, PostgreSQL and Trino, held to one set
    of answers by an engine differential; other ports refuse it until measured. (#324)

  • An enum_map fallback. A "*" last pair maps every non-NULL value no earlier pair
    lists to its target: {enum_map: ["142", won, "143", lost, "*", open]} reads an open-ended
    status id as one of three outcomes in one step. NULL stays NULL. Before, unlisted values
    passed through, so the chain needed a nulling step and a coalesce, and the nulling step
    turned the field's implicit coercible rule off. A "*" anywhere but the last pair is
    refused, and so is in_enum over a field whose map has a fallback. (#323)

  • The planner binds request values. plan(..., paramstyle="pyformat"), or any of PEP
    249's five styles, puts a placeholder where each request filter value and the row
    policy's value went, and QueryPlan.params carries the values in placeholder order. Each
    value is typed as its dimension is, so a driver binds a date as a date. A runtime on
    psycopg, which reads every % once it is given parameters, can now pass them: under the
    percent styles a % in the text is doubled. QueryPlan.shape is sha256(sql), shared
    by requests that differ only in values; fingerprint still covers the values. Without a
    paramstyle, nothing changes. A metric filter's literal that happens to spell a bind
    marker stays text. (#304)

  • Descriptions, labels and synonyms on fields and metrics. A field takes
    description:, label: and synonyms:, and a metric label: and synonyms: beside its
    description, merged from a template as the description is. Each reaches the slot every
    target has for it and no other: MetricFlow's description, label and
    config.meta.synonyms, Cube's description, title and meta.synonyms, dbt's column
    description and meta.synonyms. A mart column carries the annotations of the column it
    was flattened from. The planner labels a measure column by the metric's label, falling
    back to its description. None changes SQL, and a change to any is ADDITIVE, metadata
    only. Cube carries descriptions for the first time. (#311, #313, #343)

  • A pick transform for name/value lists. {pick: [name, "Источник заказа", value]}
    reads one attribute of a list of objects, such as МойСклад's, amoCRM's or Bitrix24's
    custom fields: the take path of the first object whose scalar field reads as the text, or
    NULL. It reads the JSON text a from: extraction yields or a variant, is nullifies, and
    names its selection in lineage (source.<relation>.$.attributes[name=…].value), so two
    attributes from one list are two nodes. On DuckDB, PostgreSQL and Trino, held to one set of
    answers by an engine differential; other ports refuse it. (#312)

  • Expression metrics have lineage. lineage() walked upstream from a metric defined
    by an expr with no requires: to nothing. An expr edge now runs from each field of the
    metric's grain entity that its expr names, so the walk reaches the entity column and,
    through its mapping, bronze. The edge is lineage only: depends_on, reachability and
    plan() are unchanged, and no artifact or fingerprint moves. (#330)

  • A prepend transform. {prepend: "amocrm:"} puts a literal in front of a string,
    as concat puts one after it, so a key two sources feed can be namespaced as
    amocrm:31234567, which sorts and groups by source. NULL stays NULL. (#325)

  • A date role may name a column flattened through a to-one step. A line-grain mart
    flattening its order can declare {date: order_placed_at, role: placed}, where before a
    role had to name a base column and projects copied the order's date onto every line at
    ingestion. The buckets trace to the joined entity's column, so the zone guard reads that
    entity's clock, and MartColumnIR.via records the join that owns them. A date: step
    naming a column a later step flattens is refused. (#310)

Changed

  • bloomery_ir_version is 29 (it was 26 in 0.7.0). The IR gains MartColumnIR.via (#310), the
    annotation fields on columns and metrics (#311, #313, #343) and MetricFilterIR.nulls
    (#357). Every project re-fingerprints once on upgrade, and plan() refuses to diff an
    artifact set compiled by 0.7 against one compiled by 0.8: re-baseline after upgrading.
    The annotations and via change no SQL.

  • A metric filter's ne or not_in on a dimension that can be NULL says what NULL
    means.
    {dimension: order_channel, op: ne, values: [Маркетплейс]} read as "every
    channel but the marketplace", but SQL's <> drops every row whose channel is NULL, so
    orders with an unmapped channel vanished from the metric with no warning. Such a filter
    now takes nulls: include (keep those rows: (order_channel IS NULL OR order_channel <> 'Маркетплейс')) or nulls: exclude (drop them, as before), and is refused as
    MetricFilterInvalid without one. A key or a required field of the mart's base
    entity cannot be NULL and needs neither. A column flattened from a joined entity always
    can. (#357)

    Upgrade note. A project with such a filter stops compiling until each one says
    nulls:; nulls: exclude keeps today's numbers.

  • A ratio's numerator may range over a subset of its denominator's rows. A conversion
    rate, converted_leads filtered to outcome eq converted over an unfiltered leads, was
    refused as RatioOperandsDisagree, because the two operands had to be restricted
    identically. Each clause of the denominator's restriction now only has to be implied by
    one of the numerator's, on the same column: the same clause, an eq whose value an in
    list holds, or an in list inside another. Orderings and negations must still be the same
    clause on both sides. A win rate, status eq won over status in [won, lost], is a subset
    too; won over lost is not. (#309)

Fixed

  • A join key is a dimension the planner can group by. Every flattened column is a
    requestable mart dimension, and MartIR.dimensions and the planner say so. But the
    MetricFlow manifest made a join key only a foreign entity, so dimensions: [customer_id]
    (or a non-key via: {article: sku}) planned and MetricFlow refused it as an unknown
    group-by. A join key is now a categorical dimension as well, unless an entity of the
    model already has its name, in which case the entity serves it. The MetricFlow manifest of
    every mart with a join gains these dimensions; no other artifact changes. (#355)

  • A sql_model body reads its bound inputs through the naming policy. Under a scoping
    policy such as PrefixNaming("t0") the model was written as t0_silver.<output> but its
    body still read silver.<input>, a relation a prefixed deployment does not have. Each
    table naming a bound input is now spelled by the policy on SQLMesh and dbt, as a Tier 3
    wrapper's read already was. A table the wiring does not bind is emitted as written.
    (#348)

  • A request filter on a date or a timestamp runs on Trino. The planner wrote the value
    as quoted ISO text, which Trino reads as a varchar and refuses to compare with a date
    (TYPE_MISMATCH: Cannot apply operator: date <= varchar(10)). A row policy on a date did
    the same. DuckDB and PostgreSQL coerced the text, so nothing else saw it. The value is
    now a typed literal, DATE '2024-03-04' or TIMESTAMP '2024-03-05 09:00:00', on every
    planner dialect. (#351)

  • A chain may parse ISO 8601 twice. [{parse_ts: ISO8601}, to_string, {parse_ts: ISO8601}] was refused on every port as an unmeasured parse: the rewrite that spells the
    outer parse did not reach the inner one inside its operand. Every marker is now spelled
    however deep it sits, which also covers an epoch parse inside another and a pick of a
    pick. (#345)

  • A zoned timestamp in a request filter reads its instant. booked_at < "2024-03-05T11:00:00+02:00" was rendered as zoned text, which DuckDB and PostgreSQL cast
    to a zoneless timestamp by dropping the offset, so it compared against 11:00 UTC instead
    of 09:00. A zoned value is now converted to the UTC wall clock a timestamp dimension
    holds. (#354)

  • A metric's expr reads the field its canonical links. A catalog template is
    written in canonical names, and nothing bound them: with the line's field mapped as qty
    (canonical: quantity), Cube and MetricFlow emitted unit_price * quantity over a mart
    whose column is qty. It compiled clean and failed at query time, and the
    identity_resolution fixture's MetricFlow measure counted a customer_ref its mart lacks.
    Each requires: canonical is now bound to the grain entity's linking field, and a measure
    whose expr still names a column a carrying mart lacks is refused as
    InvalidMetricShape. (#338)

    Upgrade note. A project whose field is spelled differently from its canonical
    re-fingerprints once. A measure naming a column its mart does not carry, which failed at
    query time before, no longer compiles.

  • Every row whose dedupe key fails to cast reaches quarantine. The dedupe window
    partitioned by the coerced key, and every key that fails its cast coerces to the same
    NULL. Two bronze rows with different uncastable keys therefore shared a partition: one was
    quarantined and the other was dropped as a dedupe loser, in neither the entity nor its
    reject table. Such a row is now partitioned by...

Read more

v0.7.0

Choose a tag to compare

@github-actions github-actions released this 17 Oct 15:21
00b55c6

Changelog

Added

  • An agent skill for writing bloomery projects. npx skills add morzecrew/bloomery
    installs bloomery-skills: a routing index from a task to the references it needs,
    covering specs, compiling, planning a change, quality rules, targets and the Python API.
    Every spec example validates against the exported JSON Schema, every Python import is in
    bloomery.__all__, and every bloomery command line names a command and flags the CLI
    accepts. A census, skills/coverage.toml, gives every spec kind, emit target, CLI command
    and quality-rule kind a reference that shows it or a reason it is out of scope, so a new
    one cannot land unrecorded. Retrieval is out of scope until a project outside bloomery's
    own examples declares a RetrievalSpec.

Changed

  • A catalog recipe's operands carry types. A recipe may declare types:, a mapping
    from names in its requires to logical types written as a canonical field's type:
    is. Before a recipe's expr: is evaluated, each operand's extraction is cast to the
    type types: names it, or else to the declared type of the canonical field of the same
    name, so arithmetic runs on declared types rather than on whatever an engine infers
    from a text-shaped extraction. A recipe whose expr: applies +, - (unary too), *
    or / to an operand typed neither way, or to a name requires does not bind, is now
    refused at compile time with UntypedRecipeOperand, naming the recipe, the operand and
    the fix; a types: key that requires does not name is refused at load.

    Upgrade note. This is breaking for a catalog whose recipe arithmetic uses an
    untyped non-canonical operand: its compile is refused until the recipe's types: gains
    the entry the refusal names. Every recipe whose expr: reads a typed operand now casts
    it, canonical ones included, so its silver model and the project's fingerprint move
    even where the catalog needs no edit; recompile and re-plan.

  • A catalog recipe's / is exact. A recipe's expr: is parsed SQL rather than a built
    transform, so its division carried no exactness marker: SQLGlot rendered it as a float
    division — an explicit CAST(x AS DOUBLE PRECISION) on PostgreSQL and CAST(x AS DOUBLE)
    on Trino — narrowed back to the declared decimal, so values needing more than ~15
    significant digits could round. Every / in the recipe, nested ones included, is now
    lowered through the divide transform's marker. Its dividend is cast to a decimal at least
    29 integer digits and 9 fractional wide, so integer operands divide fractionally, a
    dividend wider than the quotient does not overflow, and a nested quotient is not rounded
    before the next division; the result is narrowed to the declared type once. The division
    is the transform's —
    exact on PostgreSQL and Trino and rendered per dialect like any other division. DuckDB,
    which has no exact decimal division, is unchanged in being inexact there.

    Upgrade note. A project whose catalog carries a dividing recipe compiles a different
    silver model, so its fingerprint moves; recompile the project and re-plan against the new
    fingerprint. A project with no dividing recipe is unaffected.

Removed

  • The inexact_division advisory. A recipe's division is now lowered through the
    divide transform's marker, so no advisory stands where the reference's own rule says a
    refusal belongs; AdvisoryCode carries undeclared_audience only.

Fixed

  • A SQL expression SQLGlot cannot convert is refused as an unparseable one. SQLGlot's
    JSON-arrow path parser calls int() on a number token, so a text such as x->1e5 or
    x->>8e makes parse_one raise a bare ValueError rather than a SqlglotError. Every
    handler around a parse caught the latter (and RecursionError), so a step body or a
    sql_macro body carrying one crossed the compile boundary as a raw ValueError
    (#281), and an authored expr: was refused at load with SQLGlot's own text instead of
    bloomery's. The classes are now named once, (SqlglotError, ValueError), in
    spec/common.py, and caught at each of the five parse doors, so such a text is a
    StepError naming the step or a SpecParseError naming the field, as any other
    unparseable text is. DepthCappedSql still passes it on for the later stage that owns
    that field's refusals.

  • A definition's timeline shows an exact division as a division. Its changes printed
    the internal BLM_EXACT_DIV(...) marker wherever a divide transform or a catalog
    recipe's / moved a body. They now print the division it renders as:
    CAST(CAST(total AS DECIMAL(38, 9)) / qty AS DECIMAL(12, 4)) for ecom_basic's
    unit_price. The comparison still reads the IR's own text, so a body that changed
    still reports as changed.

  • A declared tax_basis: unknown is refused beside money. The tax-basis check
    tripped only on an absent tax_basis, so a field that declared tax_basis: unknown
    explicitly passed it: amount + adj compiled and added a figure whose canonical field
    says nothing usable about tax. unknown now poisons additive arithmetic by being
    absent or declared, exactly as the guardrails page states.

  • InvariantViolated reaches the shell as an internal error. It subclasses
    BloomeryError, so the command line's refusal arm claimed it and exited 1 — reading a
    bloomery bug as though the spec were at fault. It is now handled ahead of that arm: exit
    3, the traceback on stderr under the report line, as any other defect is.

v0.6.0

Choose a tag to compare

@github-actions github-actions released this 03 Oct 12:34
d4570dc

Changelog

Added

  • A late row reaches SQLMesh. An incremental_by_partition entity at scd: type1 may
    declare arrival_lag: (30h, 2d, 1w). Its SQLMesh model pins interval_unit 'day'
    and sets lookback to the lag in whole days, rounded up, so a row that lands after its
    interval ran is loaded on a later run. A cron more frequent than daily on such a model is
    refused by SQLMesh unless the caller also sets allow_partials. Declared on any other
    entity, a type 2 entity included, it is refused. dbt emits nothing for it. A project that
    declares no lag compiles as before, apart from the fingerprint header.

  • A declared key is checked, not only trusted. Every entity with a key: and no
    dedupe: gets a generated blocking audit, <entity>_key_unique: no two rows share the
    key, among the current versions on an scd: type2 entity and among all rows otherwise.
    dedupe: enforces the key by construction and removes the audit; a unique rule or an
    incremental merge does not. On SQLMesh an incremental_by_partition entity checks only
    the run's own range of keys against the whole model; every other entity, and every
    entity on dbt, audits the whole model.

    Upgrade note. A source that repeats keys now stops the build: declare dedupe: on
    the entity or fix the source. History written before the audit existed is not re-checked
    by a scheduled run: sqlmesh run audits only the range it builds, and adding an audit is
    a metadata-only change that sqlmesh plan checks over the plan's own start..end range. To
    cover it, run the audit's query once over the whole model after upgrading, or deploy the
    upgrade with a plan whose range spans the history.

Changed

  • The MetricFlow manifest carries owners. A mart's owner: reaches its semantic model
    and a metric's owner: its metric, both as config.meta.owner; with no owner, config
    stays null. An entity's owner is not carried: an entity reaches MetricFlow only as entity
    elements inside a mart's semantic model, which carries the mart's owner. An exposure
    still emits no saved_query — a saved query is a request someone wrote, an exposure a
    consumer — and the manifest's saved_queries stays empty.

    Upgrade note. A project that declares an owner on a mart, or on a metric the
    manifest emits, compiles a different semantic_manifest.json, so its manifest hash
    moves; a metric MetricFlow cannot serve is omitted, owner and all.

  • bloomery_ir_version is 26 (it was 25 in 0.5.0). EntityIR carries the declared
    arrival lag, None where none is declared, so every project's fingerprint moves, and with
    it the header of every artifact that carries one. An IR written by 0.5.0 is refused by
    plan() and ir_from_json with the version mismatch message; recompile the project.

Fixed

  • A replayed parent brings its children back. When a change lets a quarantined parent's
    rows back in, bloomery plan also names in replay_scope every child that quarantined
    on it before the change, parents first, with a cycle of referential rules broken at its first
    member by name; a child that rewrote or flagged its orphans
    (on_missing: unknown_member or flag) is named in backfill_scope. Before, the child
    stayed orphaned and a blocking coverage: check on the parent failed every run. Rows
    bronze no longer holds keep their rewrite; the quality docs name the remedy.

  • A fact from an upstream project grades as imported. The evidence guard now reads the
    composed project, so a strict (requires_evidence: locked) mart over an imported base,
    a strict mart listing an imported metric, and a strict exposure naming an imported mart
    are refused, naming the upstream alias and fingerprint. The fix is
    requires_evidence: assumed on the consumer, or carrying the requirement on the
    upstream's mart.

    Upgrade note. A strict consumer resting on an imported fact compiled before and is
    refused now.

  • A dbt replay runs only against the project it was compiled with. dbt_project.yml
    carries vars: {bloomery_fingerprint: ...}, and each replay macro refuses before its
    first statement when that var differs from the fingerprint it was emitted under, or when
    the entity or its reject table lacks a column the statements write. The refusal names
    the missing columns and the rebuild that adds them: dbt run --full-refresh for an
    incremental model, which a plain run never widens. A fingerprint mismatch is fixed by
    compiling again with bloomery into a clean directory, not by a rebuild.

  • A schema change fails loudly on dbt. Every incremental dbt model, an entity's and its
    reject table's, carries on_schema_change='fail', so a field added, dropped or renamed
    stops the build instead of leaving the table silently without the column. bloomery plan names the fix beside each such field change: dbt build --full-refresh -s over the
    relations that fail, which are the entity when it is an incremental type 1 model and its
    <entity>__reject table whenever it quarantines. A full table rebuilds whole and a
    type 2 snapshot widens itself, so neither is named. The refresh recomputes history as
    SQLMesh's rebuild does; a refreshed reject table loses its resolved rows, as the dbt
    how-to says.

    Upgrade note. On dbt, a field change on an incremental entity, or on any entity that
    quarantines, now needs the full refresh the plan names.

  • A partition transform builds on Trino. The SQLMesh model spells days(...),
    months(...), years(...) and hours(...) in Trino's singular form (day, month,
    year, hour), which Trino's Iceberg connector requires; it refused the plural. A
    Hive catalog takes no transform at all: partition by a date column there. Measured on
    Trino 483: a daily interval over a coarse partition overwrites nothing outside its own
    range, so no interval unit is derived.

  • A mapped vector is refused at the field. A mapping key or field, recipe or macro aimed
    at a vector(...) column is refused at type-check with its path. It reached emit and
    failed there with no address; a vector comes from a step's declared output.

  • The fuzz lane's first findings are closed. A refused spec no longer keeps its whole
    validation graph alive, which grew a long-running caller's memory with every refusal;
    WHILE*2 and IF a, which sqlglot parses as a statement or gives up on as a Command,
    are refused as SQL text; and authored SQL text nests at most 32 levels, refused at load
    for a spec field or a quality rule's expression, and at compile for a step body, a
    registered macro body or the expression a chain of macros composes, instead of
    overflowing the stack at a later re-parse.

    Upgrade note. SQL text nested deeper than 32 levels, or that only parses as a
    statement, is refused now.

v0.5.0

Choose a tag to compare

@github-actions github-actions released this 30 Sep 12:19
8574737

Changelog

Added

  • A mart can read the current version of a type2 entity. reading: current on a
    mart's base, or on a flatten step's via: in place of as_of:, reads only the rows
    whose valid_to is NULL, so the mart's grain is the entity key again and its measures
    count keys, not revisions. A rollup over such a mart proves as one over a type1 entity
    would (S-0017/A-1, A-2). The reading is refused on a non-historical entity, beside
    as_of: on the same hop, and on an incremental mart; a type2 base with no reading is
    refused as before, and the HistoricalFanout fix text now names the current reading
    first.
  • A metric's own filter composes across marts. A request whose metrics live on
    different marts now admits a metric carrying its own filter: each branch states it as a
    Filter scoped to that metric's measures, and a group the filtered metric's branch
    lacks reads NULL for that metric. A request filter or row policy that does not reach
    every branch is still refused, and the refusal now names the filter and the branches
    that lack it.
  • The capability matrix measures dbt Core, MetricFlow and SQLMesh on every corpus
    case.
    comparisons/ gains a reproduction bundle per system for the seven cases that
    had none, and comparisons/MATRIX.md reads its cells from those bundles alone.
  • parse_ts: ISO8601_INSTANT parses text that states its own offset. Text carrying
    Z or a numeric offset becomes the UTC instant it names, identically on every port,
    and zoneless text is NULL. The value is never a wall clock: a to_utc after it is
    refused at resolve, and a zone_in: beside it may only be UTC. A parse_ts format
    carrying %z or %Z is now refused at parse and names this format; an escaped %%z
    is literal text and passes.

Changed

  • A derived: metric whose inputs are restricted differently is refused. Inputs
    carrying different filters, or a filter on one and none on another, fold rows about
    different things into one number; R019 refuses the case as it already refused a ratio
    (operands_disagree), on the single-mart and the composed path, naming each input and
    its restriction. Migration: restrict every input the same way, or none.
  • bloomery_ir_version is 25 (it was 24 in 0.4.0). MartIR and MartJoinIR carry
    the current reading; every compiled artifact's fingerprint header moves.
  • to_utc after parse_ts: ISO8601 yields NULL for text ending in Z. The chain
    declares the text a wall clock in the named zone, so a Z contradicts it exactly as
    +01:00 does, and every port now refuses it the same way rather than shifting a UTC
    value by the zone's offset a second time. A bare parse_ts: ISO8601 still reads Z
    as UTC. Migration: a source that stamps instants should parse them with
    parse_ts: ISO8601_INSTANT, released beside this change, and drop the to_utc; the
    NULL rows are visible to the coercible rule and the reject table meanwhile.
    ecom_basic's compiled output moves by the added guard.