Repository navigation
Releases
v0.12.0
Changelog
Added
- A string step parameter can list the values it takes. A manifest declares
allocate: {type: string, values: [none, revenue, units], default: none}, and a value
outside the list, such asallocate: revenu, is refused with aStepErrorthat names
the allowed values, whether a step wiring sets it or a field mapping or repair recipe
calling asql_macrodoes. Before, a misspelled value compiled and the
step ran whichever branch its body fell through to. A manifest is refused if its
defaultis not one of itsvalues, or if a parameter that is not astringdeclares
them. (#403) - A step input can be optional, so one step serves projects with and without a
source. Asql_modeldeclares itoptional: truewith typedcolumns:,
and a wiring may leave it out. The body reads it by name, as the bound relation,
which must carry those columns, or unbound as an empty relation of them. (#402)
Fixed
- An entity field no mapping produces is a NULL column of its silver model. It was
left out, so a step whose inputrequiresit compiled and then failed on the engine
(Binder Error: … does not have a column named …). The column is now typed NULL, as a
field only some of a merged entity's sources map already was, so marts and the semantic
layers carry it too. Three kinds stay out: a required field the one mapping omits, a
vector field, and a field spelled as a canonical that a mapped field links, which a
metric's filter would otherwise read as written. A step that requires one of them is
refused at compile. (#404)
v0.11.2
Changelog
Fixed
- A spec value YAML cannot build is a
SpecParseErrornaming its line. An unquoted
date the calendar does not have, such as2026-13-01, a type tag over text it cannot
read, such as!!int ""or!!bool maybe, or a\Uescape past U+10FFFF raised a
bare Python exception out ofload_projectand the CLI. The refusal now names the
document and the line, and says to quote the value and drop any!!tag if it is
meant as text. (#398)
v0.11.1
Changelog
Fixed
- A transform applied to another transform's output renders for its engine. On
PostgreSQL, astrip_suffixorregex_extractover astrip_suffixor
regex_extractfailed at run time (function ends_with(text, unknown) does not exist, and likewiseregexp_extract), and Redshift failed the same way on a
strip_suffixof astrip_suffix. On Redshift astrip_prefixof astrip_prefix
read the inner prefix throughLIKE, where_and%are wildcards, and a
regex_extractof aregex_extracttook the inner one's whole match rather than its
group. On Databricks the innerregex_extractlost its group index and read the
engine's default, group 1. Each step now renders as it does alone, and so a nested
regex_extractnaming a group above 1 is refused on Redshift at compile, as a single
one is, where it used to compile to a call that read the group as a start position. regex_extractis NULL on DuckDB where the pattern does not match, as it already
was on PostgreSQL and Trino, rather than''. Acoalesceorenum_mapafter the
extract now sees the miss on DuckDB too, so a mapping that falls back to a default
lands the same silver on both engines, and a missing reference is NULL rather than an
id spelled''. A capture group that takes no part in the match is NULL too, as on
PostgreSQL and Trino, while a group that matches the empty text is still''. Only
the emitted SQL changes, so IR fingerprints stay the same, but DuckDB silver rebuilt
from the same bronze moves from''to NULL on those rows. (#392)
v0.11.0
Changelog
Added
- A naming policy can name a relation's catalog.
relation()may return
(catalog, namespace, relation)as well as(namespace, relation), and every target
renders the catalog as a real third part:lake.silver.orderin the SQL and the SQLMesh
model name,+databaseand a source'sdatabaseon dbt,node_relation.databaseon
MetricFlow, and Cube'ssql_table. One DuckDB project can read bronze from an attached
Postgres and write silver and gold to a DuckLake without rewriting the emitted files,
and plans read the same relations. A policy that returns a pair compiles exactly as
before. (#389)
v0.10.0
Changelog
Added
-
A key part can be a constant.
key: {marketplace: {const: Ozon}, order_id: {from: …}}
keys each mapping's rows by a value fixed for that mapping beside the source's own id,
so two sources feeding one merged entity land as distinct rows where their ids overlap,
and the collision audit stays silent on them. It is typed and lowered as a field's
constant is, and a quarantined row replays with it. A mapping that sets every key
column of the entity as a constant is refused, since every row of it would carry one
key. (#380) -
A plan change says what happened, and to which column, as values.
Changegains
action(added,removed,renamed,retypedorchanged, the newChangeAction
enum),member(the column, measure, dimension column or judged field, wheresubject
names the mart or entity holding it) andadvice(per-target advice as
(target, sentence)pairs). A program showing a plan in another language, or grouped by
column, no longer reads them offdetail's English, and a new mart column reported as a
flattened column and as a dimension carries onememberin both.bloomery plan --format jsoncarries the three fields. (#383)
Changed
-
A
per_unitfield can be read through a chain of many-to-one relationships, not
only one: a marketplace sale reading the product's unit cost through its item compiles,
and each mart reads the column it flattened the cost into. A hop read one-to-many, from
a parent to its children, anywhere in the chain still refuses it, and a refusal now
names the chain it was reached by rather than "no declared relationship". (#382) -
Change.detailno longer carries the dbt full-refresh advice. A field added to,
dropped from or renamed on an incremental entity used to end itsdetailwith; on dbt, run …; that sentence is now inadviceunderdbt, so a project shipped to SQLMesh is
not told to run a dbt command. A reader matching the olddetailtext must read
advice.bloomery plan's text output still prints it after the sentence. A flattened
mart column whose type alone moved now readsflattened column '…' retyped, with the two
types inold/new, and a newquarantine:block readsquarantine block added;
both readchangedbefore. (#383)
Fixed
-
Lineage through a step no longer shows a column named after the step's output.
Given the step registry, a step feeds only the columns its manifest produces, so a
walk or a rendered graph no longer carriescustomer_profile.profilebeside
customer_profile.segment. Without a registry, as inbloomery lineage, that node
is still the step's one out-edge, standing for the whole relation it writes. In
timeline(), a change to aquality:rule a wiring declares on its output is now
reported on the step's node. (#381) -
A field one mapping reads and another sets as a constant no longer refuses the merge.
On a merged entity in the quality system, the reading mapping's implicitcoercible
rule was taken for aquality:disagreement with the constant, which has no rule to
agree with, and nothing either mapping declared could fix it. The rule now applies to
the reading source's rows only. (#386)
v0.9.0
Changelog
Added
-
Entities, marts and step-produced columns take a
label:. An entity's label reaches
the MetricFlow entity that stands for it, a mart's becomes MetricFlow's semantic model
labeland Cube's cube and viewtitle, and a manifest'sproduces:column label
reaches the output entity's column as a mapped field's does. No SQL changes, and
changing one is metadata only inplan()andtimeline(); a step whose column label
changed is not restated. (#373) -
A field mapping can set a constant.
marketplace: {const: Wildberries}sets the field to
one literal for every row the mapping lands, typed against the field's declaration when
the project compiles and reading no source path. Two marketplaces feeding one merged
entity each name their rows, and a single mapping can too. Lineage shows no source
column for the field, and a value outside the field's type is refused with its path.
(#375) -
unit: ratiofor a dimensionless factor. A canonical field holding a percentage, a
VAT rate, a ±1 sign or a 0/1 flag declaresunit: ratio, and like a count it carries no
tax basis. Soprice * (100 - discount) / 100 - costcompiles instead of being refused
as an unknown basis beside money, andprice + discountis still aUnitMismatch.
(#367) -
A metric template can read a per-unit field of a many-to-one parent. A catalog field
that is a rate rather than an amount, such as a product's unit cost, declares
per_unit: true. A template at order-line grain can then compute
product_cost * line_quantity: the grain guard admits the field read down one
many-to-one relationship, and each mart reads the column it flattened the field into,
under its own prefix. An amount of the parent, such as order shipping read per line, is
refused as before. (#366) -
A metric says which direction of change is good news.
better: higher,loweror
neitheron a metric or its catalog template reaches MetricFlow asconfig.meta.better
and Cube as the measure'smeta.better, so a dashboard colouring a delta or reading a
target on a cost as a ceiling takes it from the spec. A metric that does not say emits
nothing, and a ratio or derived metric does not take it from its parts. No SQL changes.
(#364)
Changed
bloomery_ir_versionis 31. The IR gainsMetricIR.better(#364), andEntityIR.label,
MartIR.labelandStepColumnIR.label(#373). Every project re-fingerprints once on
upgrade, andplan()refuses to diff an artifact set compiled by 0.8 against one compiled
by this version: re-baseline after upgrading.
Fixed
-
A relationship may name a step output, so a mart can flatten what a step computes,
such as revenue by the buyer's segment. Itsviacolumns are checked against the step's
produces, and areferentialrule orcoverage:check through one is refused when
the step reads the dependent entity, a cycle. (#362) -
Two entities whose referential rules join each other are refused. Each rule is a
LEFT JOIN inside its entity's model, so the pair compiled to two models that each read
the other, which no engine can build. The refusal names the chain of reads. (#363) -
Lineage runs through a
sql_modelstep.resolve()takes the step registry as
steps=, and with it the graph draws the step to each column it produces, a metric over
a step output to the columns itsexprreads, and a step reading another step's output
from each of that output's columns. A walk from a step's input now reaches the metrics
over its output, and back. Without a registry, as inbloomery lineage, the graph is as
before. (#365) -
A metric template's filter and
semi_additive.overread the project's columns. Both
were read as one project's literal column names, so a template filtering on
deal_outcomeor reading a snapshot overstock_atcould not be used by a project
that spells those columns otherwise. A filter name that no mart listing the metric
carries, or anoverthe grain entity has no column for, now reads the one grain column
that links it as its canonical, as the template'sexpralready did. A name that is
already a column keeps it, so every spec that compiled reads what it read. (#368)
v0.8.0
Changelog
Added
-
Epoch timestamps:
parse_ts: UNIX_SECONDSandUNIX_MILLIS. A count since 1970,
as amoCRM and many other APIs send times, maps to atimestampin one step. It is an
instant, soto_utcafter it is refused andzone_in:may only beUTC. The count must
be an integer of at most fifteen digits in year 1 to 9999; anything else is NULL, which
a quality-carrying entity quarantines. It lowers to integer arithmetic from 1970 and
reads the same under any session zone on DuckDB, PostgreSQL and Trino, held to one set
of answers by an engine differential; other ports refuse it until measured. (#324) -
An
enum_mapfallback. A"*"last pair maps every non-NULL value no earlier pair
lists to its target:{enum_map: ["142", won, "143", lost, "*", open]}reads an open-ended
status id as one of three outcomes in one step. NULL stays NULL. Before, unlisted values
passed through, so the chain needed a nulling step and acoalesce, and the nulling step
turned the field's implicitcoerciblerule off. A"*"anywhere but the last pair is
refused, and so isin_enumover a field whose map has a fallback. (#323) -
The planner binds request values.
plan(..., paramstyle="pyformat"), or any of PEP
249's five styles, puts a placeholder where each request filter value and the row
policy's value went, andQueryPlan.paramscarries the values in placeholder order. Each
value is typed as its dimension is, so a driver binds adateas a date. A runtime on
psycopg, which reads every%once it is given parameters, can now pass them: under the
percent styles a%in the text is doubled.QueryPlan.shapeissha256(sql), shared
by requests that differ only in values;fingerprintstill covers the values. Without a
paramstyle, nothing changes. A metric filter's literal that happens to spell a bind
marker stays text. (#304) -
Descriptions, labels and synonyms on fields and metrics. A field takes
description:,label:andsynonyms:, and a metriclabel:andsynonyms:beside its
description, merged from a template as the description is. Each reaches the slot every
target has for it and no other: MetricFlow'sdescription,labeland
config.meta.synonyms, Cube'sdescription,titleandmeta.synonyms, dbt's column
descriptionandmeta.synonyms. A mart column carries the annotations of the column it
was flattened from. The planner labels a measure column by the metric's label, falling
back to its description. None changes SQL, and a change to any is ADDITIVE, metadata
only. Cube carries descriptions for the first time. (#311, #313, #343) -
A
picktransform for name/value lists.{pick: [name, "Источник заказа", value]}
reads one attribute of a list of objects, such as МойСклад's, amoCRM's or Bitrix24's
custom fields: the take path of the first object whose scalar field reads as the text, or
NULL. It reads the JSON text afrom:extraction yields or avariant, isnullifies, and
names its selection in lineage (source.<relation>.$.attributes[name=…].value), so two
attributes from one list are two nodes. On DuckDB, PostgreSQL and Trino, held to one set of
answers by an engine differential; other ports refuse it. (#312) -
Expression metrics have lineage.
lineage()walked upstream from a metric defined
by anexprwith norequires:to nothing. Anexpredge now runs from each field of the
metric's grain entity that itsexprnames, so the walk reaches the entity column and,
through its mapping, bronze. The edge is lineage only:depends_on, reachability and
plan()are unchanged, and no artifact or fingerprint moves. (#330) -
A
prependtransform.{prepend: "amocrm:"}puts a literal in front of a string,
asconcatputs one after it, so a key two sources feed can be namespaced as
amocrm:31234567, which sorts and groups by source. NULL stays NULL. (#325) -
A date role may name a column flattened through a to-one step. A line-grain mart
flattening its order can declare{date: order_placed_at, role: placed}, where before a
role had to name a base column and projects copied the order's date onto every line at
ingestion. The buckets trace to the joined entity's column, so the zone guard reads that
entity's clock, andMartColumnIR.viarecords the join that owns them. Adate:step
naming a column a later step flattens is refused. (#310)
Changed
-
bloomery_ir_versionis 29 (it was 26 in 0.7.0). The IR gainsMartColumnIR.via(#310), the
annotation fields on columns and metrics (#311, #313, #343) andMetricFilterIR.nulls
(#357). Every project re-fingerprints once on upgrade, andplan()refuses to diff an
artifact set compiled by 0.7 against one compiled by 0.8: re-baseline after upgrading.
The annotations andviachange no SQL. -
A metric filter's
neornot_inon a dimension that can be NULL says what NULL
means.{dimension: order_channel, op: ne, values: [Маркетплейс]}read as "every
channel but the marketplace", but SQL's<>drops every row whose channel is NULL, so
orders with an unmapped channel vanished from the metric with no warning. Such a filter
now takesnulls: include(keep those rows:(order_channel IS NULL OR order_channel <> 'Маркетплейс')) ornulls: exclude(drop them, as before), and is refused as
MetricFilterInvalidwithout one. A key or arequiredfield of the mart's base
entity cannot be NULL and needs neither. A column flattened from a joined entity always
can. (#357)Upgrade note. A project with such a filter stops compiling until each one says
nulls:;nulls: excludekeeps today's numbers. -
A ratio's numerator may range over a subset of its denominator's rows. A conversion
rate,converted_leadsfiltered tooutcome eq convertedover an unfilteredleads, was
refused asRatioOperandsDisagree, because the two operands had to be restricted
identically. Each clause of the denominator's restriction now only has to be implied by
one of the numerator's, on the same column: the same clause, aneqwhose value anin
list holds, or aninlist inside another. Orderings and negations must still be the same
clause on both sides. A win rate,status eq wonoverstatus in [won, lost], is a subset
too; won over lost is not. (#309)
Fixed
-
A join key is a dimension the planner can group by. Every flattened column is a
requestable mart dimension, andMartIR.dimensionsand the planner say so. But the
MetricFlow manifest made a join key only a foreign entity, sodimensions: [customer_id]
(or a non-keyvia: {article: sku}) planned and MetricFlow refused it as an unknown
group-by. A join key is now a categorical dimension as well, unless an entity of the
model already has its name, in which case the entity serves it. The MetricFlow manifest of
every mart with a join gains these dimensions; no other artifact changes. (#355) -
A
sql_modelbody reads its bound inputs through the naming policy. Under a scoping
policy such asPrefixNaming("t0")the model was written ast0_silver.<output>but its
body still readsilver.<input>, a relation a prefixed deployment does not have. Each
table naming a bound input is now spelled by the policy on SQLMesh and dbt, as a Tier 3
wrapper's read already was. A table the wiring does not bind is emitted as written.
(#348) -
A request filter on a date or a timestamp runs on Trino. The planner wrote the value
as quoted ISO text, which Trino reads as avarcharand refuses to compare with a date
(TYPE_MISMATCH: Cannot apply operator: date <= varchar(10)). A row policy on a date did
the same. DuckDB and PostgreSQL coerced the text, so nothing else saw it. The value is
now a typed literal,DATE '2024-03-04'orTIMESTAMP '2024-03-05 09:00:00', on every
planner dialect. (#351) -
A chain may parse ISO 8601 twice.
[{parse_ts: ISO8601}, to_string, {parse_ts: ISO8601}]was refused on every port as an unmeasured parse: the rewrite that spells the
outer parse did not reach the inner one inside its operand. Every marker is now spelled
however deep it sits, which also covers an epoch parse inside another and apickof a
pick. (#345) -
A zoned timestamp in a request filter reads its instant.
booked_at < "2024-03-05T11:00:00+02:00"was rendered as zoned text, which DuckDB and PostgreSQL cast
to a zonelesstimestampby dropping the offset, so it compared against 11:00 UTC instead
of 09:00. A zoned value is now converted to the UTC wall clock a timestamp dimension
holds. (#354) -
A metric's
exprreads the field its canonical links. A catalog template is
written in canonical names, and nothing bound them: with the line's field mapped asqty
(canonical: quantity), Cube and MetricFlow emittedunit_price * quantityover a mart
whose column isqty. It compiled clean and failed at query time, and the
identity_resolutionfixture's MetricFlow measure counted acustomer_refits mart lacks.
Eachrequires:canonical is now bound to the grain entity's linking field, and a measure
whoseexprstill names a column a carrying mart lacks is refused as
InvalidMetricShape. (#338)Upgrade note. A project whose field is spelled differently from its canonical
re-fingerprints once. A measure naming a column its mart does not carry, which failed at
query time before, no longer compiles. -
Every row whose dedupe key fails to cast reaches quarantine. The dedupe window
partitioned by the coerced key, and every key that fails its cast coerces to the same
NULL. Two bronze rows with different uncastable keys therefore shared a partition: one was
quarantined and the other was dropped as a dedupe loser, in neither the entity nor its
reject table. Such a row is now partitioned by...
v0.7.0
Changelog
Added
- An agent skill for writing bloomery projects.
npx skills add morzecrew/bloomery
installsbloomery-skills: a routing index from a task to the references it needs,
covering specs, compiling, planning a change, quality rules, targets and the Python API.
Every spec example validates against the exported JSON Schema, every Python import is in
bloomery.__all__, and everybloomerycommand line names a command and flags the CLI
accepts. A census,skills/coverage.toml, gives every spec kind, emit target, CLI command
and quality-rule kind a reference that shows it or a reason it is out of scope, so a new
one cannot land unrecorded. Retrieval is out of scope until a project outside bloomery's
own examples declares aRetrievalSpec.
Changed
-
A catalog recipe's operands carry types. A recipe may declare
types:, a mapping
from names in itsrequiresto logical types written as a canonical field'stype:
is. Before a recipe'sexpr:is evaluated, each operand's extraction is cast to the
typetypes:names it, or else to the declared type of the canonical field of the same
name, so arithmetic runs on declared types rather than on whatever an engine infers
from a text-shaped extraction. A recipe whoseexpr:applies+,-(unary too),*
or/to an operand typed neither way, or to a namerequiresdoes not bind, is now
refused at compile time withUntypedRecipeOperand, naming the recipe, the operand and
the fix; atypes:key thatrequiresdoes not name is refused at load.Upgrade note. This is breaking for a catalog whose recipe arithmetic uses an
untyped non-canonical operand: its compile is refused until the recipe'stypes:gains
the entry the refusal names. Every recipe whoseexpr:reads a typed operand now casts
it, canonical ones included, so its silver model and the project's fingerprint move
even where the catalog needs no edit; recompile and re-plan. -
A catalog recipe's
/is exact. A recipe'sexpr:is parsed SQL rather than a built
transform, so its division carried no exactness marker: SQLGlot rendered it as a float
division — an explicitCAST(x AS DOUBLE PRECISION)on PostgreSQL andCAST(x AS DOUBLE)
on Trino — narrowed back to the declared decimal, so values needing more than ~15
significant digits could round. Every/in the recipe, nested ones included, is now
lowered through thedividetransform's marker. Its dividend is cast to a decimal at least
29 integer digits and 9 fractional wide, so integer operands divide fractionally, a
dividend wider than the quotient does not overflow, and a nested quotient is not rounded
before the next division; the result is narrowed to the declared type once. The division
is the transform's —
exact on PostgreSQL and Trino and rendered per dialect like any other division. DuckDB,
which has no exact decimal division, is unchanged in being inexact there.Upgrade note. A project whose catalog carries a dividing recipe compiles a different
silver model, so its fingerprint moves; recompile the project and re-plan against the new
fingerprint. A project with no dividing recipe is unaffected.
Removed
- The
inexact_divisionadvisory. A recipe's division is now lowered through the
dividetransform's marker, so no advisory stands where the reference's own rule says a
refusal belongs;AdvisoryCodecarriesundeclared_audienceonly.
Fixed
-
A SQL expression SQLGlot cannot convert is refused as an unparseable one. SQLGlot's
JSON-arrow path parser callsint()on a number token, so a text such asx->1e5or
x->>8emakesparse_oneraise a bareValueErrorrather than aSqlglotError. Every
handler around a parse caught the latter (andRecursionError), so a step body or a
sql_macrobody carrying one crossed the compile boundary as a rawValueError
(#281), and an authoredexpr:was refused at load with SQLGlot's own text instead of
bloomery's. The classes are now named once,(SqlglotError, ValueError), in
spec/common.py, and caught at each of the five parse doors, so such a text is a
StepErrornaming the step or aSpecParseErrornaming the field, as any other
unparseable text is.DepthCappedSqlstill passes it on for the later stage that owns
that field's refusals. -
A definition's timeline shows an exact division as a division. Its changes printed
the internalBLM_EXACT_DIV(...)marker wherever adividetransform or a catalog
recipe's/moved a body. They now print the division it renders as:
CAST(CAST(total AS DECIMAL(38, 9)) / qty AS DECIMAL(12, 4))forecom_basic's
unit_price. The comparison still reads the IR's own text, so a body that changed
still reports as changed. -
A declared
tax_basis: unknownis refused beside money. The tax-basis check
tripped only on an absenttax_basis, so a field that declaredtax_basis: unknown
explicitly passed it:amount + adjcompiled and added a figure whose canonical field
says nothing usable about tax.unknownnow poisons additive arithmetic by being
absent or declared, exactly as the guardrails page states. -
InvariantViolatedreaches the shell as an internal error. It subclasses
BloomeryError, so the command line's refusal arm claimed it and exited1— reading a
bloomery bug as though the spec were at fault. It is now handled ahead of that arm: exit
3, the traceback on stderr under the report line, as any other defect is.
v0.6.0
Changelog
Added
-
A late row reaches SQLMesh. An
incremental_by_partitionentity atscd: type1may
declarearrival_lag:(30h,2d,1w). Its SQLMesh model pinsinterval_unit 'day'
and setslookbackto the lag in whole days, rounded up, so a row that lands after its
interval ran is loaded on a later run. A cron more frequent than daily on such a model is
refused by SQLMesh unless the caller also setsallow_partials. Declared on any other
entity, a type 2 entity included, it is refused. dbt emits nothing for it. A project that
declares no lag compiles as before, apart from the fingerprint header. -
A declared key is checked, not only trusted. Every entity with a
key:and no
dedupe:gets a generated blocking audit,<entity>_key_unique: no two rows share the
key, among the current versions on anscd: type2entity and among all rows otherwise.
dedupe:enforces the key by construction and removes the audit; auniquerule or an
incremental merge does not. On SQLMesh anincremental_by_partitionentity checks only
the run's own range of keys against the whole model; every other entity, and every
entity on dbt, audits the whole model.Upgrade note. A source that repeats keys now stops the build: declare
dedupe:on
the entity or fix the source. History written before the audit existed is not re-checked
by a scheduled run:sqlmesh runaudits only the range it builds, and adding an audit is
a metadata-only change thatsqlmesh planchecks over the plan's own start..end range. To
cover it, run the audit's query once over the whole model after upgrading, or deploy the
upgrade with aplanwhose range spans the history.
Changed
-
The MetricFlow manifest carries owners. A mart's
owner:reaches its semantic model
and a metric'sowner:its metric, both asconfig.meta.owner; with no owner,config
stays null. An entity's owner is not carried: an entity reaches MetricFlow only as entity
elements inside a mart's semantic model, which carries the mart's owner. An exposure
still emits nosaved_query— a saved query is a request someone wrote, an exposure a
consumer — and the manifest'ssaved_queriesstays empty.Upgrade note. A project that declares an owner on a mart, or on a metric the
manifest emits, compiles a differentsemantic_manifest.json, so its manifest hash
moves; a metric MetricFlow cannot serve is omitted, owner and all. -
bloomery_ir_versionis 26 (it was 25 in 0.5.0).EntityIRcarries the declared
arrival lag,Nonewhere none is declared, so every project's fingerprint moves, and with
it the header of every artifact that carries one. An IR written by 0.5.0 is refused by
plan()andir_from_jsonwith the version mismatch message; recompile the project.
Fixed
-
A replayed parent brings its children back. When a change lets a quarantined parent's
rows back in,bloomery planalso names inreplay_scopeevery child that quarantined
on it before the change, parents first, with a cycle ofreferentialrules broken at its first
member by name; a child that rewrote or flagged its orphans
(on_missing: unknown_memberorflag) is named inbackfill_scope. Before, the child
stayed orphaned and a blockingcoverage:check on the parent failed every run. Rows
bronze no longer holds keep their rewrite; the quality docs name the remedy. -
A fact from an upstream project grades as imported. The evidence guard now reads the
composed project, so a strict (requires_evidence: locked) mart over an imported base,
a strict mart listing an imported metric, and a strict exposure naming an imported mart
are refused, naming the upstream alias and fingerprint. The fix is
requires_evidence: assumedon the consumer, or carrying the requirement on the
upstream's mart.Upgrade note. A strict consumer resting on an imported fact compiled before and is
refused now. -
A dbt replay runs only against the project it was compiled with.
dbt_project.yml
carriesvars: {bloomery_fingerprint: ...}, and each replay macro refuses before its
first statement when that var differs from the fingerprint it was emitted under, or when
the entity or its reject table lacks a column the statements write. The refusal names
the missing columns and the rebuild that adds them:dbt run --full-refreshfor an
incremental model, which a plain run never widens. A fingerprint mismatch is fixed by
compiling again with bloomery into a clean directory, not by a rebuild. -
A schema change fails loudly on dbt. Every incremental dbt model, an entity's and its
reject table's, carrieson_schema_change='fail', so a field added, dropped or renamed
stops the build instead of leaving the table silently without the column.bloomery plannames the fix beside each such field change:dbt build --full-refresh -sover the
relations that fail, which are the entity when it is an incremental type 1 model and its
<entity>__rejecttable whenever it quarantines. Afulltable rebuilds whole and a
type 2 snapshot widens itself, so neither is named. The refresh recomputes history as
SQLMesh's rebuild does; a refreshed reject table loses its resolved rows, as the dbt
how-to says.Upgrade note. On dbt, a field change on an incremental entity, or on any entity that
quarantines, now needs the full refresh the plan names. -
A partition transform builds on Trino. The SQLMesh model spells
days(...),
months(...),years(...)andhours(...)in Trino's singular form (day,month,
year,hour), which Trino's Iceberg connector requires; it refused the plural. A
Hive catalog takes no transform at all: partition by a date column there. Measured on
Trino 483: a daily interval over a coarse partition overwrites nothing outside its own
range, so no interval unit is derived. -
A mapped vector is refused at the field. A mapping key or field, recipe or macro aimed
at avector(...)column is refused at type-check with its path. It reached emit and
failed there with no address; a vector comes from a step's declared output. -
The fuzz lane's first findings are closed. A refused spec no longer keeps its whole
validation graph alive, which grew a long-running caller's memory with every refusal;
WHILE*2andIF a, which sqlglot parses as a statement or gives up on as aCommand,
are refused as SQL text; and authored SQL text nests at most 32 levels, refused at load
for a spec field or a quality rule's expression, and at compile for a step body, a
registered macro body or the expression a chain of macros composes, instead of
overflowing the stack at a later re-parse.Upgrade note. SQL text nested deeper than 32 levels, or that only parses as a
statement, is refused now.
v0.5.0
Changelog
Added
- A mart can read the current version of a type2 entity.
reading: currenton a
mart'sbase, or on a flatten step'svia:in place ofas_of:, reads only the rows
whosevalid_tois NULL, so the mart's grain is the entity key again and its measures
count keys, not revisions. A rollup over such a mart proves as one over a type1 entity
would (S-0017/A-1, A-2). The reading is refused on a non-historical entity, beside
as_of:on the same hop, and on an incremental mart; a type2 base with no reading is
refused as before, and theHistoricalFanoutfix text now names the current reading
first. - A metric's own
filtercomposes across marts. A request whose metrics live on
different marts now admits a metric carrying its own filter: each branch states it as a
Filterscoped to that metric's measures, and a group the filtered metric's branch
lacks reads NULL for that metric. A request filter or row policy that does not reach
every branch is still refused, and the refusal now names the filter and the branches
that lack it. - The capability matrix measures dbt Core, MetricFlow and SQLMesh on every corpus
case.comparisons/gains a reproduction bundle per system for the seven cases that
had none, andcomparisons/MATRIX.mdreads its cells from those bundles alone. parse_ts: ISO8601_INSTANTparses text that states its own offset. Text carrying
Zor a numeric offset becomes the UTC instant it names, identically on every port,
and zoneless text is NULL. The value is never a wall clock: ato_utcafter it is
refused at resolve, and azone_in:beside it may only beUTC. Aparse_tsformat
carrying%zor%Zis now refused at parse and names this format; an escaped%%z
is literal text and passes.
Changed
- A
derived:metric whose inputs are restricted differently is refused. Inputs
carrying different filters, or a filter on one and none on another, fold rows about
different things into one number; R019 refuses the case as it already refused a ratio
(operands_disagree), on the single-mart and the composed path, naming each input and
its restriction. Migration: restrict every input the same way, or none. bloomery_ir_versionis 25 (it was 24 in 0.4.0).MartIRandMartJoinIRcarry
the current reading; every compiled artifact's fingerprint header moves.to_utcafterparse_ts: ISO8601yields NULL for text ending inZ. The chain
declares the text a wall clock in the named zone, so aZcontradicts it exactly as
+01:00does, and every port now refuses it the same way rather than shifting a UTC
value by the zone's offset a second time. A bareparse_ts: ISO8601still readsZ
as UTC. Migration: a source that stamps instants should parse them with
parse_ts: ISO8601_INSTANT, released beside this change, and drop theto_utc; the
NULL rows are visible to thecoerciblerule and the reject table meanwhile.
ecom_basic's compiled output moves by the added guard.