Repository navigation
v1.13.0
·
41 commits
to main
since this release
Immutable
release. Only release title and notes can be modified.
Added
- Metrics next to the model. A second, optional file beside the model,
<stem>.metrics.yml, says what the model's numbers mean, defined by its own spec, metrics
spec 0.1.0, with a JSON Schema at
https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json. A metric is a
column aggregated (measure: orders.total_amount, byaggor the column's ownmeasure), a
row count (count: orders), a ratio of two metrics' totals (ratio: {numerator, denominator}),
or arithmetic over metrics (expression: revenue - refunds). Simple metrics and row counts take
awherefilter (a value, a list, oreq,ne,in,not_in,gt,gte,lt,lte,
between,is_null, combined withanyandall, with nulls as in SQL) and atimecolumn;
a filter may name a column of any table the metric's table reaches through its foreign keys,
many-to-one, along exactly one path. Everymeasurecolumn of the model also yields a metric of
its own (orders_total_amount) and every fact a row count (orders_count), named as model2data
studio's semantic export names them;infer: falseturns that off, and a metric of the same
name in the file replaces the inferred one.format: percentexpects a fraction (0.6 shows as
60%) and is passed on as a percent format, never multiplied.dimensions:offers a column as a dimension, with a
label, or never. The generator never reads the file: a model generates the same data with
or without one. model2data validatechecks metrics files (*.metrics.yml): the schema, then against the
model,--model, or else the model beside it with the same stem. It names every unknown table
or column, an aggregation that does not fit its column, an unknown metric in a ratio or
expression, a cycle between metrics, a filter column no path reaches or two paths reach (naming
both), a value that is not a member of its enum or not of its column's type, and a comparison
on an enum, boolean or text column; it warns about a metric with no date to report it by and a
filter that can never match the generated data (agtabove a column'smax).generate --metrics FILEadds, and changes no other file:
metric_values.json, each metric's known value over the generated data, overall and per
calendar month (counts exact, everything else rounded to 6 decimal places, half to even; the
same seed gives the same bytes); a singular dbt test per metric,
data-tests/metrics/metric_<name>.sql, that computes it over the project's staging models and
fails unless it gets that value, sodbt buildproves the metric logic on the warehouse, with
defects and--daystoo; andosi/<project>.yml, the model and its metrics as an Apache Ossie
(Open Semantic Interchange) 0.1.1 semantic model, as taggedosi-0.1.1-rc1in apache/ossie (no
final 0.1.1 tag exists yet; 0.2.0 is an unreleased draft). A
metrics file beside the model is not picked up on its own: an unchanged command keeps writing
the same files.model2data metrics listshows every metric, inferred ones included, andmodel2data metrics export --to ossiewrites the Ossie file without generating. What Ossie 0.1.1 has no
field for (a metric's label, format and time, an enum's members, a table's role and grain) goes
incustom_extensionsand is listed at the top of the file.model2data.metrics, the Python API:loadandvalidatea metrics file,resolveit with
its model into one semantic representation (entities, relationships, dimensions, measures and
metrics with their join paths),known_valuesover a run's frames,to_ossie, and
write_metric_tests.examples/ecommerce.metrics.yml: revenue, orders, average order value and units sold for
the e-commerce example, for--metrics.
Changed
- The spec's example model,
model2data/spec/examples/coffee_webshop.model2data.yml, marks
orders.total_amountandorder_items.quantityas measures, for the metrics example beside
it. No model underexamples/changed, and no output changed for any model. - The README and the PyPI page were rewritten to show what model2data does today: what you
get, one capability per line with a link; a before and after of a real run with metrics; who
it is for; and how it works, with a new demo GIF. The reference that was in the README moved to
docs/, one page per topic (the model file, generating, the dbt project, days, history, time
shapes, distributions, defects, metrics, CI, agents), worded as it was.