v1.16.3

Documentation only: the same model, seed and --as-of write the same files as 1.16.2.

Changed

• The README and the PyPI page were rewritten around the problem model2data solves and the
  answer to it: the problem first, then what the engine gives you, a shortened before and after,
  who it is for, and model2data studio with a CLI / Free / Pro comparison and its pricing. The
  capability list and the limitations were tightened, and the limitation claiming no support for
  incremental models, which --days contradicted, is gone.

Full changelog: v1.16.2...v1.16.3

v1.16.2

Metric totals no longer count a row more than once when the table their time column is on repeats a key. On data without repeated keys, the data and metric_values.json are byte-identical to 1.16.1; only the SQL of some generated metric tests changes.

Fixed

• A metric's total no longer joins the table its time column is on. The query for a
  metric's total value (in metric_values.json and in the generated dbt test) joined every
  table the metric's filter or time column is on, although the total reads no time column.
  When the defects repeat a key in that parent table (the messy preset does), the join
  repeats the metric's rows and the total comes out too high: on the saas_platform example
  (seed 42, 500 rows, messy), the row count of api_usage, dated by
  organizations.created_at, came out as 502 rather than 500. The total now joins only the
  tables its filter reads; the by-month values still join the time column's table.
  • Output change, deliberate: on data without repeated keys the values are the same
    and metric_values.json is byte-identical, but a generated metric test whose metric is
    dated by a parent's column now reads one table fewer, so its SQL text differs. With
    repeated parent keys, totals change to the right numbers.

Full changelog: v1.16.1...v1.16.2

v1.16.1

Known metric values now take much less memory and time. Output is byte-identical to 1.16.0: the same model, seed and --as-of write the same files.

Performance

• Known metric values read only the columns the metrics read. metric_values.json
  used to turn every column of every table a metric reads into text, through several
  full copies, and load all of it into DuckDB. It now loads only the columns some metric's
  query names (its measure, filters, time column and join keys, recorded as the query is
  written), turns them into text one table at a time, and makes "" and null null in
  the load's SQL rather than in a pandas pass. The values are the same: a column's CSV text
  does not depend on the columns beside it, and a query reads nothing it does not name. On
  the saas_platform example at 50,000 rows, with a counts-and-a-few-measures set of
  metrics, the extra memory falls from about 420 MB to under 60 MB and the time from 1.6 s
  to 0.5 s; with metrics on nearly every column the saving is smaller (about 265 MB to
  155 MB).

Full changelog: v1.16.0...v1.16.1

v1.16.0

An order can no longer predate its customer: after now names a parent's column. Integer primary keys number the rows 1..N, in key order. This release also carries the fixes of v1.15.0, whose package never reached PyPI. Every model with an integer primary key generates different data for the same seed: see Upgrading below.

Added

• after can name a parent's column. order_date: {after: customers.created_at} keeps every order on or after the created_at of the customer its foreign key points at. In examples/ecommerce, half the orders used to predate their customer; now none do.
  • after takes a column of the same row (as before), a parent's column as <table>.<column>, or a list of either: after: [orders.ordered_at, products.launched_at]. A date compared with a timestamp compares by day.
  • The parent is reached through exactly one one-column foreign key onto its primary key or a unique column; a null foreign key constrains nothing. Two foreign keys to the same parent (sender_id and receiver_id to users) are an error naming both: choosing one is not built yet.
  • The child keeps its own shape. A row dated before its parent takes another parent created by then, so growth, seasonality, business hours and the as_of window hold exactly; only a row with no such parent (or a one-to-one) moves its date, within the column's own shape and never past as_of. On the coffee-webshop example the gift-season peak and growth are unchanged; a skewed foreign key gets steeper, because early customers have had longer to order.
  • Needs spec 0.5.0: a document using a parent's column or a list says model2data: 0.5.0, and an older document using them gets an error naming the version to write. DBML converts to 0.5.0 on its own. Schema: https://www.jbanalytica.com/model2data/spec/0.5.0/model.schema.json
• A warning where a child's date can fall before its parent's, with the hint to add: "can fall before customers.created_at (the customers row it points at through customer_id); add after: customers.created_at to keep it after". It carries a structured suggestion (path, value, spec) for tools to offer as a fix. Generation itself does not change.
• dbt test model2data_not_before_parent. With --hint-tests, each parent column an after names gets a test that fails on rows dated before their parent row. The macro ships in macros/model2data_parent_tests.sql, written only when needed. The defects report includes it in EXPECTED_FAILURES.md.
• The bundled examples use it where it is plainly right (ecommerce, ecommerce_daily, ecommerce_training, advanced_features, saas_platform) and now say spec 0.5.0.

Changed

• Integer primary keys number the rows 1..N, and rows come out in key order. They used to be a shuffled sample starting at 0 (the first customers were 4251, 4557, 1941, ...). Every output (seeds, later days' batches and change log, the defects' files) lists rows in key order. Keys with min/max or a distribution, and integer keys that are also a foreign key, are drawn as before and sorted. Text, uuid and composite keys are unchanged. Later days continue from the largest key, as before.

Fixed (first released as v1.15.0, which never reached PyPI)

• A binary column holds a short value. binary, varbinary, bytea, blob and bytes columns now hold 16 random bytes written as 32 hex characters, instead of a whole MiB per value (a 50-row table wrote a 391 MB seed that dbt refused). They stay unique when they are a key.
• A key that references a column with fewer distinct values caps its table at one row per distinct parent value, instead of failing its own unique test, and the summary names the table and the reference that capped it.

Upgrading

Output changes for the same seed:

• Every model with an integer primary key: the key is 1..N and no longer draws from the table's random stream, so every column after it draws differently.
• Dates change only where a model uses a parent's column in after.
• A table with a byte column, and a table whose key references a column short of distinct values (see Fixed).

The same model, seed, --as-of and options still give the same bytes on any machine and any day with 1.16.0. Pin model2data==1.14.0 (or the Action's version: 1.14.0) to keep fixtures of the old shape, or regenerate and commit the new ones.

Full changelog: v1.14.0...v1.16.0

v1.15.0

Not on PyPI. This release's package still said 1.14.0, so the upload failed and pip install model2data==1.15.0 does not exist. These fixes ship in 1.16.0.

Byte columns get short hex values, and a key that references a column with too few distinct values now caps its table instead of failing its own unique test. Both were found by running ERD Studio's DBML export through model2data. Some models generate different data for the same seed: see Upgrading below.

Fixed

• A binary column holds a short value. binary is also the name of a Faker provider, and
  the generic fallback called it, which returns a whole MiB of random bytes per value: about 3 MB
  of text per CSV cell. A 50-row table with two binary hash keys wrote a 391 MB seed that dbt's
  seed loader refused outright. binary, varbinary, bytea, blob and bytes columns now
  hold 16 random bytes from the seeded stream, written as 32 hex characters (an MD5-sized hash
  key), and stay unique when they are a key. The seed column loads as VARCHAR. varbinary,
  bytea and blob used to fall back to placeholder text and no longer appear among the
  generic-fallback columns.

• A key that references a column with fewer distinct values caps its table. A pk or
  unique column referencing another table's column takes each parent value once, so a table
  asked for more rows than that column holds distinct values repeated some, and failed its own
  generated unique test. The commonest source is a reference drawn the wrong way round, such
  as Ref: dim_product.product_key > fct_order.product_key. The table now gets one row per
  distinct parent value, and the summary says so:

  ⚠️  Tables generated with fewer rows than asked, so a unique reference keeps one row per parent value:
    - dim_product: 35 rows, not 50 (product_key is a unique reference to fct_order.product_key, which holds 35 distinct value(s))
  

  This is what generate --days already does when a one-to-one reference runs out of parents.
  A self-reference, or a reference into a table generated later in a cycle, is not capped.

Upgrading

Output changes for the same seed in two cases:

• A table with a byte column (binary, varbinary, bytea, blob, bytes): the byte
  column's values, and the values of the columns drawn after it in that table. Its ids, its null
  pattern and every other table are unchanged.
• A table whose key references a column short of distinct values: it has fewer rows, and the
  tables that reference it draw from those rows. These runs failed their own unique test
  before, so no passing project is affected.

A model with neither writes the same bytes as in 1.14.0. Committed fixtures from 1.14.0 that
include a byte column will differ when regenerated. Pin model2data==1.14.0 (or the Action's
version: 1.14.0) to keep them, or regenerate and commit the new ones.

Full changelog: v1.14.0...v1.15.0

v1.14.0

Added

• Lightdash metrics in the generated dbt project. generate --lightdash writes the model's
  metrics, dimensions and joins where Lightdash
  reads them: under config.meta of each staging model's own models/staging/stg_<table>.yml,
  next to its tests (dbt reads a model's properties from one file only, so the meta is merged in
  rather than written beside it). It takes the --metrics file's metrics, or without one the
  metrics the model implies. Each table is joined many-to-one to every table it reaches along
  exactly one path; a dimension gets its label and a column that is not one is hidden; a simple
  metric sits on its column, a row count is a count_distinct of the one-column primary key, and
  a ratio or expression is a type: number metric over its inputs; ai_context becomes
  ai_hint, percent and currency the formats '0.00%' and '#,##0.00'. A filter becomes
  the metric's filters wherever Lightdash's filter grammar means exactly the same (with
  '!null' beside a not-equal, since Lightdash's lets nulls through), and a CASE WHEN in its
  sql where it does not (an any group, a date comparison), so the metric counts what its
  known value counts. What Lightdash has no place for (an enum's members, a table's role and
  grain, a currency, a metric dated by another table's column, a ratio whose inputs are on two
  tables, which is left out) is listed in the run's summary. Every example's output validates
  against Lightdash's published JSON Schema for dbt YAML, parses with dbt, and compiles with
  lightdash compile --no-partial-compilation.
• model2data metrics export --to lightdash writes the same meta as one dbt properties file
  for the staging models, without generating, the losses listed at its top.
• model2data.metrics.to_lightdash and write_lightdash in the Python API, beside
  to_ossie.

Without --lightdash, a run writes byte for byte what 1.13.0 wrote, with or without
--metrics.

v1.13.0

Added

• Metrics next to the model. A second, optional file beside the model,
  <stem>.metrics.yml, says what the model's numbers mean, defined by its own spec, metrics
  spec 0.1.0, with a JSON Schema at
  https://www.jbanalytica.com/model2data/spec/metrics/0.1.0/metrics.schema.json. A metric is a
  column aggregated (measure: orders.total_amount, by agg or the column's own measure), a
  row count (count: orders), a ratio of two metrics' totals (ratio: {numerator, denominator}),
  or arithmetic over metrics (expression: revenue - refunds). Simple metrics and row counts take
  a where filter (a value, a list, or eq, ne, in, not_in, gt, gte, lt, lte,
  between, is_null, combined with any and all, with nulls as in SQL) and a time column;
  a filter may name a column of any table the metric's table reaches through its foreign keys,
  many-to-one, along exactly one path. Every measure column of the model also yields a metric of
  its own (orders_total_amount) and every fact a row count (orders_count), named as model2data
  studio's semantic export names them; infer: false turns that off, and a metric of the same
  name in the file replaces the inferred one. format: percent expects a fraction (0.6 shows as
  60%) and is passed on as a percent format, never multiplied. dimensions: offers a column as a dimension, with a
  label, or never. The generator never reads the file: a model generates the same data with
  or without one.
• model2data validate checks metrics files (*.metrics.yml): the schema, then against the
  model, --model, or else the model beside it with the same stem. It names every unknown table
  or column, an aggregation that does not fit its column, an unknown metric in a ratio or
  expression, a cycle between metrics, a filter column no path reaches or two paths reach (naming
  both), a value that is not a member of its enum or not of its column's type, and a comparison
  on an enum, boolean or text column; it warns about a metric with no date to report it by and a
  filter that can never match the generated data (a gt above a column's max).
• generate --metrics FILE adds, and changes no other file:
  metric_values.json, each metric's known value over the generated data, overall and per
  calendar month (counts exact, everything else rounded to 6 decimal places, half to even; the
  same seed gives the same bytes); a singular dbt test per metric,
  data-tests/metrics/metric_<name>.sql, that computes it over the project's staging models and
  fails unless it gets that value, so dbt build proves the metric logic on the warehouse, with
  defects and --days too; and osi/<project>.yml, the model and its metrics as an Apache Ossie
  (Open Semantic Interchange) 0.1.1 semantic model, as tagged osi-0.1.1-rc1 in apache/ossie (no
  final 0.1.1 tag exists yet; 0.2.0 is an unreleased draft). A
  metrics file beside the model is not picked up on its own: an unchanged command keeps writing
  the same files.
• model2data metrics list shows every metric, inferred ones included, and model2data metrics export --to ossie writes the Ossie file without generating. What Ossie 0.1.1 has no
  field for (a metric's label, format and time, an enum's members, a table's role and grain) goes
  in custom_extensions and is listed at the top of the file.
• model2data.metrics, the Python API: load and validate a metrics file, resolve it with
  its model into one semantic representation (entities, relationships, dimensions, measures and
  metrics with their join paths), known_values over a run's frames, to_ossie, and
  write_metric_tests.
• examples/ecommerce.metrics.yml: revenue, orders, average order value and units sold for
  the e-commerce example, for --metrics.

Changed

• The spec's example model, model2data/spec/examples/coffee_webshop.model2data.yml, marks
  orders.total_amount and order_items.quantity as measures, for the metrics example beside
  it. No model under examples/ changed, and no output changed for any model.
• The README and the PyPI page were rewritten to show what model2data does today: what you
  get, one capability per line with a link; a before and after of a real run with metrics; who
  it is for; and how it works, with a new demo GIF. The reference that was in the README moved to
  docs/, one page per topic (the model file, generating, the dbt project, days, history, time
  shapes, distributions, defects, metrics, CI, agents), worded as it was.

v1.12.0

A column placed after another now keeps its time shape, and no longer piles rows up on --as-of midnight. Both fixes change what some models generate for the same seed: see Upgrading below.

Fixed

• A column placed after another keeps its time shape. A date or timestamp column with an
  after, or one a created/updated/closed name pair places after another, was set to that
  column plus a random gap, which threw away the draw its business_hours had shaped:
  appointments after a booking with business_hours: true came out at 3am as often as at
  10am. The gap still decides roughly when; business_hours (the column's own, or the run's)
  now decides the weekday and hour. The share of such a column in weekday working hours goes
  from about 29%, the rate of a uniform draw, to about 72%, the same as a column drawn on its
  own. The column's growth and seasonality still come from the column it follows.
• No pile-up at --as-of midnight. A gap that ran past the end of the window set the value
  to exactly --as-of midnight, so about 1% of a dependent column's rows shared that one
  timestamp. Such a gap is now drawn again, and after 20 tries the value is drawn evenly
  between the column it follows and the window's end. A when column filled in on day 0 is
  placed the same way.

Changed

• The determinism promise is per version. The same model, seed, --as-of and options still
  give the same files, byte for byte, on any machine and any day, with the same model2data
  version. A new release may change what a seed produces, and when it does the changelog says
  which models are affected and why. (The README promised the same bytes across every 1.x
  release.)

Upgrading

Output changes for the same seed in a table with a column placed after another, wherever either
fix applies: a dependent column with business hours (its own or the run's), and every table
where a row's gap ran past --as-of (about 1 row in 100, so most such tables beyond a few
hundred rows). Values drawn after the first such row in a table change too. A table with no
column placed after another writes the same bytes as in 1.11.0.

Committed fixtures from 1.11.0 will differ when regenerated. Pin model2data==1.11.0 (or the
Action's version: 1.11.0) to keep them, or regenerate and commit the new ones.

Full changelog: v1.11.0...v1.12.0

v1.11.0

Added

• model2data guide [setup|triage|tune] prints one page of Markdown written for a coding
  agent about to use model2data: setup (install, a starter model, the first run), triage
  (what validate, generate and dbt build print and what to do about each, in order) and
  tune (every option, model key and exit code). Without a topic it picks triage when the
  current directory holds a *.model2data.yml or *.dbml file, else setup, and says so on
  its first line. The pages ship in the package, so they match the installed version; the
  command reads only file names in the current directory and writes nothing.
• Spec 0.4.0: a column can depend on another column's value, with when. completed_at with
  generate: {after: created_at, when: {status: [done]}} holds a timestamp on every done task
  and is null on every other one, where a todo task used to get a completed_at and a
  cancelled subscription none. when maps another column of the table (an enum, boolean,
  number or text column) to the values it must hold; several columns must all match, and a row
  whose named column is null does not. On the matching rows the column is generated as before
  and never null, unless it has a null_rate, which then counts only those rows. It composes
  with after, the time shape hints and distribution. On days after the first, a row an update
  moves into a listed status gets its value that day (its updated_at when the table has one),
  and a row moved out loses it. Each when writes a dbt test, model2data_when (and a
  null_rate beside it model2data_when_max_null_share, over the matching rows), shipped in
  macros/model2data_when_tests.sql, and the defects report checks it like any other. The
  model's checks say what is wrong with one: a column that is not there (and which one was
  meant), a value its enum does not have, a not_null, keyed, unique, foreign-key or defaulted
  column carrying it. In DBML it is a note hint like the others:
  [note: '{"when": {"status": ["done"]}}']. when is spec 0.4.0, which adds it and nothing
  else: a model using it says model2data: 0.4.0 (DBML with a when note converts to 0.4.0),
  when in a 0.2 or 0.3 document is an error saying to write 0.4.0, and 0.2 and 0.3 documents
  read exactly as before. A writer keeps the version a document was read with, and writes 0.4.0
  once it uses when. The schema's $id is now
  https://www.jbanalytica.com/model2data/spec/0.4.0/model.schema.json (0.2.0 and 0.3.0 stay
  served, see model2data/spec/PUBLISHING.md); the spec README, model2data guide and LLMS.md
  describe it.

Fixed

• A foreign key cycle with a nullable link is generated, not flagged. Tables were ordered by
  every foreign key, so accounts.owner_user_id -> users.id beside users.account_id -> accounts.id left both tables in "an unresolved FK cycle", whatever their nullability, and one
  side pointed at rows that did not exist. A cycle is now broken at a table whose foreign keys
  into it are all nullable: that table is generated first, and those columns are drawn once
  their parent exists, from its rows. Only a cycle of required foreign keys is still reported.
• Determinism: models without when generate exactly what they did before, dbt project
  included. In a model with when, only its columns change: what when decides is drawn from
  streams of its own, so every other column comes out as it does without the hint, except a date
  or timestamp that follows a when column (after it, or a later stage), which moves on a row
  where it would otherwise be earlier than the value when filled in.
• Determinism: a model without an FK cycle generates exactly what it did before. A model
  whose cycle had a nullable link now writes different foreign key values in that cycle's tables
  (valid ones, where some used to dangle), and different columns mirrored from them.
• PostgreSQL serial types are integers. serial, serial2, serial4, serial8,
  smallserial and bigserial were not recognised as integer types, so a bigserial primary
  key was filled with UUIDs and a bigint foreign key onto it failed with invalid literal for int() with base 10. They now count as integers everywhere an int type does (generation,
  min/max/distribution hints, the Int64 dtype, defect kinds, dbt seed column types), and
  a serial key generates exactly what the same model typed integer/bigint/smallint does.
  The declared type is kept as written.
• Determinism: only a model with a serial-typed column generates differently (it crashed or
  produced UUIDs before); every other model is byte-identical.

v1.10.4

Fixed

• A unique enum column gets distinct members. It was drawn like any enum, with replacement,
  so plan: [free, starter, pro, enterprise] on a four-row table could write starter twice and
  leave out pro, and the summary said nothing until dbt's unique test failed. A repeated
  member is now swapped for one no row holds yet. With fewer members than rows the remaining
  repeats are listed under "Unique columns left with duplicate values", like a numeric range too
  narrow for its rows. On days after the first, a new row that repeats a member the table already
  holds takes a free one instead of a suffixed value the enum does not have (starter2); with
  none left it keeps the repeat and the day's warnings say so.
• Determinism: only a repeat is drawn again, so a seeded run whose unique enum columns already
  came out distinct writes the same files as 1.10.3. A run that had repeats in one, whose
  generated unique test therefore failed, now writes different values from that column on.