0.9.0

Ossie schema 0.2.0.dev0. Tested with dactopus-ossie-clickhouse 0.12.0,
dbt v2 (dbt-oss 2.0.8) and dbt v1 (dbt-core 1.11 with
dbt-clickhouse 1.10) on ClickHouse 26.9, the WooCommerce package also
on 26.7. Several stores, properties or networks in one deployment: every
table starts with the label source_id, orders carries platform,
and the unions add up the tables of several databases. Every column has
a declared type, and some types change: an incremental run stops on the
changed schema, and a deployment runs dbt build --full-refresh once.
No input table changes.

Added

• Every table of every package has a first column source_id, the
  label of the store, property or network its rows come from: the
  variable ga4_source_id, shopify_source_id, woocommerce_source_id
  or sensor_community_source_id, LowCardinality(String). The default
  is the package's name (shopify); a label other than 1 to 64 of
  a-z, 0-9 and _ stops the build. A dbt test on every model fails
  on a row of another label, as an incremental run after a new label
  leaves, and names --full-refresh. deployment.yml maps the role
  source_id to the variable, and tests/check_deployment.py requires
  it.
• The Ossie models have a field source_id in every dataset. It comes
  first in every primary_key and on both sides of every relationship,
  and every count of distinct values counts them within a label
  (orders.buyer_count, product_categories.product_count,
  product_brands.product_count, readings.reporting_devices,
  sessions.users, sessions.user_conversion_rate): the tables of
  several stores, each built with its own label, keep their orders and
  customers apart. In one database the label is constant and every
  number stays the same. Its description and synonyms (store, shop;
  property, site; network) let an agent find it, and each model's
  instructions say how to answer over several: for a store, property or
  network the question names or describes, an agent lists the labels and
  takes the one that names it (for a store, also the one store in a
  currency the question names), and asks when none does; a question that
  names none means all of them; identifiers repeat across them, so an
  agent groups by one with its source_id; "today" is the date where the
  model is served, while each one's dates are in its own zone. GA4 users
  count within a property and are not matched across properties, and a
  sensor network's silent devices count from its own latest contact.
• orders has a column platform, shopify or woocommerce, after
  currency, and the commerce model a field orders.platform: an agent
  over stores of both platforms finds which store a question about
  "the Shopify store" means, and which platform's report rules apply to
  a store. The unions carry it.
• Unions: unions/commerce, unions/web_analytics and unions/iot, dbt
  projects that add up the tables of several stores, properties or
  networks, each built by its package into a database of its own under
  its own label. A union writes a view of each dataset of the domain's
  Ossie model, UNION ALL over the databases of
  commerce_union_databases, web_analytics_union_databases or
  iot_union_databases, with the columns the model reads and their
  types as a dbt contract. A commerce union takes Shopify and
  WooCommerce stores alike. The build stops on a database whose column
  types differ from the contract and on the union's own database among
  those it adds up; its test fails when a label is in two databases.
  deployment.yml maps the role input_databases to the variable.
  tests/check_contracts.py checks the unions' contracts against the
  Ossie models and the packages, that source_id comes first in every
  key and relationship of the Ossie models, that each package tests the
  key of each dataset (Sensor.Community's readings with
  sensor_community_unique_key, which replaces the test
  readings_one_per_sensor_time_measure), and that the copies of
  macros/union.sql and of each package's macros/freshness.sql are
  the same; tests/check_union.py checks that each label's answers over
  a union are its database's own.

Fixed

• WooCommerce: an incremental run over a table whose columns a release
  changed stops before its pre-hooks delete a row. dbt's own check of
  the columns comes after the hooks: the run stopped with order_lines
  and refunds short of the rows of its window, refunds empty on the
  fixture.
• WooCommerce: category and brand names (line_categories.category,
  category_path, line_brands.brand and the same fields of
  product_categories and product_brands) are decoded from the HTML
  escaping WordPress stores them with: Home & Garden, not
  Home & Garden. No table changes: the next run rebuilds the
  category and brand tables whole.

Changed

• CI runs a job only when a file it reads changed (the filters in
  ci.yml): a pull request that changes only documentation runs none,
  one that changes one package runs that package's jobs and the unions
  job. The Ossie models are validated together in a job of their own.
  The one check main requires is CI passed, which fails when a job
  failed or was cancelled; a skipped job does not block a merge.

• CI runs on ubuntu-24.04, not ubuntu-latest, and keys the uv cache
  by ci.yml: before, the key was constant and the cache was never
  refreshed.

• AGENTS.md keeps what an agent applies while writing and is not in
  CONTRIBUTING.md: the rules of source_id, contracts and unions, the
  model format, ClickHouse traits, which page a change ships with, and
  the development setup. Layout, scope, the authoring rules and the
  checks before a pull request are read from CONTRIBUTING.md, whose
  canon rule gains the two details only AGENTS.md had: a source that
  has less of a field still provides it when the meaning is the same,
  and no canonical field goes in "for completeness".

• The README is split. Each package's guide (loading, settings, known
  differences from the source's own reports) is sources/<source>/README.md;
  what each Ossie model answers and its limits is docs/<domain>.md;
  the packages in a dbt project, labels, time zones, serving and unions
  are docs/deployment.md; layout, what a source package is, out of
  scope and the authoring rules are in CONTRIBUTING.md. The README keeps
  the overview, one quick start and versioning. Notes on how the
  packages were checked against test stores, and the migration note for
  snapshots loaded before 0.5.1, are gone from the guides: the checked
  numbers are stated, the history is this changelog. The install
  recipes take dactopus-ossie-clickhouse from PyPI.

• The commerce model's instructions: one deployment can hold several
  stores, under their labels, in different currencies. An agent asks
  every amount by orders.currency, answers in the one currency or in
  each, and never adds up two; amounts of customers go by
  customers.source_id. orders.currency and customers.net_amount
  say the same. A customer of two stores counts in each.

• README, Known differences from WooCommerce Analytics: buyers in a
  month, where the reports also count a customer with only a refund.

• Every package declares the type of each column it writes, a dbt
  contract in models/schema.yml, and a build stops on any other. The
  columns an Ossie model reads come first in each table, with the same
  types in every package that serves the model: the tables of Shopify
  and WooCommerce add up with UNION ALL. tests/check_contracts.py
  checks it in CI. No value changes; these types do:

  • Amounts are Decimal(38, 8) and never NULL: orders.total,
    subtotal, discount_total, shipping_total, tax_total,
    refunded_total, net_total; customers.net_total; line_total
    and returned_amount of order_lines, line_categories and
    line_brands; refunds.refunded_amount; sales_events.amount.
    Shopify's were Decimal(18, 4) (customers.net_total
Decimal(38, 4)), and Shopify's order_lines.discount_total
    follows; WooCommerce's were Nullable(Decimal(38, 8)), all but
    orders.refunded_total.
  • orders.customer_id is Nullable(Int64), customers.customer_id
Int64: Shopify's were Nullable(UInt64) and UInt64,
    WooCommerce's orders.customer_id Int64.
  • Shopify's quantities are Int64: orders.item_quantity;
    quantity and current_quantity of order_lines,
    line_categories and line_brands. They were UInt32 and
    UInt64.
  • WooCommerce's orders.order_time, refunds.refund_time and
    sales_events.event_time are DateTime('UTC'), not
    DateTime64(6, 'UTC'): WooCommerce keeps whole seconds.
  • Flags are Bool: orders.is_cancelled and
    customers.second_order_within_*_days (Nullable(Bool)) of both
    packages, WooCommerce's orders.in_sales_reports and is_direct of
    line_categories, product_categories and
    product_categories_all. They were UInt8.
  • Shopify's order_lines.title is String, not Nullable(String).

  The columns of every table come in a new order, the model's first.
  With source_id (Added), every package runs once with
  --full-refresh in place of the next incremental run, which stops.

• GA4: on_schema_change: fail, as in the other packages; dbt requires
  it of an incremental model with a contract.

• WooCommerce: an order or refund in another currency without a rate
  the package knows now fails in its model, with a message that names
  the README's Known limits, where before the not_null tests on its
  amounts failed.

• The commerce model reads its Bool flags alone, without = 1
  (orders.is_cancelled, orders.in_sales_reports,
  customers.second_order_within_*_days and the
  customers.second_order_rate_*_days metrics, is_direct of
  line_categories and product_categories): Postgres has no operator
  for Bool = Int, so the model now translates to engines other than
  ClickHouse. The numbers are the same. tests/check_contracts.py fails
  on an expression that compares a Bool column with a number.

• The commerce model's instructions state each rule that depends on the
  platform from the field that carries it, with Shopify and WooCommerce
  as examples: which orders the sales reports count
  (orders.in_sales_reports), the order date (orders.order_date), a
  refund of 0, who a customer is (customers), what a category and a
  brand are (line_categories.category, line_brands.brand). The
  rules are the same.

• The web_analytics model's instructions say when GA4 exports a day:
  once, usually in the afternoon of the following day. The days after
  the latest date with rows have not arrived yet, and an agent says up
  to which date the data reaches instead of reporting them as days
  without visits; the latest date is a whole day, and a low count there
  is not a partial export
  (#48).
  docs/web_analytics.md says the same.

• ROADMAP.md: Magento (Adobe Commerce) under Exploring, a store whose
  MySQL database reaches ClickHouse by CDC, as WooCommerce's does.