Skip to content

0.10.0

Choose a tag to compare

@ekut ekut released this 03 Nov 04:03
· 47 commits to main since this release
e9e129b

Ossie schema 0.2.0.dev0. Tested with dactopus-ossie-clickhouse 0.12.0,
dbt v2 (dbt-oss 2.0.8) and dbt v1 (dbt-core 1.11 with
dbt-clickhouse 1.10) on ClickHouse 26.9, the WooCommerce package also
on 26.7. The unions' views read the tables they add up with the rights
of the user who built them, so a user that reads a model needs SELECT
on the union's database alone; where row policies on those tables hide
rows from the reader, the numbers through a union change.
deployment.yml names the zone of the days of a package that fixes it.
No table changes and no rebuild is needed: a union changes on its next
build.

Added

  • zone in deployment.yml: the zone of the days of a package that
    fixes it in its model, so a deployment no longer has to be told it.
    The Sensor.Community package declares zone: UTC; Shopify and
    WooCommerce take the zone from the variable with the role time_zone,
    and GA4's days are in the property's zone; none of the three has the
    key. zone and time_zone exclude each other, and a union's file has
    no zone.
    tests/check_deployment.py requires an IANA name and counts in every
    daily table the rows whose date is not the day of their time in
    zone, with or without --time-zone, and fails on any such row or on
    a --time-zone that differs
    (#60).

  • tests/<source>/canon/ and canon_windows/ for Shopify and
    WooCommerce: the canonical tables of entities/commerce.yaml (the
    columns the model reads, in the order of the contracts) over
    fixture.sql, and over fixture.sql with windows.sql, one CSV file
    a table, with tests/commerce/tables.sql, their DDL in standard SQL.
    The questions and their expected rows are in
    tests/<source>/questions.yaml. tests/run_canon.py loads the files
    into ClickHouse and asks the questions through dactopus-ossie-clickhouse,
    so the model's numbers can be checked without dbt or the platform's raw
    rows; --engine duckdb asks them through DuckDB's community extension
    ossie instead. Its 0.3.0 does not read the metrics of a dataset
    (apache/ossie#591), so the model does not load there as it is:
    tests/run_canon.py loads a copy with them moved to the model's metrics. With that
    copy, 72 of the 95 questions give the same
    numbers, 19 are refused, and 4 differ: it leaves out the rows a
    many-to-one relationship finds no match for (by products.status, the
    lines of products the products table does not have; by
    customers.first_order_month, the guests' orders). tests/export_canon.py writes
    the files from a built package, and CI fails when a build differs from
    them.

Changed

  • The distinct counts that pair source_id with the counted column
    (orders.buyer_count, product_categories.product_count,
    product_brands.product_count, sessions.users,
    sessions.user_conversion_rate, readings.reporting_devices) join the
    two: COUNT(DISTINCT source_id || '/' || CAST(customer_id AS VARCHAR)).
    COUNT(DISTINCT source_id, customer_id) is not standard SQL, and DuckDB
    rejects it. The numbers are unchanged.
  • The questions of tests/shopify/check_numbers.py and
    tests/woocommerce/check_numbers.py moved to questions.yaml, with
    their derivations as comments; the two scripts read it and need
    pyyaml (uv run installs it). The questions and their numbers are
    unchanged.
  • The example databases and labels in docs/deployment.md name what
    the store is, not its platform: databases main_store, outlet_store,
    wholesale_store, labels usa_store, wholesale_store.
  • The unions' views read the tables of the stores, properties and
    networks with the rights of the user who built them
    (+sql_security: definer, +definer: CURRENT_USER). A user with
    SELECT on a union's database reads its model without a grant on the
    databases it adds up, and those stay closed to it. Queries through a
    view go under the builder's row policies on those tables, no longer
    the reader's: this changes the numbers of a deployment whose row
    policies there hide rows from the reader. A union changes on its next
    build
    (Several stores, properties or networks).