Repository navigation
0.10.0
Ossie schema 0.2.0.dev0. Tested with dactopus-ossie-clickhouse 0.12.0,
dbt v2 (dbt-oss 2.0.8) and dbt v1 (dbt-core 1.11 with
dbt-clickhouse 1.10) on ClickHouse 26.9, the WooCommerce package also
on 26.7. The unions' views read the tables they add up with the rights
of the user who built them, so a user that reads a model needs SELECT
on the union's database alone; where row policies on those tables hide
rows from the reader, the numbers through a union change.
deployment.yml names the zone of the days of a package that fixes it.
No table changes and no rebuild is needed: a union changes on its next
build.
Added
-
zoneindeployment.yml: the zone of the days of a package that
fixes it in its model, so a deployment no longer has to be told it.
The Sensor.Community package declareszone: UTC; Shopify and
WooCommerce take the zone from the variable with the roletime_zone,
and GA4's days are in the property's zone; none of the three has the
key.zoneandtime_zoneexclude each other, and a union's file has
nozone.
tests/check_deployment.pyrequires an IANA name and counts in every
daily table the rows whose date is not the day of their time in
zone, with or without--time-zone, and fails on any such row or on
a--time-zonethat differs
(#60). -
tests/<source>/canon/andcanon_windows/for Shopify and
WooCommerce: the canonical tables ofentities/commerce.yaml(the
columns the model reads, in the order of the contracts) over
fixture.sql, and overfixture.sqlwithwindows.sql, one CSV file
a table, withtests/commerce/tables.sql, their DDL in standard SQL.
The questions and their expected rows are in
tests/<source>/questions.yaml.tests/run_canon.pyloads the files
into ClickHouse and asks the questions through dactopus-ossie-clickhouse,
so the model's numbers can be checked without dbt or the platform's raw
rows;--engine duckdbasks them through DuckDB's community extension
ossieinstead. Its 0.3.0 does not read the metrics of a dataset
(apache/ossie#591), so the model does not load there as it is:
tests/run_canon.pyloads a copy with them moved to the model's metrics. With that
copy, 72 of the 95 questions give the same
numbers, 19 are refused, and 4 differ: it leaves out the rows a
many-to-one relationship finds no match for (byproducts.status, the
lines of products theproductstable does not have; by
customers.first_order_month, the guests' orders).tests/export_canon.pywrites
the files from a built package, and CI fails when a build differs from
them.
Changed
- The distinct counts that pair
source_idwith the counted column
(orders.buyer_count,product_categories.product_count,
product_brands.product_count,sessions.users,
sessions.user_conversion_rate,readings.reporting_devices) join the
two:COUNT(DISTINCT source_id || '/' || CAST(customer_id AS VARCHAR)).
COUNT(DISTINCT source_id, customer_id)is not standard SQL, and DuckDB
rejects it. The numbers are unchanged. - The questions of
tests/shopify/check_numbers.pyand
tests/woocommerce/check_numbers.pymoved toquestions.yaml, with
their derivations as comments; the two scripts read it and need
pyyaml(uv runinstalls it). The questions and their numbers are
unchanged. - The example databases and labels in
docs/deployment.mdname what
the store is, not its platform: databasesmain_store,outlet_store,
wholesale_store, labelsusa_store,wholesale_store. - The unions' views read the tables of the stores, properties and
networks with the rights of the user who built them
(+sql_security: definer,+definer: CURRENT_USER). A user with
SELECTon a union's database reads its model without a grant on the
databases it adds up, and those stay closed to it. Queries through a
view go under the builder's row policies on those tables, no longer
the reader's: this changes the numbers of a deployment whose row
policies there hide rows from the reader. A union changes on its next
build
(Several stores, properties or networks).