[IDEA & PROPOSAL] Ossie Extension: Shared Filters, Shared Dimensions & Metric References #342
Replies: 6 comments 2 replies
|
|
Building on For example: dimensions:
- name: country
datatype: String
values:
- { value: "United States", name: "US" }
- { value: "China", name: "China" }
datasets:
- name: customer
source: sales.public.customer
fields:
- name: c_birth_country
expression:
dialects:
- dialect: ANSI_SQL
expression: "c_birth_country"
dimension:
ref: country
# Proposed: reference the logical dimension instead of a physical column
filters:
- name: domestic_customer
dimension: country
expression:
dialects:
- dialect: ANSI_SQL
expression: "${country} = 'United States'"The compiler could resolve This would keep the Shared Dimension's value vocabulary as the single source of definition while avoiding equivalent raw SQL filters being re-encoded for each dataset. How should this behave if multiple fields in the same dataset reference the same Shared Dimension, or if different datasets use different physical encodings for the same logical value? |
|
Thanks for the detailed write-up, and for the follow-ups to @yang85470-afk and I have comments on all five changes, but they're long enough that I'll split them across a ⑤ Metric references — overlaps with PR #343, and I think there's a clean mergePR #343 (dataset-scoped metrics) raises (a) Qualify the reference inside
|
| Referencing metric | May reference |
|---|---|
| Model-scoped | Any metric, model- or dataset-scoped |
| Dataset-scoped | Only metrics in the same dataset |
A dataset-scoped metric referencing a model-scoped one would break the self-containment
property that makes dataset scoping useful, so I'd disallow it. Either PR can land first;
whichever is second should state this.
(c) Acyclicity
Not mentioned in the write-up. Suggested normative text:
The metric reference graph MUST be acyclic. A metric MUST NOT reference itself, directly
or transitively. Validators MUST reject cyclic reference graphs.
(d) Interaction with hidden (PR #287)
PR #287 adds a hidden flag. Combined with ⑤
that gives the intermediate-calculation pattern, which several implementations support:
metrics:
- name: total_cost_basis
hidden: true # exists only to be composed, not exposed to consumers
expression:
dialects:
- dialect: ANSI_SQL
expression: "SUM(store_sales.ss_wholesale_cost)"
- name: gross_margin_pct
expression:
dialects:
- dialect: ANSI_SQL
expression: "(${total_sales} - ${total_cost_basis}) / NULLIF(${total_sales}, 0)"Worth confirming explicitly that referencing a hidden metric is legal — otherwise
implementations will differ on it.
(e) On aggregation scope
Your answer to @yang85470-afk — each referenced metric evaluated in its own aggregation scope
via conditional aggregation or subqueries — is more developed than #343's sketch, and I think
it's correct. It's also the part most likely to diverge between implementations, so I'd put
it in the spec as a requirement rather than leaving it to the compiler:
A referenced metric MUST be evaluated within its own aggregation scope, including any
filters attached to it. A derived metric's filters MUST NOT be applied to the expressions
of the metrics it references.
I think this one we should do, then I can update my metrics proposal as well.
⑤ is the best-precedented change in the proposal — it exists in dbt MetricFlow, in Cube, in
LookML, and in Snowflake's semantic view YAML (where model-level derived metrics reference
table-scoped metrics by qualified name). The concept matches everywhere; only the syntax
differs. It also has an Ossie PR touching adjacent ground already. If the proposal needs to
be split for reviewability, this is the piece I'd separate out and merge first.
|
Continuing on ① and ② — the filter changes. Problem 1 is real and I've hit it too. Recommendation: use the existing
|
|
On ③ and ④. The cross-model problem is real. I'm less sure about the mechanism, and I wanted to ③/④ moves the duplication instead of removing itIf So this doesn't really give you a single source of truth for dimension semantics. I tried Problem 2 with the current spec, does this hold up?I wanted to see how far today's spec gets on your example before adding anything, so I tried datasets:
# The definition lives here, once.
- name: order_status
source: sales.public.order_status_lookup
primary_key: [status_code]
fields:
- name: status_code
datatype: String
description: "Lifecycle state of an order"
ai_context: "PAID=paid, SHIPPED=shipped, CANCELLED=cancelled"
expression:
dialects:
- dialect: ANSI_SQL
expression: "status_code"
dimension:
is_time: false
# Both fact datasets now carry a plain foreign key and nothing else.
- name: transactions
source: sales.public.transactions
fields:
- name: status
datatype: String
expression:
dialects:
- dialect: ANSI_SQL
expression: "status"
- name: orders
source: sales.public.orders
fields:
- name: order_status
datatype: String
expression:
dialects:
- dialect: ANSI_SQL
expression: "order_status"
relationships:
- name: transactions_to_order_status
from: transactions
to: order_status
from_columns: [status] # the degenerate dimension is the foreign key
to_columns: [status_code]
- name: orders_to_order_status
from: orders
to: order_status
from_columns: [order_status]
to_columns: [status_code]As far as I can tell this holds up. I validated the datasets and relationships above against the current schema, but schema-valid There is a real cost here. It adds a dataset and two relationships, which for three enum values One question on the example itself. The comment says the orders dataset is "maintained by another Proposal: defer ③/④ to a composability discussionLanding a model-level So I'd pull ③/④ out and take ①/②/⑤ forward on their own. They stand up independently and don't I'd frame the separate discussion around DRY specifically, meaning how a definition written once On
|
|
Coming back to ③/④, because I re-read your Problem 2 example and I think I under-read it the first The actual problem is a shared degenerate dimension: Correction to my earlier exampleI wrote datasets:
- name: order_status
source: >-
SELECT DISTINCT status AS status_code FROM sales.public.transactions
UNION
SELECT DISTINCT order_status AS status_code FROM sales.public.orders
primary_key: [status_code]
fields:
- name: status_code
datatype: String
description: "Lifecycle state of an order"
ai_context: "PAID=paid, SHIPPED=shipped, CANCELLED=cancelled"
expression:
dialects:
- dialect: ANSI_SQL
expression: "status_code"
dimension:
is_time: falseRelationships from each fact stay as I had them. The UNION isn't cosmetic: if This is the standard answer across the ecosystem, so I don't think it's a hack. dbt uses a seed or The part I missed: "shared dimension" is two different needsThis is what I'd now push on in ③/④. Need 1, shared metadata. Need 2, a shared queryable dimension. One dimension any metric can be grouped by, whichever fact ③/④ delivers Need 1 and not Need 2. If I think Need 2 is the harder and more valuable half, and a Why I no longer think the workaround is good enoughMy earlier reply leaned on "this is solvable today, so defer it." The mechanism does work, but I
That last point is what changed my mind. The workaround works by putting the problem somewhere the Proposal: make conformance declarative rather than adding a dimensions registryIf this becomes first-class, I'd rather state the conformance directly and let an implementation datasets:
- name: order_status
# Instead of hand-written UNION SQL in `source`.
conformed_from:
- transactions.status
- orders.order_status
primary_key: [status_code]
fields:
- name: status_code
datatype: String
description: "Lifecycle state of an order"
ai_context: "PAID=paid, SHIPPED=shipped, CANCELLED=cancelled"
expression:
dialects:
- dialect: ANSI_SQL
expression: "status_code"Why this shape over ③/④:
Open questions I don't have good answers to: whether relationships should be inferred from Where this leaves my earlier recommendationI'd still not merge ③/④ as it stands, but the reason is better than the one I gave. It isn't that the So: split the shared-dimension piece out and give it a focused discussion, framed as the degenerate Before that, I'd want your read, since you've built far more models against this spec than I have. |
Uh oh!
There was an error while loading. Please reload this page.
Our team is adopting Ossie as the semantic modeling layer for our enterprise data platform. After building dozens of semantic models for TPC-DS and our internal data warehouse, we've found Ossie's dataset/metric/relationship model to be an excellent foundation. However, as we scaled to production, we hit three major pain points:
Problem 1: No Filtering Mechanism for Metrics
In practice, almost every metric needs filtering — "valid sales only", "domestic customers", "last 12 months". Currently, Ossie Metrics only support
expression, forcing filter logic to be hardcoded into SQL:As the number of metrics grows, the same filter logic gets copied N times. Any definition change (e.g., adding
payment_verified = true) requires locating and updating every single occurrence.Problem 2: Duplicate Dimension Definitions — Spec Lacks Dimension-Level Business Semantics
The same business concept —
order_status,sales_region,customer_segment— appears across fields in multiple datasets. Currently, Ossie'sdimensionobject only supports an is_time flag, offering no way to capture richer business semantics..datatype,description, andai_contextcan only be defined as field-level attributes, and enumerated values (values) cannot be expressed at the dimension level at all:When the same dimension surfaces across multiple datasets, these scattered definitions are highly prone to drift, causing BI tools to display contradictory metadata.
Problem 3: Derived Metrics Require Full Expression Duplication
For derived metrics like
profit_margin = total_profit / total_sales, the only option today is to copy and paste the full SQL expressions of the base metrics. There's no way to reference them declaratively. Any change to a base metric requires manually syncing all derived metrics.What We Propose
Following the "define once, reference everywhere" DRY principle, we've designed 5 backward-compatible Schema changes:
filters— Model-level Shared Filter${filter_name}metrics.filters— Metric filter fieldjoin_pathdimensions— Model-level Shared Dimensionfields.dimension.ref— Field-to-dimension binding${metric}— Expression-level metric referenceAll changes are optional — models without them behave exactly as before.
Quick Demo (TPC-DS based)
Alignment with Industry Practice
These patterns have well-established equivalents in major semantic layers (dbt MetricFlow, Cube.js, LookML). Our proposal builds on industry best practices while strengthening support for multi-dataset enterprise scenarios:
filterfiltersfiltersmetrics.filterstype: derivedmeasurescomposition${measure}composition${metric}syntaxsegmentsalways_filterfiltersdimensions+refKey Differences and Advantages Analysis:
Top-level
filters(eliminates logic duplication)dbt MetricFlow's filters are inline per-metric, forcing the same rule (e.g.,
valid_sales) to be copied across dozens of metrics. Inspired by Cube.jssegments, our proposal elevates Filters to first-class SemanticModel citizens, enabling single-point maintenance. Unlike Cube.js segments, Shared Filters embed directly into Metric expressions via${filter_name}syntax, making filter conditions an intrinsic part of the metric definition rather than an optional query-time overlay — ensuring no consumer can accidentally omit critical business rules.dimensionsfor unified business context governance (eliminates metadata fragmentation)LookML provides View-level dimension reuse via
extends, but dimensions remain coupled to physical views. Proposed Shared Dimensions decouple business concepts from physical storage (Fields), using declarativerefbinding to establish a single source of truth for dimension semantics across multiple datasets — resolving the metadata inconsistency problem that surfaces when BI/LLM consumers access the same dimension through different dataset paths.Full Technical Documentation
Detailed schema definitions, parsing rules, a complete e-commerce example, and a minimal TPC-DS demo:
📄 Full proposal specification: https://docs.google.com/document/d/e/2PACX-1vTxGFGlcXK5BHHX5g9SVdGsmFb-IENndonvn5Qhw1_NUEwed8wNyFX3aCr68wHPnp9uHJ6GvoH__fC9/pub
Feedback Appreciated
filtersanddimensionsappropriate as SemanticModel top-level node names?join_pathto individual filter items the right design for cross-dataset filtering?Looking forward to your thoughts! 🙏
All reactions