Is a custom SemanticLayer extension the right way to query an Apache Ossie model stored in Apache Polaris?

[mfournioux]

What we are trying to do

We want to describe our metrics once, in Apache Ossie format, store that definition in Apache Polaris, and then build charts and dashboards on it in Superset. The data itself lives in Iceberg tables, queried through a database connection Superset already has.

Our stack:

Component	Version
Superset	7.0.0rc1, with SEMANTIC_LAYERS and ENABLE_EXTENSIONS on
Apache Polaris	1.8.0
Apache Ossie	0.2.0.dev0

The extension mechanism itself already works for us: we deployed betodealmeida/pandas-semantic-layer as a local extension, the connection form renders, and charts against it work. What we are unsure about is the design of an Ossie provider, which is why we are asking before writing code.

What we found

Our starting assumption was that something, somewhere, already knew how to turn an Ossie model into a SQL query. We looked, and could not find it. Here is what we checked:

Component	Turns Ossie into SQL?	What we observed
Polaris 1.8.0	No	Its semantic-models extension is a CRUD store: create, list, get, update, delete. On a live instance, query, sql, compile, resolve, metrics and plan all return 404, while semantic-models returns 401.
The Ossie spec	No, deliberately	expression_language.md states: "It does not define a query interface… Query interfaces will be defined in separate documents." A reference Ossie → SQL compiler is listed as future work.
ossie-converters/polaris	No	It creates empty Iceberg tables from a model and drops metrics, relationships and expressions. It never calls the /semantic-models API.
Superset 7.0.0rc1	Provides the contract only	SemanticLayer and SemanticView ship as abstract base classes, and get_table() is abstract.

So our reading is that generating the SQL is the consuming client's job, which also matches the intent described in apache/polaris#4522: each client reads the same definition, picks its dialect from expression.dialects[], and produces the query itself.

We would like that confirmed, because everything else we plan follows from it.

One distinction we want to state clearly

It is easy to blur two different things, so to be explicit about what we are and are not reusing:

• Generating SQL. Superset has get_sqla_query in superset/models/helpers.py, which builds SQL for Superset's own datasets. It is not used for semantic views. SemanticView.get_query_str() returns the literal string "Not implemented for semantic layers", and superset/semantic_layers/mapper.py hands the request to semantic_view.get_table(). So Superset generates no SQL on this path, and we are not trying to reuse that code.

• Executing SQL. Separately, Database.execute(sql, options) takes a SQL string that the caller already produced and runs it against an existing connection. That is the only piece we want to reuse, so that the extension does not need to ship its own driver or handle authentication.

In short: we write the SQL, an existing Superset connection runs it.

What we plan to build

A backend-only Superset extension acting as that consuming client. Nothing in it is specific to a particular engine: the dialect comes from the model, and execution goes through whichever Superset connection is configured.

Ossie model (YAML)
   │  POST /api/catalog/polaris/v1/{cat}/namespaces/{ns}/semantic-models
   ▼
Apache Polaris ─────────── Iceberg catalog ───────────┐
   │  GET  /semantic-models                           │
   ▼                                                  ▼
Superset extension "ossie-semantic-layer"      SQL engine
   OssieSemanticLayer : reads and parses the model    ▲
   OssieSemanticView  : builds the SQL ───────────────┘
                        runs it via Database.execute()
   ▼
Explore / dashboards / MCP tools

Step by step:

1. from_configuration() fetches the model from the Polaris semantic-models API and parses it with the ossie Python library, caching it against the entity-version Polaris returns.
2. get_dimensions() and get_metrics() map Ossie fields and metrics onto Superset's types.
3. get_table() receives a SemanticQuery (metrics, dimensions, filters, order, limit) and builds a SELECT from it using SQLGlot, in the dialect the model declares through expression.dialects[]. Because Ossie expressions are already SQL fragments, this is mostly assembly: project the dimensions, add the aggregates, and derive the joins by walking relationships[].
4. The resulting SQL is executed through Database.execute() on an existing Superset connection, and the DataFrame is converted to a pa.Table.
5. The generated SQL is returned in SemanticResult.requests, so it can be inspected when a chart looks wrong.

Questions

1. Is our reading correct that no component compiles Ossie into SQL today, and that a provider is expected to do it itself?
2. Is a custom extension the right home for this, or is an official Ossie provider already planned, in which case we would rather help with that than duplicate it?
3. Which API should an extension target for the 7.0 line? On master, from_configuration takes cache_token, context and credentials, and there is a SemanticExecutionContext. None of these exist in 7.0.0rc1, where the signature is simply from_configuration(cls, configuration). The guide at docs/developer_docs/extensions/extension-points/semantic-layers.md documents the master signature and is not present in the 7.0.0rc1 tag, so following it on 7.0 produces an extension that fails to load. Is there guidance for extension authors on 7.0?
4. Is Database.execute() the intended way for a provider to run the SQL it generated, or is a provider expected to open its own connection to the engine?
5. Is a 7.0-compatible reference provider planned? pandas-semantic-layer is the only working example we found for rc1; example_provider.py and tests/e2e_extensions/semantic_stub/ exist only on master.

Contribution

If the approach is confirmed, we will open a PR.

One thing we noticed while scoping the work: the Ossie → SQL part is not specific to Polaris or to us. Any Superset deployment consuming an Ossie model would need the same logic. If that seems useful to others, we are happy to shape it as something reusable rather than keeping it inside our own extension, and to contribute it wherever maintainers think it belongs. If that is out of scope, we will simply keep it internal and ship the provider on its own.

---

Related: SIP-182, apache/polaris#4522