The Future of the MetricFlow Project

[tlento]

Transform, home to the creators of MetricFlow, was acquired by dbt labs in February 2023. Since the acquisition, the MetricFlow team has been working hard to develop a vision for what dbt and MetricFlow can achieve together. Our focus has been on creating the best integration possible, with a user experience that combines the strengths of both projects. We are thrilled about the possibilities and want to help everyone transition smoothly. Roxi, a PM on the semantic layer team, lays out our plans for this year wonderfully in this blog post.

This discussion outlines our thinking about the integration, and the related changes to MetricFlow. It also highlights the breaking changes that will occur in MetricFlow as a result of the integration.

Where we are

At its heart, MetricFlow provides three key pieces of core functionality:

1. The ability for users to define semantic models and metric computations in a structured, source-control-friendly way
2. The ability to process metric queries, generate a query plan, and render SQL targeted at any supported warehouse dialect
3. The ability to execute the rendered SQL against the target warehouse and return results

Today, MetricFlow is distributed as an all in one package allowing users to define semantic models and compute metrics against either a dbt project or a SQL warehouse. This configuration has three key weaknesses, all of which were causing problems for the development team prior to the acquisition:

1. The package is monolithic, with the spec for defining semantic models and metrics coupled to the MetricFlow runtime, preventing the broader adoption of the spec as a standard, and precluding us from supporting common interfaces and mechanisms for integrating with other packages (as evidenced by the multiple mechanisms for integrating with dbt models built into MetricFlow’s parsing and validation stack, and the “internal” schema objects which only ever worked with Transform’s paid product)
2. The package is complex, with multiple mechanisms for querying metrics and multiple ways of defining what we currently call data sources
3. The package is large, with dependencies on 5 separate warehouse engine connectors

Where we are going

One of the things we’ve learned as MetricFlow developers is a stand-alone semantic layer is much harder for people to use than one that can plug in to existing logical models of the underlying data warehouse. Consequently, the team is focusing on building the best possible integration between MetricFlow and dbt. Doing this well requires us to address all three problems with our current packaging approach and to create first class support for MetricFlow from within dbt.

This new phase of development for MetricFlow will bring about the following changes:

1. The configuration spec for authoring semantic models and metric definitions will be separated into an independent repository, which will be open source. This will allow us to keep dbt-core and MetricFlow decoupled while providing a cohesive, first-class integration between the two projects.
2. The core MetricFlow package will become a stand-alone library for processing metric queries, generating a query plan, and rendering SQL against a target dialect. This focus allows callers to import a much lighter-weight set of dependencies while getting the query planning and dialect-specific SQL rendering that MetricFlow does so well today.
3. The warehouse-level query capabilities for MetricFlow will be provided by separate engine-specific packages. The specifics of what this will ultimately look like are under discussion.

We aim to release a public preview of these new features in Q4 2023, with a public beta available sooner for those who want to test the new features early. Once released, dbt users on supported data platforms will be able to easily configure semantic models and metric definitions on top of the logical models defined in their dbt projects. They'll also be able to query these models using supported semantic layer interfaces.

What we are breaking

While we generally try to avoid breaking changes on our public packages, in this case they are unavoidable. Here are the things we expect to change for end users of the existing MetricFlow package:

1. MetricFlow will now require a dbt project. As part of the first class integration we are building, MetricFlow’s input configurations will be derived from dbt models, instead of manually specified warehouse tables or SQL queries.
2. We are changing the MetricFlow license from AGPL to BSL. This is consistent with our licensing strategy for dbt server (which powered dbt metrics). Tristan wrote about the thinking behind this change here.. Please reach out to us if you have further questions about licensing! We’re happy to chat more about it.
3. Existing installation methods for MetricFlow will not work for new versions. The current installation approaches - including dependency specifications for importing metricflow as a library - will freeze with our final minor version release, and no new features for those versions will be deployed once the integrated package is released. We will make a separate announcement on this with more details as we approach deployment of the new packaging structure.
4. There will be breaking changes in the config spec for defining semantic models and metrics. As part of the separation of the spec from the core MetricFlow package, we will be making some changes to both the API interface specs we use to represent these models and the YAML input schemas associated with them (see discussion in the dbt-core repo). We are building some tooling for converting existing configs to conform to the new specification. Stay tuned for updates!
5. The Python query APIs will be changing. MetricFlow’s existing Python API will necessarily change along with the updates to the config spec and other changes related to splitting out the various warehouse connectors into separate packages.
6. Existing dbt integrations with MetricFlow will be removed in favor of the first class integration we are building out. It is currently possible to reference dbt models from within MetricFlow, and to convert dbt metrics configs into objects MetricFlow’s toolchain can query. These integrations are a bit hacky since we had not yet designed a first class integration, and we will be removing them in favor of enabling seamless extension of dbt projects to include semantic models and metric definitions.
7. We are removing support for marterializations. Materializations were a somewhat hacky feature to work around certain limitations of MetricFlow in the context of operating directly within end user warehouses. We are removing support for materializations, and will seek to address their various use cases in more directly appropriate ways once the integration phase is complete.

Parting thoughts

We're excited about the changes coming to MetricFlow and the future of its tighter integration with dbt. We believe this is the best path forward for creating a universally adopted semantic layer, and ultimately changing the way analysts and engineers do their jobs.