[SL-1873] Remove MetricFlow's direct dependency on dbt-core

[tlento]

tl;dr - for historical reasons MetricFlow has a direct dependency on dbt-core. This is causing problems with package management and deployment, particularly for dbt-core updates (such as 1.8) which are not fully backwards compatible with older adapter package versions.

MetricFlow was originally kind of a semantic layer in a box, with three different SqlClient implementations targeting three major SQL warehouse engines (and all of their associated dependencies). This was not sustainable, so when we built the integration with dbt-core last year we moved in the direction of having MetricFlow become a query building library. The first step was to eliminate the custom SqlClient implementations in favor of using dbt adapters to power the CLI and test suites. Alongside this we also forked the semantic model and metric spec into a separate package (dbt-semantic-interfaces) in order to allow dbt-core and MetricFlow to share the same model for expressing metric definitions.

For historical reasons having to do with the way dbt parses project configuration and how, prior to dbt-core 1.8, the adapter interfaces were bundled in dbt core, MetricFlow needed to depend on dbt-core to preserve testability and CLI query access for local development.

This situation has caused a number of problems with package updates, which we've worked around with some degree of effort. Those issues are back with the update to 1.8, and worse than before since we are also having some difficulties with third party adapters (e.g., DuckDB) that are burdensome to us.

It is, therefore, time to finish the job and move all dbt-core dependencies out of the main MetricFlow package. This involves the following steps:

Tasks

Beta Give feedback

1. [SL-1874] Move MetricFlow tests to repo root so they can access modules in, and be executed from, both dbt-metricflow and metricflow #1088
   +0
2. [SL-1876] Move the CLI and other adapter/dbt-core dependent components into dbt-metricflow. #1090
   +0
3. [SL-1878] Clean up metricflow distribution package #1092
   +0

_SL-1873

[markfickett]

+1 It would be great to ease the update to dbt-core 1.8.0 which we'd like to do to pick up unit tests.

[tlento]

This is done. The next MetricFlow release will not depend on dbt-core. Due to some separate package configuration concerns the simplest path for installation will, as a practical matter, require dbt-core 1.8.0 or later but this sort of version locking will also be relaxed in the future.