implement dbt-utils macros for dbt-materialize adapter

[JLDLaughlin]

More details available here: MaterializeInc/dbt-materialize#2

dbt-utils contains utility macros to be used and reused across dbt projects. Contributed adapters (like Materialize's) have their own [adapter]-utils packages. We have started implementing utility macros in materialize-dbt-utils, but support is incomplete. We should implement the rest of the macros, where possible. Some macros (like current_timestamp()) will not be able to be used with Materialize.

[JLDLaughlin]

@jtcohen6 - I was just chatting with @benesch and we were wondering if it would be okay to squish the materialize-dbt-utils macros into our dbt-materialize adapter? or is it important that we keep these two things separate?

[jtcohen6]

@JLDLaughlin Fair question, and sorry for the delayed response! I'll share some rationale in favor of keeping separate, which animate our own ongoing decision to keep dbt-utils separate from dbt:

Release cadence: You may want to add, update, and release utility macros separate from dbt-materialize releases. Generally, the latter should follow and be pinned to dbt-core releases, whereas utility packages can support a wider range of dbt-core versions. We find ourselves making frequent patch releases of dbt-utils, which tends to lend itself to many small fixes or additions, without many breaking changes. By a similar token, the contracts of dbt plugins are better established and have firmer guarantees than the contracts of dbt-utils. I'd hate for you to have to scramble to release a new plugin version, just because a macro changed...

Ease of contributing: Related to the above, we try to ensure that contributing to dbt-utils is feasible for anyone who knows SQL and Jinja, whereas contributing to dbt plugins requires more python chops and familiarity (or patience) with local dev and testing environments. Our community includes many folks for whom, despite deep analytics experience, dbt may be the first open source software with which they've engaged meaningfully. That may be less relevant for the average member of the Materialize community.

Namespacing: These macros should be in a different project namespace (dbt_utils, materialize_dbt_utils) from the macros defined within the adapter plugin, which are in the special dbt and adapter namespaces. The way we've created and communicated adapter.dispatch() thus far, we've been erring on the side of overly explicit, such that users need to "opt in" to overriding dbt_utils macros with implementations from a different package/namespace. It's a subtle thing, and we don't want it to be completely obfuscated. So in order for dbt_utils.dateadd to find materialize__dateadd, instead of defaulting to dbt_utils.default__dateadd, users need to set a config variable in their project. In the case of a materialize_dbt_utils "shim" package, this looks like:

vars:
  dbt_utils_dispatch_list: ['materialize_dbt_utils']

If these macros shipped inside dbt-materialize/dbt/include/materialize/macros/ instead, users would instead write:

vars:
  dbt_utils_dispatch_list: ['dbt']

That looks a bit weird to my eyes, and it feels a bit circular. We think of the inheritance order as dbt-core → dbt-[adapter] → installed packages → root project.

Ultimately, after we have more qualitative data about how folks are using and understanding dispatch, we may decide to make things work more implicitly: perhaps switching our convention to a single dispatch_list variable, rather than a unique one for each package being shimmed; or tweaking dispatch to always check the adapter plugins for package macros, too.

Organization: The best way to distribute dbt adapter plugins is via PyPi; the best way to distribute dbt packages is via hub.getdbt.com, since that enables users to install it with specified version ranges and, in the event of overlapping dependencies, dbt deps will perform (fairly naive) version resolution.

I see and totally respect that Materialize errs toward using well-organized monorepos vs. many-repos—at least, in the sense of having docs, demos, and the core product cohabitate in this repo! It would be totally possible to squish the materialize-dbt-utils project into the same repo as the dbt-materialize adapter, for ease-of-maintenance, while still keeping them as independent projects (i.e. separate dbt_project.yml nested file structures). The drawback? Currently, there is no way for the Hub site to host repositories that don't have dbt_project.yml at the root. That's on us, and it's something we should eventually enable.

[benesch]

Thanks very much for the detailed response @jtcohen6. I think we ought to keep them separate for now, then, so we can get the utilities into dbt hub.

[benesch]

We're in dbt hub now, and the next release of materialize-dbt-utils will include support for almost all of the utility functions. I filed MaterializeInc/materialize-dbt-utils#7 to track the remainder.