Managed sources: adding DDL capabilities to sources

[Fleid]

Hey all,

Let's talk streaming

As you may have noticed, we started work on supporting streaming scenarios via:

• [CT-2068] [Feature] Let's add Materialized View as a materialization, finally #6911
• [CT-2070] [Feature] Materialized tests #6914

The next step in our mind is extending sources so they can manage the usual objects required in an end-to-end streaming pipeline. But we don't to limit ourselves to stream objects, we want to think about all the external thingies that we had always wanted to manage via dbt. This new type of sources should be valuable outside of streaming scenarios.

This discussion may be the most relevant, and I will be parsing it to absorb as much as I can from it:

• Manage data warehouse objects (other than tables and views) #5099

Though I didn't want to continue the conversation there, as we went to focus here on "storage" objects. dbt is about ELT, not ETL. Our atomic unit is the model, a unit of transformation that results in rows. Compute only objects (tasks/jobs…) typically used in streaming (as in streaming ETLs) are not in scope. Same for UDFs.

Managed sources (?)

Here's a list of traits I can imagine for that new object:

• You define attributes for them in YAML
  • Mostly connection information, wiring
  • No transformations allowed, so no SQL supported at all → This means that schema declaration should not be done here. This may be controversial as it’s usually a recommended practices in engines - or may even be mandatory. We will have to reconcile that.
  • Most engines don’t allow transformation code anyway, but for those that do we should ignore it
• We generate the DDL, potentially creating multiple objects (locations, formats…) but not credentials. Credentials are infrastructure.
• Sources should be stateless
• Ideally we want another colour in the DAG (if sources are green, managed sources should be dark green or teal)

Obviously there is a large common surface area with dbt-external-tables, we will see how we handle that once we have a firmer design in mind.

Not in scope?

Flink has a nice concept of hybrid sources, as in wrapping the definition of historical sources in addition to the live/streaming one, while managing the “lambda” view on top auto-magically. We won’t do that in sources as it involves transformation logic. It’s user land logic that needs to be composed with 2 sources and a model.

While we are looking at sources, we may have an opportunity to tackle other asks in the area. I'm pessimistic about it (some of it just looks like codegen) but we should still do our due diligence:

• Flexible multiplexed sources #6185
• Dynamic source definitions #5686

Next steps

I would like a first version of managed sources to ship with 1.6 (July). I will be doing a bit more research and come back here with a draft of what they could look like in the coming weeks. So if you have a strong opinion on the mater, now is the best time to voice it ;)

[Fleid]

We should look at solving that too: #7012