Alternative Entrypoints

[cmcarthur]

Feature

Feature description

In https://github.com/fishtown-analytics/dbt/compare/hackathon-2, we experimented with adding "alternative entrypoints" to dbt. Specifically, we built code into dbt that allowed it to load an already-generated manifest from disk and deserialize it. Then, when including dbt as a library, it was possible to use that loaded manifest to invoke other parts of dbt without re-compiling. For example, it would be possible to compile a single node, or run a single SQL query, when including dbt as a library.

This is an important first step towards client server.

The changes suggested here are:

• Give dbt a way to load a manifest file from disk and deserialize it
• Give dbt a way to use that manifest to take small, specific actions, e.g.
  • compile a SQL string
  • run a SQL query against the warehouse

[drewbanin]

Great writeup @cmcarthur. One thing I'd add: This entrypoint should optionally support compiling the manifest upon entry. Then, we'll always have a working manifest to operate with. As it stands from the hackathon-2 branch, you needed to manually compile the project before you could use the manifest.

[cmcarthur]

@drewbanin agree -- this issue probably encapsulates dozens of issues related to the specific alternative entrypoints. Off the top of my head, here are some example entry points that I'd love to see dbt support:

• given a manifest, and a change to one file, generate an updated manifest
• given a manifest, and some SQL, compile that SQL
• given a manifest, run a specific model in the manifest
• given a manifest, run some SQL and return the result

Notably the common thread here in every case that I can imagine is the manifest. It's pretty clear to me that the manifest is the protocol for passing around the configuration for a dbt project. In a client-server world, I have to imagine that the manifest is the most frequently passed around entity.

[cmcarthur]

You can imagine how those little atomic operations give you lots of powerful orchestration options, and power lots of different dbt clients.

[drewbanin]

how does "a full-fledged server" sound for an alternative entrypoint? :)

We'll still probably want to do something like this in the future, but we can queue up a more actionable issue when we get there