[CT-1453] [Spike] A pattern for plugins

[jtcohen6]

We want dbt-core to be much more pluggable in the future, with several supported interfaces. Think: modeling languages, backend implementations, authentication, ...

We have a single blessed plugin interface today, for adapter plugins (authentication + runtime engines for models). Basically (or as I understand it), we use importlib to dynamically import the dbt-somedb adapter plugin, and make it available to dbt-core:

dbt-core/core/dbt/adapters/factory.py

Line 57 in be4a91a

mod: Any = import_module("." + name, "dbt.adapters")

We don't love the way this works. Specifically:

• All adapter plugins must follow a prescriptive submodule/file structure, which matches exactly the one in the core adapters module
• The control flow between dbt-core and plugins is difficult to follow through the code: BaseAdapter (in dbt-core) → PostgresAdapter (in dbt-postgres) → parsing, tasks, etc (in dbt-core)
• Lack of clear decoration/demarcation of which methods are "public" to the plugin interface, given complete mutual inheritance
• Plugins must be installed in the same Python runtime / virtualenv (not sure if there's any getting away from this, but worth calling out all the same)
• ... (more reasons) ...

What are modern plugin patterns for Python? Let's do some investigation here. Bonus points for architectures that offer a degree of language agnosticism. This is very much aligned with our tech debt initiatives over the next few months, around "API-ification" and logging, and it will be relevant for all our sub-teams into next year.

[jtcohen6]

From initial conversation with @nathaniel-may:

• Current ideas: gRPC, unix sockets [not sockets, it's never sockets], ...
• Online/offline: Needs to work equally well in an offline system. That's how dbt-core + existing Python plugins would continue to work by default / in most OSS deployments. In Python, abysmally slow to spin up interpreter every time you need to "ask a question."
• Compatibility: Back compat for current adapters (ideally with zero effort). The interface that could be back/fwd compatible, so plugins can be compatible with dbt-core at the major-version level. And the interface should be language agnostic.

This should also enable us to separate, in cloud deployments, dbt-core from arbitrary plugins (untrusted code), by having each one run in a different environment/container/...

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.