Python wrapper for dbt-core to extend dbt with custom Python.
This package is a shim for dbt-core, inspired by (cough stolen from cough) my old boss, @darkdreamingdan:
Before using this package, it's recommended to get up to speed with the Python modules that are already available in dbt:
The existing Python modules are available in the dbt Jinja context under the modules object, for example:
{{ modules.datetime.datetime.now() }}While in preview, this package is only available from GitHub:
pip install git+https://github.com/Bilbottom/dbt-py@v0.0.3
This will be made available on PyPI once it's ready for general use.
This package adds a new executable, dbt-py, which injects your custom Python into dbt and then runs dbt. Either a custom module or a custom package can be injected. A custom module is the simplest to get started with.
The default module/package name is custom which would make custom Python available in the dbt Jinja context under the modules.custom object. This can be configured (see the Configuration section below).
Create a module called custom.py in the root of your dbt project. This module can contain any Python code you like, for example:
def salutation(name: str) -> str:
return f"Hello, {name}!"Reference this module and function in the dbt Jinja context of a dbt model:
{{ modules.custom.salutation("World") }}Rather than run dbt with the dbt command, instead run it with dbt-py:
dbt-py clean
dbt-py build
Note that dbt-py is a wrapper around dbt so all the usual dbt commands are available -- all the arguments passed to dbt-py are passed through to dbt, too.
dbt-py --help
dbt-py run --select my_model
dbt-py test --select tag:unit-test
Using a custom package is similar to using a custom module: create a package called custom in the root of your dbt project.
The submodules of this package will be available in the dbt Jinja context too. For example, suppose you have a package called custom with a submodule called greetings:
custom/
__init__.py
greetings.py
If the greetings.py submodule contains the same salutation function as above, then it can be referenced in the dbt Jinja context as follows:
{{ modules.custom.greetings.salutation("World") }}Alternatively, you can expose the salutation function via the __init__.py file and then reference it directly via custom:
{{ modules.custom.salutation("World") }}The default module/package and Jinja context name is custom but both can be configured with the following environment variables:
DBT_PY_PACKAGE_ROOT: The Python-style ref to the custom module/package, e.g.package.module.submoduleDBT_PY_PACKAGE_NAME: The name to give the custom module/package in the dbt Jinja context, e.g.custom_py. Defaults to the value ofDBT_PY_PACKAGE_ROOT
In particular, you can use the DBT_PY_PACKAGE_ROOT environment variable to reference a custom module/package that is not at the root of your dbt project.
Warning
If you set the DBT_PY_PACKAGE_ROOT environment variable to a name that already exists, this package will use the existing module/package rather than your custom one. Make sure that your custom module/package name does not clash with any existing modules/packages.
This is likely to change in a future release, but for now you may choose to exploit this behaviour to use an existing module/package in your dbt Jinja context. For example, you could set DBT_PY_PACKAGE_ROOT to math and then reference the math standard library in your dbt Jinja context:
{{ modules.math.pi }}This is still in preview, and there are a few things to be added before it's ready for general use:
- Support for importing any number of packages (currently only one package is supported)
- Configuration via config files and CLI arguments (currently only environment variables are supported)
- More robust testing
Raise an issue, or fork the repo and open a pull request.
This project uses Poetry for dependency management and pre-commit for linting. After cloning the repo, install the dependencies and enable pre-commit:
poetry install --sync --with dev,test
pre-commit install --install-hooks