New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for custom backends #40
Conversation
Codecov Report
@@ Coverage Diff @@
## master #40 +/- ##
==========================================
+ Coverage 97.34% 97.70% +0.36%
==========================================
Files 5 8 +3
Lines 677 828 +151
==========================================
+ Hits 659 809 +150
- Misses 18 19 +1
Flags with carried forward coverage won't be shown. Click here to find out more.
Continue to review full report at Codecov.
|
b775a56
to
360a810
Compare
Ah, it seems positional-only keywords are not supported in Python 3.7. Even though I don't see any reasons to not upgrade to the latest Python version, I do want to support all the Python versions dbt does. Edit: I don't see any workaround for this. I'll drop positional-only keywords and eventually merge them once dbt drops Python 3.7. |
ba965e5
to
41f5353
Compare
Python 3.7 support seems to be more tricky than expected... |
78b897d
to
28d1f89
Compare
28d1f89
to
194a975
Compare
This PR adds support for custom backends for storing dbt projects and
profiles.yml
: up until now, only S3 was supported, and local directories worked (kind of) unintentionally.This PR creates a new abstraction (
DbtBackend
) to support any backend that implements the necessary push and pull methods. Concrete implementations for the local filesystem and S3 are provided, but in the future more will be added. Backends may use Airflow hooks (DbtS3Backend
uses the Amazon provider's S3Hook) but this is not required.I'm not entirely sure on the naming scheme though: I went with backend since Airflow's XCom also refers as "backend" to the underlying storage and use "push" and "pull" methods for them (see: https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html#custom-backends). But I'm not super convinced the semantics of the word backend fit what we are doing here as it's much simpler storage. I also thought about: