Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for custom backends #40

Merged
merged 8 commits into from Feb 18, 2022
Merged

Support for custom backends #40

merged 8 commits into from Feb 18, 2022

Conversation

tomasfarias
Copy link
Owner

@tomasfarias tomasfarias commented Feb 7, 2022

This PR adds support for custom backends for storing dbt projects and profiles.yml: up until now, only S3 was supported, and local directories worked (kind of) unintentionally.

This PR creates a new abstraction (DbtBackend) to support any backend that implements the necessary push and pull methods. Concrete implementations for the local filesystem and S3 are provided, but in the future more will be added. Backends may use Airflow hooks (DbtS3Backend uses the Amazon provider's S3Hook) but this is not required.

I'm not entirely sure on the naming scheme though: I went with backend since Airflow's XCom also refers as "backend" to the underlying storage and use "push" and "pull" methods for them (see: https://airflow.apache.org/docs/apache-airflow/stable/concepts/xcoms.html#custom-backends). But I'm not super convinced the semantics of the word backend fit what we are doing here as it's much simpler storage. I also thought about:

  • "Storage" with "store" and "fetch".
  • "Warehouse": discarded as can be confused with something to do with a Data Warehouse.

@codecov
Copy link

codecov bot commented Feb 7, 2022

Codecov Report

Merging #40 (194a975) into master (fc0a12f) will increase coverage by 0.36%.
The diff coverage is 95.48%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #40      +/-   ##
==========================================
+ Coverage   97.34%   97.70%   +0.36%     
==========================================
  Files           5        8       +3     
  Lines         677      828     +151     
==========================================
+ Hits          659      809     +150     
- Misses         18       19       +1     
Flag Coverage Δ
unittests 97.70% <95.48%> (+0.36%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
airflow_dbt_python/hooks/backends/__init__.py 86.66% <86.66%> (ø)
airflow_dbt_python/hooks/backends/s3.py 93.10% <93.10%> (ø)
airflow_dbt_python/hooks/dbt.py 97.54% <93.75%> (+1.51%) ⬆️
airflow_dbt_python/operators/dbt.py 99.32% <97.14%> (+<0.01%) ⬆️
airflow_dbt_python/hooks/backends/localfs.py 98.14% <98.14%> (ø)
airflow_dbt_python/hooks/backends/base.py 100.00% <100.00%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc0a12f...194a975. Read the comment docs.

@tomasfarias tomasfarias changed the title Custom backends Support for custom backends Feb 8, 2022
@tomasfarias
Copy link
Owner Author

tomasfarias commented Feb 8, 2022

Ah, it seems positional-only keywords are not supported in Python 3.7. Even though I don't see any reasons to not upgrade to the latest Python version, I do want to support all the Python versions dbt does.

Edit: I don't see any workaround for this. I'll drop positional-only keywords and eventually merge them once dbt drops Python 3.7.

@tomasfarias
Copy link
Owner Author

Python 3.7 support seems to be more tricky than expected...

@tomasfarias tomasfarias force-pushed the custom-backends branch 2 times, most recently from 78b897d to 28d1f89 Compare February 9, 2022 18:23
@tomasfarias tomasfarias marked this pull request as ready for review February 9, 2022 18:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant