Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make apache-airflow not required #35

Merged
merged 2 commits into from Feb 8, 2022
Merged

Conversation

tomasfarias
Copy link
Owner

@tomasfarias tomasfarias commented Jan 31, 2022

Since we wish to support all MWAA supported versions of Airflow,
this requires supporting both 1.10.12 and 2.X releases. However, There
are several conflicting dependencies between dbt-core and
apache-airflow. None of the conflicts should break us, but they do
make dependency resolution impossible, which means airflow-dbt-python
cannot be installed.

For this reason we have removed the dependency to
apache-airflow. Assuming that as an Airflow operator we are always
installing airflow-dbt-python in an environment that already has
Airflow installed, this shouldn't cause any issues in production
deployments.

Moreover, this makes testing multiple Airflow versions easier as we
can pre-install apache-airflow.

@codecov
Copy link

codecov bot commented Jan 31, 2022

Codecov Report

Merging #35 (32115a2) into master (08f0125) will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@           Coverage Diff           @@
##           master      #35   +/-   ##
=======================================
  Coverage   96.16%   96.16%           
=======================================
  Files           5        5           
  Lines         678      678           
=======================================
  Hits          652      652           
  Misses         26       26           
Flag Coverage Δ
unittests 96.16% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.


Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 08f0125...32115a2. Read the comment docs.

@tomasfarias
Copy link
Owner Author

The likelihood of this breaking somebody's environment is low as it's very reasonable to assume Airflow deployments already have, well, Airflow installed. But, I may be missing so we'll leave this one open for a bit.

Since we wish to support all MWAA supported versions of Airflow,
this requires supporting both 1.10.12 and 2.X releases. However, There
are several conflicting dependencies between dbt-core and
apache-airflow. None of the conflicts should break us, but they do
make dependency resolution impossible, which means airflow-dbt-python
cannot be installed.

For this reason we have removed the dependency to
apache-airflow. Assuming that as an Airflow operator we are always
installing airflow-dbt-python in an environment that already has
Airflow installed, this shouldn't cause any issues in production
deployments.

Moreover, this makes testing multiple Airflow versions easier as we
can pre-install apache-airflow.
@nugic
Copy link

nugic commented Feb 2, 2022

I like your project @tomasfarias !
Was trying to get airflow-dbt-python working on Astronomer, but during the build conflicts came up due to apache-airflow version requirements (Astronomer uses their own airflow builds with version epoch-s).
So this was exactly what was needed, pulled the branch and installed airflow-dbt-python without airflow and seems to work perfectly.
Thanks again and hope it gets released soon!

@tomasfarias tomasfarias marked this pull request as ready for review February 2, 2022 14:31
@tomasfarias
Copy link
Owner Author

Hey @nugic. Awesome comment! Thanks for letting me know! I was hoping to get some confirmation that this is a needed change as it's also affecting our MWAA setup. Now I'll definitely merge this and release it shortly.

@tomasfarias tomasfarias merged commit fc0a12f into master Feb 8, 2022
@tomasfarias tomasfarias deleted the airflow-not-required branch February 8, 2022 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants