Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Separate out bigquery as standlone package from apache-airflow-providers-google #30224

Closed
1 of 2 tasks
ying-w opened this issue Mar 21, 2023 · 2 comments
Closed
1 of 2 tasks
Labels
kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet

Comments

@ying-w
Copy link
Contributor

ying-w commented Mar 21, 2023

Description

apache-airflow-providers-google has a lot of dependencies, some pinned to pretty old versions compared to current release versions. In particular, many of these older versions requires protobuf<4.0.0dev whereas if you already have newer versions (say google-cloud-bigquery==3.7.0) it requires newer versions of protobuf and causes incompatibility

Use case/motivation

Installing dbt-bigquery then apache-airflow-providers-google takes forever to resolve dependencies.

Installing apache-airflow-providers-google then dbt-bigquery leads to dependency errors:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
google-cloud-aiplatform 1.16.1 requires google-cloud-bigquery<3.0.0dev,>=1.15.0, but you have google-cloud-bigquery 3.7.0 which is incompatible.
google-ads 18.0.0 requires protobuf!=3.18.*,!=3.19.*,<=3.20.0,>=3.12.0, but you have protobuf 3.20.3 which is incompatible.

Related issues

No response

Are you willing to submit a PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@ying-w ying-w added kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet labels Mar 21, 2023
@boring-cyborg
Copy link

boring-cyborg bot commented Mar 21, 2023

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

@potiuk
Copy link
Member

potiuk commented Mar 21, 2023

This is not easy and does not solve the problem, because bigquery has a lot of common code between those and splitting the provider would actuallly add complexity rather than solve it. There was this issue opened a long time ago #15933 - but it makes very little sense considering the common code.

You can read discussions there to understand this is not really possible to split - but if you would like to attempt it - feel free.

Bigquery is just one of those.

Instead a better solution is something that Google team works on #30067 which is supposed to bring all dependencies of google to latest versions. This is far more feasible approach and I suggest to subscribe to it to follow the progress.

Closing it as essentially duplicate of both (in various ways) of #15933 and #30067

@potiuk potiuk closed this as completed Mar 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature Feature Requests needs-triage label for new issues that we didn't triage yet
Projects
None yet
Development

No branches or pull requests

2 participants