Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update google-cloud deps to v2 #12116

Closed
mik-laj opened this issue Nov 5, 2020 · 6 comments
Closed

Update google-cloud deps to v2 #12116

mik-laj opened this issue Nov 5, 2020 · 6 comments
Assignees
Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues

Comments

@mik-laj
Copy link
Member

mik-laj commented Nov 5, 2020

Hello,

Google Cloud API Libraries introduced breaking changes in 2.0.0 (https://github.com/googleapis/python-container/blob/master/UPGRADING.md) and they already caused a number of changes. For this reason, we limit all our google-cloud deps to <2.0.0. This limit means our updates will not receive updates, but everything will still work.

We should try to update all operators to use newer versions so that our users can use the newest API features.

You can track progress using this notebook:
https://colab.research.google.com/drive/1F5Lw8qNcxCvWaYUrGZ1x3W3v3080Dq0U#scrollTo=2IjL5GZH4PY_

Related issue: #10316

@mik-laj mik-laj added the kind:feature Feature Requests label Nov 5, 2020
@mik-laj mik-laj changed the title Update google-cloud deps to lv2 Update google-cloud deps to v2 Nov 5, 2020
@potiuk
Copy link
Member

potiuk commented Nov 5, 2020

Thanks @mik-laj . You are a gem!

@michalslowikowski00
Copy link
Contributor

I am working on Dataproc package.

@mik-laj
Copy link
Member Author

mik-laj commented Dec 10, 2020

If we want users to be able to install backport packages, we should also remember about the limitations that exist in Airflow 1.10 We don't support them anymore, but users can still use them and it would be great if they could use new and old operators at the same time.

    'google-cloud-bigtable>=1.0.0,<2.0.0',
    'google-cloud-container>=0.1.1,<2.0.0',
    'google-cloud-dlp>=0.11.0,<2.0.0',
    'google-cloud-language>=1.1.1,<2.0.0',
    'google-cloud-secret-manager>=0.2.0,<2.0.0',
    'google-cloud-spanner>=1.10.0,<2.0.0',
    'google-cloud-speech>=0.36.3,<2.0.0',
    'google-cloud-storage>=1.16,<2.0.0',
    'google-cloud-texttospeech>=0.4.0,<2',
    'google-cloud-translate>=1.3.3,<2.0.0',
    'google-cloud-videointelligence>=1.7.0,<2.0.0',
    'google-cloud-vision>=0.35.2,<2.0.0',

We can only update these operators when we stop releasing backport packages.

In addition, Backport providers within 1.10.x, will be supported for critical fixes for three months after Airflow 2.0.0 is released.

http://apache-airflow-docs.s3-website.eu-central-1.amazonaws.com/docs/apache-airflow/latest/upgrading-to-2.html

CC: @rafalbiegacz @potiuk

@mik-laj
Copy link
Member Author

mik-laj commented Dec 10, 2020

@michalslowikowski00 @turbaszek Fortunately, Dataproc does not conflict, but other libraries will cause problems.

@mik-laj
Copy link
Member Author

mik-laj commented Dec 14, 2020

I added a new column "To update?", which specifies whether we can update this library or rather we should wait until we drop Airflow 1.10 support.

@mik-laj
Copy link
Member Author

mik-laj commented Dec 20, 2020

Part of: #12120

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:feature Feature Requests provider:google Google (including GCP) related issues
Projects
None yet
Development

No branches or pull requests

4 participants