google-cloud-dlp dependency is out of date

[stfines-clgx]

Apache Airflow Provider(s)

google

Versions of Apache Airflow Providers

apache-airflow = "^2.4.1"
apache-airflow-providers-google = "^8.4.0"

Apache Airflow version

2.4.1

Operating System

Darwin C02CP187MD6R 21.6.0 Darwin Kernel Version 21.6.0: Mon Aug 22 20:17:10 PDT 2022; root:xnu-8020.140.49~2/RELEASE_X86_64 x86_64

Deployment

Docker-Compose

Deployment details

Development environment

What happened

Latest apache beam cannot be installed because of the out of date google-cloud-dlp dependency:

  Because apache-airflow-providers-google (8.4.0) depends on google-cloud-dlp (>=0.11.0,<2.0.0)
   and no versions of apache-airflow-providers-google match >8.4.0,<9.0.0, apache-airflow-providers-google (>=8.4.0,<9.0.0) requires google-cloud-dlp (>=0.11.0,<2.0.0).
  So, because python-template depends on both apache-airflow-providers-google (^8.4.0) and google-cloud-dlp (^3.9.2), version solving failed.

What you think should happen instead

the apache beam google provider should install using a much more current version of google-cloud-dlp; as of the creation of the ticket it is 3.9.2

How to reproduce

set up a project with the two providers above and attempt to install any beam with gcp extensions at the current version @2.42

I tested down to 2.40 with the same result

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[eladkal]

This is not a bug.
Many of google packages are not up to date, updating was an effort started a while ago #12116

we can make progress in such issues only if users of google (like yourself) will raise PRs.
Feel free to raise one. The list of dependencies can be found on:

airflow/airflow/providers/google/provider.yaml

Lines 56 to 119 in 40d19b9

	dependencies:
	- apache-airflow>=2.2.0
	- apache-airflow-providers-common-sql>=1.2.0
	# Google has very clear rules on what dependencies should be used. All the limits below
	# follow strict guidelines of Google Libraries as quoted here:
	# While this issue is open, dependents of google-api-core, google-cloud-core. and google-auth
	# should preserve >1, <3 pins on these packages.
	# https://github.com/googleapis/google-cloud-python/issues/10566
	# Some of Google Packages are limited to <2.0.0 because 2.0.0 releases of the libraries
	# Introduced breaking changes across the board. Those libraries should be upgraded soon
	# TODO: Upgrade all Google libraries that are limited to <2.0.0
	- PyOpenSSL
	- asgiref>=3.5.2
	- gcloud-aio-bigquery>=6.1.2
	- gcloud-aio-storage
	- google-ads>=15.1.1
	- google-api-core>=2.7.0,<3.0.0
	- google-api-python-client>=1.6.0,<2.0.0
	- google-auth>=1.0.0
	- google-auth-httplib2>=0.0.1
	- google-cloud-aiplatform>=1.7.1,<2.0.0
	- google-cloud-automl>=2.1.0
	- google-cloud-bigquery-datatransfer>=3.0.0
	- google-cloud-bigtable>=1.0.0,<2.0.0
	- google-cloud-build>=3.0.0
	- google-cloud-container>=2.2.0,<3.0.0
	- google-cloud-dataform>=0.2.0
	- google-cloud-datacatalog>=3.0.0
	- google-cloud-dataplex>=0.1.0
	- google-cloud-dataproc>=3.1.0
	- google-cloud-dataproc-metastore>=1.2.0,<2.0.0
	- google-cloud-dlp>=0.11.0,<2.0.0
	- google-cloud-kms>=2.0.0
	- google-cloud-language>=1.1.1,<2.0.0
	- google-cloud-logging>=2.1.1
	- google-cloud-memcache>=0.2.0
	- google-cloud-monitoring>=2.0.0
	- google-cloud-os-login>=2.0.0
	- google-cloud-orchestration-airflow>=1.0.0,<2.0.0
	- google-cloud-pubsub>=2.0.0
	- google-cloud-redis>=2.0.0
	- google-cloud-secret-manager>=0.2.0,<2.0.0
	- google-cloud-spanner>=1.10.0,<2.0.0
	- google-cloud-speech>=0.36.3,<2.0.0
	- google-cloud-storage>=1.30,<3.0.0
	- google-cloud-tasks>=2.0.0
	- google-cloud-texttospeech>=0.4.0,<2.0.0
	- google-cloud-translate>=1.5.0,<2.0.0
	- google-cloud-videointelligence>=1.7.0,<2.0.0
	- google-cloud-vision>=0.35.2,<2.0.0
	- google-cloud-workflows>=0.1.0,<2.0.0
	- grpcio-gcp>=0.2.2
	- httpx
	- json-merge-patch>=0.2
	- looker-sdk>=22.2.0
	- pandas-gbq
	- pandas>=0.17.1
	- sqlalchemy-bigquery>=1.2.1
	# A transient dependency of google-cloud-bigquery-datatransfer, but we
	# further constrain it since older versions are buggy.
	- proto-plus>=1.19.6
	# Google bigtable client require protobuf <= 3.20.0. We can remove the limitation
	# when this limitation is removed
	- protobuf<=3.20.0

I'm closing this issue because we are not going to have separated issue per package.

[potiuk]

And let me add to what @eladkal wrote - @stfines-clgx - upgrading some dependencies is one of the best contributions you might provide - if your organisation wants to upgrade a dependency in a free software that is mostly maintained by voluntaries, spending your company resources to upgrade those deps is one of the best ways your organisation pays back (and easy way to become a contributor).

This is easy - open a pr with updating a dependncy in provider.yaml of google provider (airflow/providers/google/provider.yaml that @eladkal added ) - and update it and fix all the problems in unit tests etc. that are result of it.

Happy to help to review and guide you on that. Looking forward to it!