Skip to content

Commit

Permalink
Suspend Yandex provider due to protobuf limitation (#30667)
Browse files Browse the repository at this point in the history
Yandex provider brings protobuf dependency down to <4 and we are gearing
up to updating it everywhere else. Protobuf3 support ends in Q2 2023
for Python https://protobuf.dev/support/version-support/#python

Yandex is the last provider that we do not closely collaborate with on fixing
* Gogle provider dependencies are actively upgraded to latest version
  by Google led team: #30067 (some of the libraries are already updated)
  with target to update all dependencies by mid-May
* Apache-Beam has already merged protobuf4 support
  apache/beam#25874 with the target of
  releasing it in 2.47.0 mid-May
* The mysql-connector-python in MySQL provider is already turned into
  optional dependency: #30487

The only remaining dependency limiting us to protobuf 3 (<3.21) is
yandexcloud. We've opened an issue to yandexcloud
yandex-cloud/python-sdk#71 3 weeks ago
and while there was an initial interest, there is no progress on
the issue, therefore - in order to prepare for running all
the tests and final migration to protobuf4 we need to suspend
Yandex provider - following the suspension process we agreed
and got a LAZY CONSENSUS on in
the https://lists.apache.org/thread/g8b3k028qhzgw6c3yz4jvmlc67kcr9hj
mailing list discussion.

The yandex provider can be removed from suspension by a PR reverting
this change once yandexcloud dependency removes the protobuf limitation
in their release and PR reverting this change (and fixing all tests
and static check that will be needed) is the way it can be done.
  • Loading branch information
potiuk committed Apr 19, 2023
1 parent 29bd9bf commit b4d6e83
Show file tree
Hide file tree
Showing 12 changed files with 184 additions and 201 deletions.
2 changes: 1 addition & 1 deletion CONTRIBUTING.rst
Expand Up @@ -621,7 +621,7 @@ microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql, mysql, neo4j, od
openlineage, opsgenie, oracle, otel, pagerduty, pandas, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram,
trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
trino, vertica, virtualenv, webhdfs, winrm, zendesk
.. END EXTRAS HERE
Provider packages
Expand Down
2 changes: 1 addition & 1 deletion INSTALL
Expand Up @@ -106,7 +106,7 @@ microsoft.mssql, microsoft.psrp, microsoft.winrm, mongo, mssql, mysql, neo4j, od
openlineage, opsgenie, oracle, otel, pagerduty, pandas, papermill, password, pinot, plexus,
postgres, presto, qds, qubole, rabbitmq, redis, s3, salesforce, samba, segment, sendgrid, sentry,
sftp, singularity, slack, smtp, snowflake, spark, sqlite, ssh, statsd, tableau, tabular, telegram,
trino, vertica, virtualenv, webhdfs, winrm, yandex, zendesk
trino, vertica, virtualenv, webhdfs, winrm, zendesk
# END EXTRAS HERE

# For installing Airflow in development environments - see CONTRIBUTING.rst
Expand Down
2 changes: 1 addition & 1 deletion airflow/providers/yandex/provider.yaml
Expand Up @@ -20,7 +20,7 @@ package-name: apache-airflow-providers-yandex
name: Yandex
description: |
Yandex including `Yandex.Cloud <https://cloud.yandex.com/>`__
suspended: false
suspended: true # see https://github.com/apache/airflow/pull/30667/
versions:
- 3.3.0
- 3.2.0
Expand Down
2 changes: 0 additions & 2 deletions docs/apache-airflow/extra-packages-ref.rst
Expand Up @@ -214,8 +214,6 @@ These are extras that add dependencies needed for integration with external serv
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| vertica | ``pip install 'apache-airflow[vertica]'`` | Vertica hook support as an Airflow backend |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| yandex | ``pip install 'apache-airflow[yandex]'`` | Yandex.cloud hooks and operators |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+
| zendesk | ``pip install 'apache-airflow[zendesk]'`` | Zendesk hooks |
+---------------------+-----------------------------------------------------+-----------------------------------------------------+

Expand Down
7 changes: 0 additions & 7 deletions generated/provider_dependencies.json
Expand Up @@ -773,13 +773,6 @@
"common.sql"
]
},
"yandex": {
"deps": [
"apache-airflow>=2.3.0",
"yandexcloud>=0.173.0"
],
"cross-providers-deps": []
},
"zendesk": {
"deps": [
"apache-airflow>=2.3.0",
Expand Down
10 changes: 5 additions & 5 deletions images/breeze/output-commands-hash.txt
Expand Up @@ -2,7 +2,7 @@
# Please do not solve it but run `breeze setup regenerate-command-images`.
# This command should fix the conflict and regenerate help images that you have conflict with.
main:83de6a9bf2b1afecd1f9ce4cd0493733
build-docs:a9ee0b7c62d647cc7562df83076cbc2b
build-docs:d449f8ee7b20545a2c7c46ad65226f94
ci:find-newer-dependencies:8fa2b57f5f0523c928743b235ee3ab5a
ci:fix-ownership:fee2c9ec9ef19686792002ae054fecdd
ci:free-space:47234aa0a60b0efd84972e6e797379f8
Expand Down Expand Up @@ -37,15 +37,15 @@ prod-image:verify:31bc5efada1d70a0a31990025db1a093
prod-image:79bd4cc9de03ab7e1d75f025d75eee46
release-management:create-minor-branch:6a01066dce15e09fb269a8385626657c
release-management:generate-constraints:ae30d6ad49a1b2c15b61cb29080fd957
release-management:generate-issue-content-providers:649bd0f56af072754af3e5c2580d7039
release-management:generate-issue-content-providers:767a85195f6e686df63b8f8ea0fb7142
release-management:prepare-airflow-package:3ac14ea6d2b09614959c0ec4fd564789
release-management:prepare-provider-documentation:d2aade2a4f4fb176a89e4db0c1226386
release-management:prepare-provider-packages:7f5ca821bb9c5f80e1473b7b6584adcc
release-management:prepare-provider-documentation:8dab0cba3d0bf3d36ec60d71c8c23d50
release-management:prepare-provider-packages:9d803d0eb5f55d1a178fff2f7951eec8
release-management:release-prod-images:c9bc40938e0efad49e51ef66e83f9527
release-management:start-rc-process:6aafbaceabd7b67b9a1af4c2f59abc4c
release-management:start-release:acb384d86e02ff5fde1bf971897be17c
release-management:verify-provider-packages:88bd609aff6d09d52ab8d80d6e055e7b
release-management:493c6163a3aa0ed6f7dbd4613afc7f3c
release-management:ff43546fa41c3d8a028a32b37036a362
setup:autocomplete:03343478bf1d0cf9c101d454cdb63b68
setup:check-all-params-in-groups:4d0f8c19cbdb56290055d863b08a3376
setup:config:3ffcd35dd24b486ddf1d08b797e3d017
Expand Down
90 changes: 45 additions & 45 deletions images/breeze/output-commands.svg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
128 changes: 62 additions & 66 deletions images/breeze/output_build-docs.svg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
30 changes: 15 additions & 15 deletions images/breeze/output_release-management.svg
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit b4d6e83

Please sign in to comment.