Skip to content

Commit

Permalink
Bump min version for grpcio-status in spark provider (#36662)
Browse files Browse the repository at this point in the history
Previously we limited grpcio minimum version to stop backtracking
of `pip` from happening and we could not do it in the limits of
spark provider, becaue some google dependencies used it and
conflicted with it. This problem is now gone as we have newer
versions of google dependencies and we can not only safely move
it to spark provider but also bump it slightly higher to limit
the amount of backtracking we need to do.

Extracted from #36537
  • Loading branch information
potiuk committed Jan 8, 2024
1 parent 3dc99d8 commit ded01a5
Show file tree
Hide file tree
Showing 3 changed files with 3 additions and 6 deletions.
5 changes: 1 addition & 4 deletions Dockerfile.ci
Expand Up @@ -1117,14 +1117,11 @@ RUN echo "Airflow version: ${AIRFLOW_VERSION}"
# force them on the main Airflow package. Currently we need no extra limits as PIP 23.1+ has much better
# dependency resolution and we do not need to limit the versions of the dependencies
#
# Without grpcio-status limit, pip gets into very long backtracking
# We should attempt to remove it in the future
#
# Aiobotocore is limited for eager upgrade because it either causes a long backtracking or
# conflict when we do not limit it. It seems that `pip` has a hard time figuring the right
# combination of dependencies for aiobotocore, botocore, boto3 and s3fs together
#
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="grpcio-status>=1.55.0 aiobotocore>=2.5.4"
ARG EAGER_UPGRADE_ADDITIONAL_REQUIREMENTS="aiobotocore>=2.5.4"
ARG UPGRADE_TO_NEWER_DEPENDENCIES="false"
ARG VERSION_SUFFIX_FOR_PYPI=""

Expand Down
2 changes: 1 addition & 1 deletion airflow/providers/apache/spark/provider.yaml
Expand Up @@ -55,7 +55,7 @@ versions:
dependencies:
- apache-airflow>=2.6.0
- pyspark
- grpcio-status
- grpcio-status>=1.59.0

integrations:
- integration-name: Apache Spark
Expand Down
2 changes: 1 addition & 1 deletion generated/provider_dependencies.json
Expand Up @@ -206,7 +206,7 @@
"apache.spark": {
"deps": [
"apache-airflow>=2.6.0",
"grpcio-status",
"grpcio-status>=1.59.0",
"pyspark"
],
"cross-providers-deps": [
Expand Down

0 comments on commit ded01a5

Please sign in to comment.