From 61435b42fdc4071f35aba6af9248ff9ad8fc8514 Mon Sep 17 00:00:00 2001 From: Hyukjin Kwon Date: Wed, 13 Sep 2023 21:20:19 +0900 Subject: [PATCH] [SPARK-45142][INFRA] Specify the range for Spark Connect dependencies in pyspark base image ### What changes were proposed in this pull request? This PR proposes to pin the dependencies related to Spark Connect in its base image according to the range we support. See also https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst#dependencies ### Why are the changes needed? To properly test the dependency versions we support. ### Does this PR introduce _any_ user-facing change? No, dev-only. ### How was this patch tested? In this PR, it will be tested. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #42898 from HyukjinKwon/SPARK-45142. Authored-by: Hyukjin Kwon Signed-off-by: Hyukjin Kwon --- dev/infra/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/dev/infra/Dockerfile b/dev/infra/Dockerfile index feee741500472..99423ce072cd7 100644 --- a/dev/infra/Dockerfile +++ b/dev/infra/Dockerfile @@ -88,7 +88,7 @@ RUN pypy3 -m pip install numpy 'pandas<=2.0.3' scipy coverage matplotlib RUN python3.9 -m pip install numpy 'pyarrow==12.0.1' 'pandas<=2.0.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*' # Add Python deps for Spark Connect. -RUN python3.9 -m pip install grpcio protobuf googleapis-common-protos grpcio-status +RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' 'googleapis-common-protos==1.56.4' # Add torch as a testing dependency for TorchDistributor RUN python3.9 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu