Skip to content

Commit

Permalink
[SPARK-45142][INFRA] Specify the range for Spark Connect dependencies…
Browse files Browse the repository at this point in the history
… in pyspark base image

### What changes were proposed in this pull request?

This PR proposes to pin the dependencies related to Spark Connect in its base image according to the range we support.
See also https://github.com/apache/spark/blob/master/python/docs/source/getting_started/install.rst#dependencies

### Why are the changes needed?

To properly test the dependency versions we support.

### Does this PR introduce _any_ user-facing change?

No, dev-only.

### How was this patch tested?

In this PR, it will be tested.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#42898 from HyukjinKwon/SPARK-45142.

Authored-by: Hyukjin Kwon <gurwls223@apache.org>
Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
  • Loading branch information
HyukjinKwon committed Sep 13, 2023
1 parent 2710dbe commit 61435b4
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dev/infra/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ RUN pypy3 -m pip install numpy 'pandas<=2.0.3' scipy coverage matplotlib
RUN python3.9 -m pip install numpy 'pyarrow==12.0.1' 'pandas<=2.0.3' scipy unittest-xml-reporting plotly>=4.8 'mlflow>=2.3.1' coverage matplotlib openpyxl 'memory-profiler==0.60.0' 'scikit-learn==1.1.*'

# Add Python deps for Spark Connect.
RUN python3.9 -m pip install grpcio protobuf googleapis-common-protos grpcio-status
RUN python3.9 -m pip install 'grpcio>=1.48,<1.57' 'grpcio-status>=1.48,<1.57' 'protobuf==3.20.3' 'googleapis-common-protos==1.56.4'

# Add torch as a testing dependency for TorchDistributor
RUN python3.9 -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpu
Expand Down

0 comments on commit 61435b4

Please sign in to comment.