Skip to content

Commit

Permalink
[SPARK-46906][INFRA][3.5] Bump python libraries (pandas, pyarrow) in …
Browse files Browse the repository at this point in the history
…Docker image for release script

### What changes were proposed in this pull request?

This PR proposes to bump python libraries (pandas to 2.0.3, pyarrow to 4.0.0) in Docker image for release script.

### Why are the changes needed?

Without this change, release script (do-release-docker.sh) fails on docs phase. Changing this fixes the release process against branch-3.5.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Confirmed with dry-run of release script against branch-3.5.

`dev/create-release/do-release-docker.sh -d ~/spark-release -n -s docs`

```
Generating HTML files for SQL API documentation.
INFO    -  Cleaning site directory
INFO    -  Building documentation to directory: /opt/spark-rm/output/spark/sql/site
INFO    -  Documentation built in 0.85 seconds
/opt/spark-rm/output/spark/sql
Moving back into docs dir.
Making directory api/sql
cp -r ../sql/site/. api/sql
            Source: /opt/spark-rm/output/spark/docs
       Destination: /opt/spark-rm/output/spark/docs/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
                    done in 7.469 seconds.
 Auto-regeneration: disabled. Use --watch to enable.
```

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes #45111 from HeartSaVioR/SPARK-46906-3.5.

Authored-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensource@gmail.com>
  • Loading branch information
HeartSaVioR committed Feb 15, 2024
1 parent ea6b257 commit 9b4778f
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion dev/create-release/spark-rm/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ ARG APT_INSTALL="apt-get install --no-install-recommends -y"
# We should use the latest Sphinx version once this is fixed.
# TODO(SPARK-35375): Jinja2 3.0.0+ causes error when building with Sphinx.
# See also https://issues.apache.org/jira/browse/SPARK-35375.
ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.20.3 pydata_sphinx_theme==0.8.0 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3 sphinx-copybutton==0.5.2 pandas==1.5.3 pyarrow==3.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.56.0 protobuf==4.21.6 grpcio-status==1.56.0 googleapis-common-protos==1.56.4"
ARG PIP_PKGS="sphinx==3.0.4 mkdocs==1.1.2 numpy==1.20.3 pydata_sphinx_theme==0.8.0 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==2.11.3 twine==3.4.1 sphinx-plotly-directive==0.1.3 sphinx-copybutton==0.5.2 pandas==2.0.3 pyarrow==4.0.0 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.56.0 protobuf==4.21.6 grpcio-status==1.56.0 googleapis-common-protos==1.56.4"
ARG GEM_PKGS="bundler:2.3.8"

# Install extra needed repos and refresh.
Expand Down

0 comments on commit 9b4778f

Please sign in to comment.