Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file #46512

Closed
wants to merge 2 commits into from
Closed

[SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file #46512

wants to merge 2 commits into from

Conversation

nchammas
Copy link
Contributor

@nchammas nchammas commented May 9, 2024 •
edited

What changes were proposed in this pull request?

Sync the version of Bundler that we are using across various scripts and documentation. Also refresh the Gem lock file.

Why are the changes needed?

We are seeing inconsistent build behavior, likely due to the inconsistent Bundler versions.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

CI + the preview release process.

Was this patch authored or co-authored using generative AI tooling?

No.

@nchammas nchammas mentioned this pull request May 9, 2024
Closed
@nchammas nchammas changed the title Sync Ruby Bundler to 2.4.22 and refresh Gem lock file [SPARK-48222][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file May 9, 2024
@nchammas nchammas marked this pull request as ready for review May 9, 2024 17:26
@dongjoon-hyun
Copy link
Member

Thank you, @nchammas .

cc @cloud-fan although I'm sure he is testing this PR already. 😄

@dongjoon-hyun dongjoon-hyun changed the title [SPARK-48222][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file [SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Gem lock file May 9, 2024
cloud-fan
cloud-fan reviewed May 9, 2024
View reviewed changes
dev/create-release/spark-rm/Dockerfile
@@ -38,7 +38,7 @@ ENV DEBCONF_NONINTERACTIVE_SEEN true
ARG APT_INSTALL="apt-get install --no-install-recommends -y"

ARG PIP_PKGS="sphinx==4.5.0 mkdocs==1.1.2 numpy==1.20.3 pydata_sphinx_theme==0.13.3 ipython==7.19.0 nbsphinx==0.8.0 numpydoc==1.1.0 jinja2==3.1.2 twine==3.4.1 sphinx-plotly-directive==0.1.3 sphinx-copybutton==0.5.2 pandas==2.0.3 pyarrow==10.0.1 plotly==5.4.0 markupsafe==2.0.1 docutils<0.17 grpcio==1.62.0 protobuf==4.21.6 grpcio-status==1.62.0 googleapis-common-protos==1.56.4"
ARG GEM_PKGS="bundler:2.3.8"
ARG GEM_PKGS="bundler:2.4.22"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shall we also install ruby 3?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Eventually, yes, we should migrate to Ruby 3. However, since we have been running CI on Ruby 2, I think it would be safer to keep it at that for now and get the release process unblocked.

cloud-fan
cloud-fan reviewed May 9, 2024
View reviewed changes
docs/Gemfile.lock
rb-fsevent (0.11.2)
rb-inotify (0.10.1)
ffi (~> 1.0)
rexml (3.2.6)
rouge (3.30.0)
safe_yaml (1.0.5)
sass-embedded (1.69.7)
google-protobuf (~> 3.25)
sass-embedded (1.63.6)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm why is this downgraded?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably because I was running a different version of Ruby and/or Bundler when I last updated the lock file.

But as part of this PR I made sure to run Ruby 2.7 and Bundler 2.4.22 to match the CI environment.

Keep in mind that Bundler selects the versions in this lock file based on the Ruby version and required dependencies.

@cloud-fan
Copy link
Contributor

thanks, merging to master! (it's easier for me to test after merging it)

@cloud-fan cloud-fan closed this in 9a28188 May 10, 2024
@cloud-fan
Copy link
Contributor

the Bundler issue is resolved, but I hit a new issue for generating pyspark docs

Configuration error:
There is a programmable error in your configuration file:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/sphinx/config.py", line 332, in eval_config_file
    exec(code, namespace)
  File "/opt/spark-rm/output/spark/python/docs/source/conf.py", line 27, in <module>
    from pyspark.pandas.supported_api_gen import generate_supported_api
  File "/opt/spark-rm/output/spark/python/pyspark/__init__.py", line 53, in <module>
    from pyspark.util import is_remote_only
  File "/opt/spark-rm/output/spark/python/pyspark/util.py", line 33, in <module>
    from pyspark.errors import PySparkRuntimeError
  File "/opt/spark-rm/output/spark/python/pyspark/errors/__init__.py", line 21, in <module>
    from pyspark.errors.exceptions.base import (  # noqa: F401
  File "/opt/spark-rm/output/spark/python/pyspark/errors/exceptions/base.py", line 21, in <module>
    from pyspark.errors.utils import ErrorClassesReader
  File "/opt/spark-rm/output/spark/python/pyspark/errors/utils.py", line 23, in <module>
    from pyspark.errors.error_classes import ERROR_CLASSES_MAP
  File "/opt/spark-rm/output/spark/python/pyspark/errors/error_classes.py", line 26, in <module>
    importlib.resources
AttributeError: module 'importlib.resources' has no attribute 'files'

maybe there is some python library version inconsistency between Github Action and release docker

@cloud-fan
Copy link
Contributor

cc @HyukjinKwon

@dongjoon-hyun
Copy link
Member

It seems that files attribute is added at Python 3.9, but the running python version is Python 3.8, @cloud-fan .

@nchammas
Copy link
Contributor Author

Yes, and we dropped support for Python 3.8 in #46228.

@dongjoon-hyun
Copy link
Member

Ya, as @nchammas mentioned, it seems that we missed to bump Python to 3.9 in spark-rm in the following PR.

@cloud-fan
Copy link
Contributor

I see, I'll install python 3.9 on the release docker image.

@nchammas nchammas deleted the bundler-sync branch May 10, 2024 14:34
JacobZheng0927 pushed a commit to JacobZheng0927/spark that referenced this pull request May 11, 2024
@nchammas @JacobZheng0927
[SPARK-48222][INFRA][DOCS] Sync Ruby Bundler to 2.4.22 and refresh Ge…
4f9dfab 
…m lock file

### What changes were proposed in this pull request?

Sync the version of Bundler that we are using across various scripts and documentation. Also refresh the Gem lock file.

### Why are the changes needed?

We are seeing inconsistent build behavior, likely due to the inconsistent Bundler versions.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

CI + the preview release process.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#46512 from nchammas/bundler-sync.

Authored-by: Nicholas Chammas <nicholas.chammas@gmail.com>
Signed-off-by: Wenchen Fan <wenchen@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dongjoon-hyun dongjoon-hyun dongjoon-hyun left review comments

@cloud-fan cloud-fan cloud-fan approved these changes

Assignees
No one assigned
Labels
BUILD DOCS INFRA
Projects
None yet
Milestone
No milestone
3 participants
@nchammas @dongjoon-hyun @cloud-fan