[SPARK-56889][INFRA][FOLLOWUP] Restore python3.10 in dev/infra/Dockerfile#55946
Closed
zhengruifeng wants to merge 1 commit into
Closed
[SPARK-56889][INFRA][FOLLOWUP] Restore python3.10 in dev/infra/Dockerfile#55946zhengruifeng wants to merge 1 commit into
zhengruifeng wants to merge 1 commit into
Conversation
…file Generated-by: Claude Code (model: claude-opus-4-7)
Contributor
Author
|
maybe we don't need this image in master/4.x, but let me restore it first |
LuciferYang
approved these changes
May 18, 2026
dongjoon-hyun
requested changes
May 18, 2026
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you for reporting, but shall we fix the root cause?
According to the reporting, we can install python3-pyparsing before add-apt-repository ppa:deadsnakes/ppa, can't we?
Member
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
Thank you, @zhengruifeng and @LuciferYang . I made an alternative follow-up.
Member
|
It seems that the alternative isn't enough. Let's merge this to recover CIs. |
dongjoon-hyun
approved these changes
May 18, 2026
dongjoon-hyun
pushed a commit
that referenced
this pull request
May 18, 2026
…file ### What changes were proposed in this pull request? This is a partial revert of #55914 (SPARK-56889) restricted to `dev/infra/Dockerfile`. It restores the file to its state at `0a0d31bea00~1` — switching the system Python used in the base CI image from `python3.11` back to `python3.10`. All other 20 files changed by #55914 are kept as-is. ### Why are the changes needed? The `Build / Cache base image` workflow has been failing on every branch since #55914 was merged (2026-05-16): https://github.com/apache/spark/actions/runs/26010261434/job/76449371588 The failure aborts at the first `Build and push` step (the base `./dev/infra/` image), with: ``` RUN add-apt-repository ppa:deadsnakes/ppa ... ModuleNotFoundError: No module named 'pyparsing' ERROR: process "/bin/sh -c add-apt-repository ppa:deadsnakes/ppa" did not complete successfully: exit code: 1 ``` This breaks the cache build for all downstream image jobs, since the base layer is shared. Restoring the previous Dockerfile is the smallest change that unblocks the cache workflow while a forward fix is investigated. ### Does this PR introduce _any_ user-facing change? No. CI-only change. PySpark itself is unaffected — only the system Python inside the test image. ### How was this patch tested? This PR re-runs `Build / Cache base image` against the restored Dockerfile. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (model: claude-opus-4-7) Closes #55946 from zhengruifeng/restore-infra-dockerfile-python310. Authored-by: Ruifeng Zheng <ruifengz@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 3608538) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
Member
|
Merged to master/4.x. For the CIs, I'll keep monitoring the effective Python version. |
Member
|
Thank you again, @zhengruifeng and @LuciferYang . |
zhengruifeng
pushed a commit
that referenced
this pull request
May 19, 2026
…ng from dockerfile ### What changes were proposed in this pull request? Remove `python3-pyparsing` from `dev/infra/Dockerfile`. ### Why are the changes needed? This line was added in #55956 as an attempt to fix the image build issue caused by removing 3.10. This was not the correct fix. The reason CI failed was because we somehow uninstalled `pyparsing` from system python3. We mitigated the issue later in #55946 . We should not keep this line around - it could confuse people in the future. This specific dockerfile is only used by 3.5. We should just retire the file after 3.5 EOL. We don't have to upgrade the 3.10 part (we didn't touch the 3.9 part of the file). We can just keep it as it is and delete it in the future. As we are using the docker file for 3.5, we probably shouldn't remove 3.10 because it's actually supported in 3.5. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should work. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55965 from gaogaotiantian/remove-pyparsing. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
zhengruifeng
pushed a commit
that referenced
this pull request
May 19, 2026
…ng from dockerfile ### What changes were proposed in this pull request? Remove `python3-pyparsing` from `dev/infra/Dockerfile`. ### Why are the changes needed? This line was added in #55956 as an attempt to fix the image build issue caused by removing 3.10. This was not the correct fix. The reason CI failed was because we somehow uninstalled `pyparsing` from system python3. We mitigated the issue later in #55946 . We should not keep this line around - it could confuse people in the future. This specific dockerfile is only used by 3.5. We should just retire the file after 3.5 EOL. We don't have to upgrade the 3.10 part (we didn't touch the 3.9 part of the file). We can just keep it as it is and delete it in the future. As we are using the docker file for 3.5, we probably shouldn't remove 3.10 because it's actually supported in 3.5. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? CI should work. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #55965 from gaogaotiantian/remove-pyparsing. Authored-by: Tian Gao <gaogaotiantian@hotmail.com> Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com> (cherry picked from commit eeecea4) Signed-off-by: Ruifeng Zheng <ruifengz@foxmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This is a partial revert of #55914 (SPARK-56889) restricted to
dev/infra/Dockerfile. It restores the file to its state at0a0d31bea00~1— switching the system Python used in the base CI image frompython3.11back topython3.10. All other 20 files changed by #55914 are kept as-is.Why are the changes needed?
The
Build / Cache base imageworkflow has been failing on every branch since #55914 was merged (2026-05-16): https://github.com/apache/spark/actions/runs/26010261434/job/76449371588The failure aborts at the first
Build and pushstep (the base./dev/infra/image), with:This breaks the cache build for all downstream image jobs, since the base layer is shared. Restoring the previous Dockerfile is the smallest change that unblocks the cache workflow while a forward fix is investigated.
Does this PR introduce any user-facing change?
No. CI-only change. PySpark itself is unaffected — only the system Python inside the test image.
How was this patch tested?
This PR re-runs
Build / Cache base imageagainst the restored Dockerfile.Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (model: claude-opus-4-7)