Skip to content

[GLUTEN-10926][VL] Update Spark mirror to speed up downloading in CI docker image#11143

Merged
philo-he merged 1 commit intoapache:mainfrom
zhouyuan:wip_docker_image_merge_refine2
Nov 22, 2025
Merged

[GLUTEN-10926][VL] Update Spark mirror to speed up downloading in CI docker image#11143
philo-he merged 1 commit intoapache:mainfrom
zhouyuan:wip_docker_image_merge_refine2

Conversation

@zhouyuan
Copy link
Copy Markdown
Member

@zhouyuan zhouyuan commented Nov 21, 2025

What changes are proposed in this pull request?

The mirror of "https://www.apache.org/dyn/closer.lua" is very slow and the docker image build timeout
This patch adds a new mirror to download Spark binary/source

How was this patch tested?

locally verified

Related issue: #10926

Signed-off-by: Yuan <yuanzhou@apache.org>
@github-actions github-actions Bot added the INFRA label Nov 21, 2025
@zhouyuan zhouyuan changed the title [GLUTEN-10926][VL] Update Spark mirror to speed up in CI docker image [GLUTEN-10926][VL] Update Spark mirror to speed up downloading in CI docker image Nov 21, 2025
@zhouyuan zhouyuan requested a review from philo-he November 21, 2025 13:58
@FelixYBW
Copy link
Copy Markdown
Contributor

can we push the base image with spark and .m2 installed to github hub?

@zhouyuan
Copy link
Copy Markdown
Member Author

zhouyuan commented Nov 22, 2025

can we push the base image with spark and .m2 installed to github hub?

We should have installed the basic dependencies in centos8/9 dynamic images already.
https://github.com/apache/incubator-gluten/blob/main/dev/docker/Dockerfile.centos8-dynamic-build#L61
For the TPC tests we are using the clean images from docker hub(different version of centos/ubuntu), we should be able to make a cache for the m2 folder and reuse it. Will follow up on this.

Copy link
Copy Markdown
Member

@philo-he philo-he left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Thanks.

@philo-he philo-he merged commit a458d0a into apache:main Nov 22, 2025
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants