-
Notifications
You must be signed in to change notification settings - Fork 194
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update Spark Docker Images publish workflow #458
Conversation
Merged to asf-site branch. |
@HyukjinKwon @zhengruifeng @yaooqinn Thanks all! If no more comments, I will publish the docker image with new tags for 3.4.0 version later today! |
I hitted a issue when I try to push the |
The INFRA-24476 had been resolved, and docker image already published at https://hub.docker.com/r/apache/spark/tags |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you!
Hi, @Yikun , @HyukjinKwon , @yaooqinn , @zhengruifeng . it seems that this PR introduced two user-facing breaking changes
If these breaking changes are intentional, I'm wondering if we discuss the rationals in the |
cc @sunchao , too |
Thanks for catch and reminder it! @dongjoon-hyun For java17, I think we can switch to java17 in docker image. For tag, it's intentional. In 3.4.0, It should be a new feature but not breaking changes. (But for latest tag where to point is a breaking changes). So do you have any suggestion how we could migrate to new image fluently? Maybe
|
It's already Java 17 by default.
|
Yes, I meant we still need to update in apache/spark-docker. |
Given (1) and (2), I don't think this PR claims to maintain an existing manual publish process. In other words, this PR already burn the bridge. I believe |
The reason of we remove
At this point, below might the solutions to continue the work: Solution 1:
Solution 2:
|
@Yikun . We cannot simply remove anything like this PR. The official way to adopt new standard is to send the email on dev mailing list and build a community consensus. In other words, you need to inform and persuade.
As you know, Apache Spark dev mailing list is the only official way to build a community consensus. All the other channels (GitHub or even user mailing list). Please shoot the email. :) |
Sure! I will send the mail in this week! |
Probably these are guidelines but not hard and fast rules. And I didn't see the prefix |
@yaooqinn I also raise a issue on DOI as our discussion input: docker-library/official-images#14506 . |
Yeah, let's discuss in the dev mailing list, and set that down. I personally think it's fine to make some breaking changes at this moment, but the more important thing is that we should set that down after the discussion so we don't make breaking changes in the future. |
This PR try to update
Create and upload Spark Docker Images
to use new workflow, the new workflow will first build docker image, and then test (k8s / standalone) the image, finally publish the docker image, see example in here.The different between old workflow and new workflow:
After
We already configure the
DOCKER_USER
andDOCKER_TOKEN
in https://issues.apache.org/jira/browse/INFRA-23882 .After this patch merged, the 3.4.0 image can be published as first docker image to
apache/spark
dockerhub (new version). In future release, only new version docker images will be published.