[SPARK-38430][K8S][DOCS] Add SBT commands to K8s IT README#35745
[SPARK-38430][K8S][DOCS] Add SBT commands to K8s IT README#35745williamhyun wants to merge 2 commits intoapache:masterfrom williamhyun:sbtdoc
SBT commands to K8s IT README#35745Conversation
williamhyun
changed the title
dongjoon-hyun
changed the title
SBT commands to K8s IT README
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
+1, LGTM. Thank you, @williamhyun .
Merged to master/3.2.
### What changes were proposed in this pull request? This PR aims to add SBT commands to K8s IT README. ### Why are the changes needed? This will introduce new SBT commands to developers. ### Does this PR introduce _any_ user-facing change? No, this is a dev-only change. ### How was this patch tested? Manual. Closes #35745 from williamhyun/sbtdoc. Authored-by: William Hyun <william@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 3bbc43d) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
martin-g
left a comment
martin-g
left a comment
There was a problem hiding this comment.
I am a bit late with the review - the PR is merged already. I hope the comments are useful anyway.
| -Dspark.kubernetes.test.imageTag=2022-03-06 \ | ||
| 'kubernetes-integration-tests/test' | ||
|
|
||
| The following is an example to rerun tests with the pre-built image. |
There was a problem hiding this comment.
Thanks for adding this documentation!
It is not very clear to me what is the difference between .../test and .../runIts ?
Is it that the first one (re-)builds the image and the second would fail if there is no image ?
There was a problem hiding this comment.
Your understanding is correct and consistent with what @williamhyun described.
There was a problem hiding this comment.
To make it sure,
../testis slower and sometimes accidentally overwrite the images.../runItsis faster and guarantees that there will be no new image.
|
|
||
| You can use SBT in the same way to build image and run all K8s integration tests except Minikube-only ones. | ||
|
|
||
| build/sbt -Psparkr -Pkubernetes -Pkubernetes-integration-tests \ |
There was a problem hiding this comment.
Not directly related to this PR but for ./build/mvn one has to add a profile for Hadoop 2/3 (e.g. -Phadoop-3). Otherwise the build fails due to missing AWS classes.
There was a problem hiding this comment.
@martin-g . Why do you add your irrelevant comments like that? Actually, it doesn't look like a good way to collaborate because this is open to all the community member and your comments are distracting the the other people from the PR content itself.
There was a problem hiding this comment.
I'd like to recommend you to file a JIRA officially instead and go for it. :)
|
Sorry for commenting on closed PR but there was no time for others to review it. It was closed very soon after being opened. I just tried to say that the new documentation was not clear enough for me (a newbie). I had to read it few times and to consult with the code to understand the difference. I was hoping that it might be improved. I noticed that the PR is closed when I was finishing the code review - there were no radio buttons to Comment/Approve/Request changes. I will file tickets for both issues! |
|
It seems that you misunderstand the point. Apache Spark uses post-commit review also actively. That's totally fine and helpful. My comment is about only your irrelevant comment, #35745 (comment) . That's no good, isn't it?
|
### What changes were proposed in this pull request? This PR aims to add SBT commands to K8s IT README. ### Why are the changes needed? This will introduce new SBT commands to developers. ### Does this PR introduce _any_ user-facing change? No, this is a dev-only change. ### How was this patch tested? Manual. Closes apache#35745 from williamhyun/sbtdoc. Authored-by: William Hyun <william@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 3bbc43d) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 7eafadb) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
3 participants
What changes were proposed in this pull request?
This PR aims to add SBT commands to K8s IT README.
Why are the changes needed?
This will introduce new SBT commands to developers.
Does this PR introduce any user-facing change?
No, this is a dev-only change.
How was this patch tested?
Manual.