New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test in BasicTestsSuite
#35318
Conversation
…n BasicTestsSuite
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep, the file path should be trim the spark home prefix like: resource-managers/kubernetes/integration-tests/target/scala-2.12/test-classes/pagerank_data.txt
It also passed in my env with k8s v1.23.2 Platform:"linux/amd64".
[info] - Run SparkRemoteFileTest using a remote data file (12 seconds, 701 milliseconds)
LGTM, thanks!
Thank you, @Yikun . This is a part of recovering K8s integration suite which has been broken for a while. |
Could you review this please, @viirya ? |
Thank you so much! |
lgtm |
…n `BasicTestsSuite` ### What changes were proposed in this pull request? This PR aims to use `relativePath` for K8s remote file test in `BasicTestsSuite`. ### Why are the changes needed? To make `Run SparkRemoteFileTest using a remote data file` test pass. **BEFORE** ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds) [info] The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452) ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds) ... ``` Closes #35318 from dongjoon-hyun/SPARK-38022. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 2773228) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…n `BasicTestsSuite` ### What changes were proposed in this pull request? This PR aims to use `relativePath` for K8s remote file test in `BasicTestsSuite`. ### Why are the changes needed? To make `Run SparkRemoteFileTest using a remote data file` test pass. **BEFORE** ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds) [info] The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452) ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds) ... ``` Closes apache#35318 from dongjoon-hyun/SPARK-38022. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
…n `BasicTestsSuite` ### What changes were proposed in this pull request? This PR aims to use `relativePath` for K8s remote file test in `BasicTestsSuite`. ### Why are the changes needed? To make `Run SparkRemoteFileTest using a remote data file` test pass. **BEFORE** ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds) [info] The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452) ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? ``` $ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test" ... [info] KubernetesSuite: ... [info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds) ... ``` Closes apache#35318 from dongjoon-hyun/SPARK-38022. Authored-by: Dongjoon Hyun <dongjoon@apache.org> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org> (cherry picked from commit 2773228) Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
What changes were proposed in this pull request?
This PR aims to use
relativePath
for K8s remote file test inBasicTestsSuite
.Why are the changes needed?
To make
Run SparkRemoteFileTest using a remote data file
test pass.BEFORE
Does this PR introduce any user-facing change?
No.
How was this patch tested?