Skip to content

Commit

Permalink
[SPARK-38022][K8S][TESTS] Use relativePath for K8s remote file test i…
Browse files Browse the repository at this point in the history
…n `BasicTestsSuite`

### What changes were proposed in this pull request?

This PR aims to use `relativePath` for K8s remote file test in `BasicTestsSuite`.

### Why are the changes needed?

To make `Run SparkRemoteFileTest using a remote data file` test pass.

**BEFORE**
```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file *** FAILED *** (3 minutes, 3 seconds)
[info]   The code passed to eventually never returned normally. Attempted 190 times over 3.012265011116667 minutes. Last failure message: false was not true. (KubernetesSuite.scala:452)
...
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

```
$ build/sbt -Pkubernetes -Pkubernetes-integration-tests -Dspark.kubernetes.test.dockerFile=resource-managers/kubernetes/docker/src/main/dockerfiles/spark/Dockerfile.java17 -Dtest.exclude.tags=minikube,r "kubernetes-integration-tests/test"
...
[info] KubernetesSuite:
...
[info] - Run SparkRemoteFileTest using a remote data file (8 seconds, 608 milliseconds)
...
```

Closes apache#35318 from dongjoon-hyun/SPARK-38022.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit 2773228)
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
  • Loading branch information
dongjoon-hyun authored and viirya committed Jan 29, 2022
1 parent fb1b1d5 commit c3c2fb5
Showing 1 changed file with 2 additions and 2 deletions.
Expand Up @@ -102,8 +102,8 @@ private[spark] trait BasicTestsSuite { k8sSuite: KubernetesSuite =>
test("Run SparkRemoteFileTest using a remote data file", k8sTestTag) {
assert(sys.props.contains("spark.test.home"), "spark.test.home is not set!")
TestUtils.withHttpServer(sys.props("spark.test.home")) { baseURL =>
sparkAppConf
.set("spark.files", baseURL.toString + REMOTE_PAGE_RANK_DATA_FILE)
sparkAppConf.set("spark.files", baseURL.toString +
REMOTE_PAGE_RANK_DATA_FILE.replace(sys.props("spark.test.home"), "").substring(1))
runSparkRemoteCheckAndVerifyCompletion(appArgs = Array(REMOTE_PAGE_RANK_FILE_NAME))
}
}
Expand Down

0 comments on commit c3c2fb5

Please sign in to comment.