Skip to content

Commit

Permalink
[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This is a follow-up for SPARK-34346 which causes a flakiness due to `core-site.xml` test resource file addition. This PR aims to remove the test resource `core/src/test/resources/core-site.xml` from `core` module.

### Why are the changes needed?

Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub Action and Jenkins.
```
$ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn
...
[info] YarnClusterSuite:
[info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds)
[info]   FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:210)
```

To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn` module's `yarn/Client`, but it seems an overkill in `core` module.
```
// SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs so that user's
// environments do not interfere with tests. This allows a special env variable during
// tests so that custom conf dirs can be used by unit tests.
val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++
  (if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil)
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

Closes #31515 from dongjoon-hyun/SPARK-34346-2.

Authored-by: Dongjoon Hyun <dhyun@apple.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
(cherry picked from commit dcaf62a)
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
dongjoon-hyun authored and HyukjinKwon committed Feb 8, 2021
1 parent b223135 commit 579e158
Show file tree
Hide file tree
Showing 2 changed files with 2 additions and 25 deletions.
24 changes: 0 additions & 24 deletions core/src/test/resources/core-site.xml

This file was deleted.

3 changes: 2 additions & 1 deletion core/src/test/scala/org/apache/spark/SparkContextSuite.scala
Original file line number Diff line number Diff line change
Expand Up @@ -979,11 +979,12 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu
val testKey = "hadoop.tmp.dir"
val bufferKey = "io.file.buffer.size"
val hadoopConf0 = new Configuration()
hadoopConf0.set(testKey, "/tmp/hive_zero")

val hiveConfFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml")
assert(hiveConfFile != null)
hadoopConf0.addResource(hiveConfFile)
assert(hadoopConf0.get(testKey) === "/tmp/hive_one")
assert(hadoopConf0.get(testKey) === "/tmp/hive_zero")
assert(hadoopConf0.get(bufferKey) === "201811")

val sparkConf = new SparkConf()
Expand Down

0 comments on commit 579e158

Please sign in to comment.