[SPARK-34346][CORE][TESTS][FOLLOWUP] Fix UT by removing core-site.xml #31515
Conversation
| @@ -1155,11 +1155,12 @@ class SparkContextSuite extends SparkFunSuite with LocalSparkContext with Eventu | |||
| val testKey = "hadoop.tmp.dir" | |||
| val bufferKey = "io.file.buffer.size" | |||
| val hadoopConf0 = new Configuration() | |||
| hadoopConf0.set(testKey, "/tmp/hive_zero") | |||
There was a problem hiding this comment.
This is still a Hadoop conf although it's Overlay property.
|
|
||
| val hiveConfFile = Utils.getContextOrSparkClassLoader.getResource("hive-site.xml") | ||
| assert(hiveConfFile != null) | ||
| hadoopConf0.addResource(hiveConfFile) | ||
| assert(hadoopConf0.get(testKey) === "/tmp/hive_one") | ||
| assert(hadoopConf0.get(testKey) === "/tmp/hive_zero") |
There was a problem hiding this comment.
This is okay because Apache Spark UT aims to test line 1170 ~ 1172.
assert(sc.hadoopConfiguration.get(testKey) === "/tmp/hive_one",
"hive configs have higher priority than hadoop ones ")
assert(sc.hadoopConfiguration.get(bufferKey).toInt === 65536,
"spark configs have higher priority than hive ones")
There was a problem hiding this comment.
w/ this change, "/tmp/hive_one" in hive-site.xml overlays "/tmp/hive_{user}" not "/tmp/hive_zero", which happens in SparkHadoopUtil.appendS3AndSparkHadoopHiveConfigurations.
As hadoopConf0 is not passed to SparkHadoopUtil, while core-site.xml is, so here to reduce test flakiness, we can remove those asserts for hadoopConf0, as they are just used to check hive-site.xml overrides core-site.xml.
There was a problem hiding this comment.
@yaooqinn, can you open a PR for that, or push some changes into this PR? I think the test is not flaky but broken. It doesn't look obvious because we don't always run YarnClusterSuite.
|
Could you review this, @yaooqinn , @HyukjinKwon , @cloud-fan , @srowen ? |
|
Kubernetes integration test starting |
|
Hey guys, let me just merge this for now. I do think the test did not pass and it is broken. @yaooqinn, can you make another followup to remove the asserts? |
|
Merged to master, branch-3.1 and branch-3.0. |
### What changes were proposed in this pull request? This is a follow-up for SPARK-34346 which causes a flakiness due to `core-site.xml` test resource file addition. This PR aims to remove the test resource `core/src/test/resources/core-site.xml` from `core` module. ### Why are the changes needed? Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub Action and Jenkins. ``` $ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn ... [info] YarnClusterSuite: [info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds) [info] FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:210) ``` To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn` module's `yarn/Client`, but it seems an overkill in `core` module. ``` // SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs so that user's // environments do not interfere with tests. This allows a special env variable during // tests so that custom conf dirs can be used by unit tests. val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++ (if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #31515 from dongjoon-hyun/SPARK-34346-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit dcaf62a) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
### What changes were proposed in this pull request? This is a follow-up for SPARK-34346 which causes a flakiness due to `core-site.xml` test resource file addition. This PR aims to remove the test resource `core/src/test/resources/core-site.xml` from `core` module. ### Why are the changes needed? Due to the test resource `core-site.xml`, YARN UT becomes flaky in GitHub Action and Jenkins. ``` $ build/sbt "yarn/testOnly *.YarnClusterSuite -- -z SPARK-16414" -Pyarn ... [info] YarnClusterSuite: [info] - yarn-cluster should respect conf overrides in SparkHadoopUtil (SPARK-16414, SPARK-23630) *** FAILED *** (20 seconds, 209 milliseconds) [info] FAILED did not equal FINISHED (stdout/stderr was not captured) (BaseYarnClusterSuite.scala:210) ``` To isolate more, we may use `SPARK_TEST_HADOOP_CONF_DIR` like `yarn` module's `yarn/Client`, but it seems an overkill in `core` module. ``` // SPARK-23630: during testing, Spark scripts filter out hadoop conf dirs so that user's // environments do not interfere with tests. This allows a special env variable during // tests so that custom conf dirs can be used by unit tests. val confDirs = Seq("HADOOP_CONF_DIR", "YARN_CONF_DIR") ++ (if (Utils.isTesting) Seq("SPARK_TEST_HADOOP_CONF_DIR") else Nil) ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the CIs. Closes #31515 from dongjoon-hyun/SPARK-34346-2. Authored-by: Dongjoon Hyun <dhyun@apple.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit dcaf62a) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
|
Kubernetes integration test status success |
|
OK~ |
|
Test build #134993 has finished for PR 31515 at commit
|
|
Thank you, @srowen , @yaooqinn , @HyukjinKwon . |
What changes were proposed in this pull request?
This is a follow-up for SPARK-34346 which causes a flakiness due to
core-site.xmltest resource file addition. This PR aims to remove the test resourcecore/src/test/resources/core-site.xmlfromcoremodule.Why are the changes needed?
Due to the test resource
core-site.xml, YARN UT becomes flaky in GitHub Action and Jenkins.To isolate more, we may use
SPARK_TEST_HADOOP_CONF_DIRlikeyarnmodule'syarn/Client, but it seems an overkill incoremodule.Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass the CIs.