From 926f756d4a9ef58f97473a65049cde6c08a005e9 Mon Sep 17 00:00:00 2001 From: Dongjoon Hyun Date: Sat, 8 Jan 2022 09:28:24 -0800 Subject: [PATCH] [SPARK-37844][CORE][TESTS] Remove `slf4j-log4j12` transitive test dependency from `hadoop-minikdc` ### What changes were proposed in this pull request? This PR removes `slf4j-log4j12` dependency from `hadoop-minikdc`. ### Why are the changes needed? This causes `Maven` test failure. **BEFORE** ``` $ build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.SparkSubmitSuite test ... 00:09:19.585 - launch simple application with spark-submit *** FAILED *** 00:10:17.712 Timeout of '/Users/m1/.jenkins/workspace/master-sbt/bin/spark-submit' '--class' 'org.apache.spark.deploy.SimpleApplicationTest' '--name' 'testApp' '--master' 'local' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' 'file:/Users/m1/.jenkins/workspace/master-sbt/core/target/tmp/spark-9bc87b17-c585-4661-adf4-f51a3c01586f/testJar-1641631487327.jar' See the log4j logs for more detail. 00:10:17.712 2022-01-08 00:44:47.955 - stderr> SLF4J: Class path contains multiple SLF4J bindings. 00:10:17.712 2022-01-08 00:44:47.955 - stderr> SLF4J: Found binding in [jar:file:/Users/m1/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] 00:10:17.712 2022-01-08 00:44:47.956 - stderr> SLF4J: Found binding in [jar:file:/Users/m1/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class] 00:10:17.712 2022-01-08 00:44:47.956 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. 00:10:17.712 2022-01-08 00:44:47.956 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 00:10:17.712 2022-01-08 00:44:49.246 - stderr> Exception in thread "Executor task launch worker for task 0.0 in stage 0.0 (TID 0)" java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC 00:10:17.712 2022-01-08 00:44:49.246 - stderr> at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:751) 00:10:17.712 2022-01-08 00:44:49.246 - stderr> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:441) 00:10:17.712 2022-01-08 00:44:49.246 - stderr> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) 00:10:17.712 2022-01-08 00:44:49.246 - stderr> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) 00:10:17.712 2022-01-08 00:44:49.246 - stderr> at java.base/java.lang.Thread.run(Thread.java:833) (SparkSubmitTestUtils.scala:107) ``` ``` [INFO] +- org.apache.hadoop:hadoop-minikdc:jar:3.3.1:test [INFO] | +- org.apache.kerby:kerb-simplekdc:jar:1.0.1:test ... [INFO] | | \- org.apache.kerby:kerby-xdr:jar:1.0.1:test [INFO] | \- org.slf4j:slf4j-log4j12:jar:1.7.30:test ``` **AFTER** ``` $ build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.SparkSubmitSuite test ... SparkSubmitSuite: - prints usage on empty input - prints usage with only --help - prints error with unrecognized options - handle binary specified but not class - handles arguments with --key=val - handles arguments to user program - handles arguments to user program with name collision - print the right queue name - SPARK-24241: do not fail fast if executor num is 0 when dynamic allocation is enabled - specify deploy mode through configuration - handles YARN cluster mode - handles YARN client mode - SPARK-33530: handles standalone mode with archives - handles standalone cluster mode - handles legacy standalone cluster mode - handles standalone client mode - handles mesos client mode - handles k8s cluster mode - automatically sets mainClass if primary resource is S3 JAR in client mode - automatically sets mainClass if primary resource is S3 JAR in cluster mode - error informatively when mainClass isn't set and S3 JAR doesn't exist - handles confs with flag equivalents - SPARK-21568 ConsoleProgressBar should be enabled only in shells - launch simple application with spark-submit - launch simple application with spark-submit with redaction - includes jars passed in through --jars - includes jars passed through spark.jars.packages and spark.jars.repositories - correctly builds R packages included in a jar with --packages !!! IGNORED !!! - include an external JAR in SparkR !!! CANCELED !!! org.apache.spark.api.r.RUtils.isSparkRInstalled was false SparkR is not installed in this build. (SparkSubmitSuite.scala:740) - resolves command line argument paths correctly - ambiguous archive mapping results in error message - resolves config paths correctly - user classpath first in driver - SPARK_CONF_DIR overrides spark-defaults.conf - support glob path - SPARK-27575: yarn confs should merge new value with existing value - downloadFile - invalid url - downloadFile - file doesn't exist - downloadFile does not download local file - download one file to local - download list of files to local - remove copies of application jar from classpath - Avoid re-upload remote resources in yarn client mode - download remote resource if it is not supported by yarn service - avoid downloading remote resource if it is supported by yarn service - force download from forced schemes - force download for all the schemes - SPARK-32119: Jars and files should be loaded when Executors launch for plugins - start SparkApplication without modifying system properties - support --py-files/spark.submit.pyFiles in non pyspark application - handles natural line delimiters in --properties-file and --conf uniformly - get a Spark configuration from arguments Run completed in 25 seconds, 216 milliseconds. Total number of tests run: 51 Suites: completed 2, aborted 0 Tests: succeeded 51, failed 0, canceled 1, ignored 1, pending 0 All tests passed. ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass the test with Maven. Closes #35143 from dongjoon-hyun/SPARK-37844. Authored-by: Dongjoon Hyun Signed-off-by: Dongjoon Hyun --- pom.xml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/pom.xml b/pom.xml index fdda77c65b07b..323d9cc257a05 100644 --- a/pom.xml +++ b/pom.xml @@ -1365,6 +1365,10 @@ log4j log4j + + org.slf4j + slf4j-log4j12 +