Skip to content

Commit

Permalink
[SPARK-37844][CORE][TESTS] Remove slf4j-log4j12 transitive test dep…
Browse files Browse the repository at this point in the history
…endency from `hadoop-minikdc`

### What changes were proposed in this pull request?

This PR removes `slf4j-log4j12` dependency from `hadoop-minikdc`.

### Why are the changes needed?

This causes `Maven` test failure.

**BEFORE**
```
$ build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.SparkSubmitSuite test
...
00:09:19.585 - launch simple application with spark-submit *** FAILED ***
00:10:17.712   Timeout of '/Users/m1/.jenkins/workspace/master-sbt/bin/spark-submit' '--class' 'org.apache.spark.deploy.SimpleApplicationTest' '--name' 'testApp' '--master' 'local' '--conf' 'spark.ui.enabled=false' '--conf' 'spark.master.rest.enabled=false' 'file:/Users/m1/.jenkins/workspace/master-sbt/core/target/tmp/spark-9bc87b17-c585-4661-adf4-f51a3c01586f/testJar-1641631487327.jar' See the log4j logs for more detail.
00:10:17.712   2022-01-08 00:44:47.955 - stderr> SLF4J: Class path contains multiple SLF4J bindings.
00:10:17.712   2022-01-08 00:44:47.955 - stderr> SLF4J: Found binding in [jar:file:/Users/m1/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
00:10:17.712   2022-01-08 00:44:47.956 - stderr> SLF4J: Found binding in [jar:file:/Users/m1/.m2/repository/org/apache/logging/log4j/log4j-slf4j-impl/2.17.1/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
00:10:17.712   2022-01-08 00:44:47.956 - stderr> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
00:10:17.712   2022-01-08 00:44:47.956 - stderr> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
00:10:17.712   2022-01-08 00:44:49.246 - stderr> Exception in thread "Executor task launch worker for task 0.0 in stage 0.0 (TID 0)" java.lang.NoClassDefFoundError: Could not initialize class org.slf4j.MDC
00:10:17.712   2022-01-08 00:44:49.246 - stderr> 	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:751)
00:10:17.712   2022-01-08 00:44:49.246 - stderr> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:441)
00:10:17.712   2022-01-08 00:44:49.246 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
00:10:17.712   2022-01-08 00:44:49.246 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
00:10:17.712   2022-01-08 00:44:49.246 - stderr> 	at java.base/java.lang.Thread.run(Thread.java:833) (SparkSubmitTestUtils.scala:107)
```

```
[INFO] +- org.apache.hadoop:hadoop-minikdc:jar:3.3.1:test
[INFO] |  +- org.apache.kerby:kerb-simplekdc:jar:1.0.1:test
...
[INFO] |  |     \- org.apache.kerby:kerby-xdr:jar:1.0.1:test
[INFO] |  \- org.slf4j:slf4j-log4j12:jar:1.7.30:test
```

**AFTER**
```
$ build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.deploy.SparkSubmitSuite test
...
SparkSubmitSuite:
- prints usage on empty input
- prints usage with only --help
- prints error with unrecognized options
- handle binary specified but not class
- handles arguments with --key=val
- handles arguments to user program
- handles arguments to user program with name collision
- print the right queue name
- SPARK-24241: do not fail fast if executor num is 0 when dynamic allocation is enabled
- specify deploy mode through configuration
- handles YARN cluster mode
- handles YARN client mode
- SPARK-33530: handles standalone mode with archives
- handles standalone cluster mode
- handles legacy standalone cluster mode
- handles standalone client mode
- handles mesos client mode
- handles k8s cluster mode
- automatically sets mainClass if primary resource is S3 JAR in client mode
- automatically sets mainClass if primary resource is S3 JAR in cluster mode
- error informatively when mainClass isn't set and S3 JAR doesn't exist
- handles confs with flag equivalents
- SPARK-21568 ConsoleProgressBar should be enabled only in shells
- launch simple application with spark-submit
- launch simple application with spark-submit with redaction
- includes jars passed in through --jars
- includes jars passed through spark.jars.packages and spark.jars.repositories
- correctly builds R packages included in a jar with --packages !!! IGNORED !!!
- include an external JAR in SparkR !!! CANCELED !!!
  org.apache.spark.api.r.RUtils.isSparkRInstalled was false SparkR is not installed in this build. (SparkSubmitSuite.scala:740)
- resolves command line argument paths correctly
- ambiguous archive mapping results in error message
- resolves config paths correctly
- user classpath first in driver
- SPARK_CONF_DIR overrides spark-defaults.conf
- support glob path
- SPARK-27575: yarn confs should merge new value with existing value
- downloadFile - invalid url
- downloadFile - file doesn't exist
- downloadFile does not download local file
- download one file to local
- download list of files to local
- remove copies of application jar from classpath
- Avoid re-upload remote resources in yarn client mode
- download remote resource if it is not supported by yarn service
- avoid downloading remote resource if it is supported by yarn service
- force download from forced schemes
- force download for all the schemes
- SPARK-32119: Jars and files should be loaded when Executors launch for plugins
- start SparkApplication without modifying system properties
- support --py-files/spark.submit.pyFiles in non pyspark application
- handles natural line delimiters in --properties-file and --conf uniformly
- get a Spark configuration from arguments
Run completed in 25 seconds, 216 milliseconds.
Total number of tests run: 51
Suites: completed 2, aborted 0
Tests: succeeded 51, failed 0, canceled 1, ignored 1, pending 0
All tests passed.
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the test with Maven.

Closes apache#35143 from dongjoon-hyun/SPARK-37844.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit ef5d1f9)
  • Loading branch information
dongjoon-hyun authored and asiunov committed Aug 25, 2022
1 parent 8a75087 commit 2b53114
Showing 1 changed file with 4 additions and 0 deletions.
4 changes: 4 additions & 0 deletions pom.xml
Expand Up @@ -1262,6 +1262,10 @@
<groupId>log4j</groupId>
<artifactId>log4j</artifactId>
</exclusion>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down

0 comments on commit 2b53114

Please sign in to comment.