Skip to content

Commit

Permalink
[SPARK-37843][CORE] Suppress NoSuchFieldError at setMDCForTask
Browse files Browse the repository at this point in the history
### What changes were proposed in this pull request?

This PR aims to suppress `NoSuchFieldError` at `setMDCForTask`.

### Why are the changes needed?

This is observed on `master` branch, Java 17, Apple Silicon combination.
```
$ build/mvn package -Dtest.exclude.tags=org.apache.spark.tags.ExtendedLevelDBTest,org.apache.spark.tags.ExtendedRocksDBTest
```

```
00:57:11 2022-01-07 15:57:11.693 - stderr> Exception in thread "Executor task launch worker-0" java.lang.NoSuchFieldError: mdc
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.log4j.MDCFriend.fixForJava9(MDCFriend.java:11)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.impl.Log4jMDCAdapter.<clinit>(Log4jMDCAdapter.java:38)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.impl.StaticMDCBinder.getMDCA(StaticMDCBinder.java:59)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.MDC.bwCompatibleGetMDCAdapterFromBinder(MDC.java:99)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.slf4j.MDC.<clinit>(MDC.java:108)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.spark.executor.Executor.org$apache$spark$executor$Executor$$setMDCForTask(Executor.scala:750)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:441)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
00:57:11 2022-01-07 15:57:11.693 - stderr> 	at java.base/java.lang.Thread.run(Thread.java:833)
```

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs.

Closes apache#35141 from dongjoon-hyun/SPARK-37843.

Authored-by: Dongjoon Hyun <dongjoon@apache.org>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
(cherry picked from commit f051b4b)
  • Loading branch information
dongjoon-hyun authored and asiunov committed Aug 25, 2022
1 parent 9d201bf commit 8a75087
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions core/src/main/scala/org/apache/spark/executor/Executor.scala
Original file line number Diff line number Diff line change
Expand Up @@ -745,11 +745,15 @@ private[spark] class Executor(
}

private def setMDCForTask(taskName: String, mdc: Seq[(String, String)]): Unit = {
// make sure we run the task with the user-specified mdc properties only
MDC.clear()
mdc.foreach { case (key, value) => MDC.put(key, value) }
// avoid overriding the takName by the user
MDC.put("mdc.taskName", taskName)
try {
// make sure we run the task with the user-specified mdc properties only
MDC.clear()
mdc.foreach { case (key, value) => MDC.put(key, value) }
// avoid overriding the takName by the user
MDC.put("mdc.taskName", taskName)
} catch {
case _: NoSuchFieldError => logInfo("MDC is not supported.")
}
}

/**
Expand Down

0 comments on commit 8a75087

Please sign in to comment.