[SPARK-17649][CORE] Log how many Spark events got dropped in AsynchronousListenerBus (branch 1.6) #15226
Conversation
…enerBus Log how many Spark events got dropped in LiveListenerBus so that the user can get insights on how to set a correct event queue size. Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #15220 from zsxwing/SPARK-17649.
| val droppedEvents = droppedEventsCounter.get | ||
| if (droppedEvents > 0) { | ||
| // Don't log too frequently | ||
| if (System.currentTimeMillis() - lastReportTimestamp >= 60 * 1000) { |
There was a problem hiding this comment.
Won't nanotime be overkill ? Even if there is a single dropped event, this check will get executed with every post() so having currentTimeMillis (which is less costly) is preferable.
There was a problem hiding this comment.
@rxin this is not for measuring accurate elapsed time and we also want to log it using Date. According to the javadoc:
This method can only be used to measure elapsed time and is not related to any other notion of system or wall-clock time. The value returned represents nanoseconds since some fixed but arbitrary origin time (perhaps in the future, so values may be negative). The same origin is used by all invocations of this method in an instance of a Java virtual machine; other virtual machine instances are likely to use a different origin.
It's better to use System.currentTimeMillis() since we want to report the timestamp in the log.
|
Test build #65855 has finished for PR 15226 at commit
|
zsxwing
changed the title
|
Merging to 1.6. |
…nousListenerBus (branch 1.6) ## What changes were proposed in this pull request? Backport #15220 to 1.6. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes #15226 from zsxwing/SPARK-17649-branch-1.6.
…nousListenerBus (branch 1.6) ## What changes were proposed in this pull request? Backport apache#15220 to 1.6. ## How was this patch tested? Jenkins Author: Shixiong Zhu <shixiong@databricks.com> Closes apache#15226 from zsxwing/SPARK-17649-branch-1.6. (cherry picked from commit 7aded55)
What changes were proposed in this pull request?
Backport #15220 to 1.6.
How was this patch tested?
Jenkins