[SPARK-24398] [SQL] Improve SQLAppStatusListener.aggregateMetrics() too slow #21438
Conversation
|
please see http://spark.apache.org/contributing.html on "Pull Request" also fix the PR title to start with |
frb502
changed the title
frb502
changed the title
|
Please update the PR description. |
|
|
||
| private def aggregateMetrics(exec: LiveExecutionData): Map[Long, String] = { | ||
| val metricIds = exec.metrics.map(_.accumulatorId).sorted | ||
| val metricIds = exec.metrics.map(_.accumulatorId).toSet |
There was a problem hiding this comment.
I think we can get rid of metricIds
There was a problem hiding this comment.
+1， we can use metricTypes
|
ok to test |
|
Test build #91234 has finished for PR 21438 at commit
|
frb502
changed the title
|
ok to test |
|
I think filtering off |
|
Test build #91641 has finished for PR 21438 at commit
|
|
@felixcheung |
|
Just following up on this old small PR @gatorsmile @frb502 -- was there a different solution here that makes sense? should this be closed? |
6 participants
What changes were proposed in this pull request?
JIRA Issue: https://issues.apache.org/jira/browse/SPARK-24398
How was this patch tested?
(Please explain how this patch was tested. E.g. unit tests, integration tests, manual tests)
(If this patch involves UI changes, please attach a screenshot; otherwise, remove this)
Please review http://spark.apache.org/contributing.html before opening a pull request.