Skip to content

Commit

Permalink
[SPARK-22052] Incorrect Metric assigned in MetricsReporter.scala
Browse files Browse the repository at this point in the history
Current implementation for processingRate-total uses wrong metric:
mistakenly uses inputRowsPerSecond instead of processedRowsPerSecond

## What changes were proposed in this pull request?
Adjust processingRate-total from using inputRowsPerSecond to processedRowsPerSecond

## How was this patch tested?

Built spark from source with proposed change and tested output with correct parameter. Before change the csv metrics file for inputRate-total and processingRate-total displayed the same values due to the error. After changing MetricsReporter.scala the processingRate-total csv file displayed the correct metric.
<img width="963" alt="processed rows per second" src="https://user-images.githubusercontent.com/32072374/30554340-82eea12c-9ca4-11e7-8370-8168526ff9a2.png">

Please review http://spark.apache.org/contributing.html before opening a pull request.

Author: Taaffy <32072374+Taaffy@users.noreply.github.com>

Closes #19268 from Taaffy/patch-1.

(cherry picked from commit 1bc17a6)
Signed-off-by: Sean Owen <sowen@cloudera.com>
  • Loading branch information
Taaffy authored and srowen committed Sep 19, 2017
1 parent b35136a commit 30ce056
Showing 1 changed file with 1 addition and 1 deletion.
Expand Up @@ -40,7 +40,7 @@ class MetricsReporter(
// Metric names should not have . in them, so that all the metrics of a query are identified
// together in Ganglia as a single metric group
registerGauge("inputRate-total", () => stream.lastProgress.inputRowsPerSecond)
registerGauge("processingRate-total", () => stream.lastProgress.inputRowsPerSecond)
registerGauge("processingRate-total", () => stream.lastProgress.processedRowsPerSecond)
registerGauge("latency", () => stream.lastProgress.durationMs.get("triggerExecution").longValue())

private def registerGauge[T](name: String, f: () => T)(implicit num: Numeric[T]): Unit = {
Expand Down

0 comments on commit 30ce056

Please sign in to comment.