diff --git a/docs/monitoring.md b/docs/monitoring.md index ba3f1dc86becc..131cd2a844e44 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -689,31 +689,75 @@ A list of the available metrics, with a short description: ### Executor Metrics Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information. -Executor metric values and their measured peak values per executor are exposed via the REST API at the end point `/applications/[app-id]/executors`. -In addition, aggregated per-stage peak values of the executor metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true. -Executor metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. +Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format. +The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`. +The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). +In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if +`spark.eventLog.logStageExecutorMetrics` is true. +Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. A list of the available metrics, with a short description:
Executor Level Metric name | Short description | |
---|---|---|
rddBlocks | +RDD blocks in the block manager of this executor. | +|
memoryUsed | +Storage memory used by this executor. | +|
diskUsed | +Disk space used for RDD storage by this executor. | +|
totalCores | +Number of cores available in this executor. | +|
maxTasks | +Maximum number of tasks that can run concurrently in this executor. | +|
activeTasks | +Number of tasks currently executing. | +|
failedTasks | +Number of tasks that have failed in this executor. | +|
completedTasks | +Number of tasks that have completed in this executor. | +|
totalTasks | +Total number of tasks (running, failed and completed) in this executor. | +|
totalDuration | +Elapsed time the JVM spent executing tasks in this executor. + The value is expressed in milliseconds. | +|
totalGCTime | -Elapsed time the JVM spent in garbage collection summed in this Executor. + | Elapsed time the JVM spent in garbage collection summed in this executor. The value is expressed in milliseconds. |
totalInputBytes | -Total input bytes summed in this Executor. | +Total input bytes summed in this executor. |
totalShuffleRead | -Total shuffer read bytes summed in this Executor. | +Total shuffle read bytes summed in this executor. |
totalShuffleWrite | -Total shuffer write bytes summed in this Executor. | +Total shuffle write bytes summed in this executor. |
maxMemory |