diff --git a/docs/monitoring.md b/docs/monitoring.md index c30aa99679399..bfa0d3afb3497 100644 --- a/docs/monitoring.md +++ b/docs/monitoring.md @@ -658,31 +658,75 @@ A list of the available metrics, with a short description: ### Executor Metrics Executor-level metrics are sent from each executor to the driver as part of the Heartbeat to describe the performance metrics of Executor itself like JVM heap memory, GC information. -Executor metric values and their measured peak values per executor are exposed via the REST API at the end point `/applications/[app-id]/executors`. -In addition, aggregated per-stage peak values of the executor metrics are written to the event log if `spark.eventLog.logStageExecutorMetrics` is true. -Executor metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. +Executor metric values and their measured memory peak values per executor are exposed via the REST API in JSON format and in Prometheus format. +The JSON end point is exposed at: `/applications/[app-id]/executors`, and the Prometheus endpoint at: `/metrics/executors/prometheus`. +The Prometheus endpoint is conditional to a configuration parameter: `spark.ui.prometheus.enabled=true` (the default is `false`). +In addition, aggregated per-stage peak values of the executor memory metrics are written to the event log if +`spark.eventLog.logStageExecutorMetrics` is true. +Executor memory metrics are also exposed via the Spark metrics system based on the Dropwizard metrics library. A list of the available metrics, with a short description: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - - + - + - +
Executor Level Metric name Short description
rddBlocksRDD blocks in the block manager of this executor.
memoryUsedStorage memory used by this executor.
diskUsedDisk space used for RDD storage by this executor.
totalCoresNumber of cores available in this executor.
maxTasksMaximum number of tasks that can run concurrently in this executor.
activeTasksNumber of tasks currently executing.
failedTasksNumber of tasks that have failed in this executor.
completedTasksNumber of tasks that have completed in this executor.
totalTasksTotal number of tasks (running, failed and completed) in this executor.
totalDurationElapsed time the JVM spent executing tasks in this executor. + The value is expressed in milliseconds.
totalGCTimeElapsed time the JVM spent in garbage collection summed in this Executor. + Elapsed time the JVM spent in garbage collection summed in this executor. The value is expressed in milliseconds.
totalInputBytesTotal input bytes summed in this Executor.Total input bytes summed in this executor.
totalShuffleReadTotal shuffer read bytes summed in this Executor.Total shuffle read bytes summed in this executor.
totalShuffleWriteTotal shuffer write bytes summed in this Executor.Total shuffle write bytes summed in this executor.
maxMemory