Skip to content

Commit

Permalink
Add note on metrics to the README
Browse files Browse the repository at this point in the history
  • Loading branch information
LucaCanali committed Mar 6, 2024
1 parent 4a6f525 commit fa58795
Show file tree
Hide file tree
Showing 2 changed files with 19 additions and 1 deletion.
17 changes: 17 additions & 0 deletions README.md
Expand Up @@ -31,6 +31,7 @@ and spark-shell/pyspark environments.
### Contents
- [Getting started with sparkMeasure](#getting-started-with-sparkmeasure)
- [Documentation and API reference](#documentation-api-and-examples)
- [Notes on Metrics](#notes-on-metrics)
- [Architecture diagram](#architecture-diagram)
- [Concepts and FAQ](#main-concepts-underlying-sparkmeasure-implementation)

Expand Down Expand Up @@ -189,6 +190,22 @@ Stage 3 OnHeapExecutionMemory maxVal bytes => 0 (0 Bytes)
taskmetrics = TaskMetrics(spark)
taskmetrics.runandmeasure(globals(), 'spark.sql("select count(*) from range(1000) cross join range(1000) cross join range(1000)").show()')
```
---
### Notes on Metrics
Spark is instrumented with several metrics, collected at task execution, they are described in the documentation:
- [Spark Task Metrics docs](https://spark.apache.org/docs/latest/monitoring.html#executor-task-metrics)

Some of the key metrics when looking at a sparkMeasure report are:
- **elapsedTime:** the time taken by the stage or task to complete (in millisec)
- **executorRunTime:** the time the executors spent running the task, (in millisec). Note this time is cumulative across all tasks executed by the executor.
- **executorCpuTime:** the time the executors spent running the task, (in millisec). Note this time is cumulative across all tasks executed by the executor.
- **jvmGCTime:** the time the executors spent in garbage collection, (in millisec).
- shuffle metrics: several metrics with details on the I/O and time spend on shuffle
- I/O metrics: details on the I/O (reads and writes). Note, currently there are no time-based metrics for I/O operations.

To learn more about hte metrics, I advise you set up your lab environment and run some tests to see the metrics in action.
A good place to start with is [TPCDS PySpark](https://github.com/LucaCanali/Miscellaneous/tree/master/Performance_Testing/TPCDS_PySpark) - A tool you can use run TPCDS with PySpark, instrumented with sparkMeasure

---
### Documentation, API, and examples
SparkMeasure is one tool for many different use cases, languages, and environments:
Expand Down
3 changes: 2 additions & 1 deletion docs/Reference_SparkMeasure_API_and_Configs.md
Expand Up @@ -13,6 +13,7 @@ Contents:
- [Flight Recorder Mode - File Sink](#flight-recorder-mode---file-sink)
- [InfluxDBSink and InfluxDBSinkExtended](#influxdbsink-and-influxdbsinkextended)
- [KafkaSink and KafkaSinkExtended](#kafkasink-and-kafkasinkextended)
- [Prometheus PushGatewaySink](#prometheus-pushgatewaysink)
- [IOUtils](#ioutils)
- [Utils](#utils)

Expand Down Expand Up @@ -432,7 +433,7 @@ This code depends on "kafka-clients", you may need to add the dependency explici
--packages org.apache.kafka:kafka-clients:3.7.0
```

## PushGatewaySink
## Prometheus PushGatewaySink
```
class PushGatewaySink(conf: SparkConf) extends SparkListener
Expand Down

0 comments on commit fa58795

Please sign in to comment.