| title | tags | metaDescription | redirects | freshnessValidatedDate | |||||
|---|---|---|---|---|---|---|---|---|---|
AWS Glue monitoring integration |
|
New Relic's AWS Glue monitoring integration: what data it reports, and how to enable it. |
|
never |
New Relic offers an integration for reporting your AWS Glue data. This document explains how to activate this integration and describes the data that can be reported.
To enable this integration follow standard procedures to Connect AWS services to New Relic.
You can change the polling frequency and filter data using configuration options.
Default polling information for the AWS Glue integration:
- New Relic polling interval: 5 minutes
- Amazon CloudWatch data interval: 1 minute
To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > AWS and select an integration.
You can query and explore your data using the following event type:
<th>
Event Type
</th>
<th>
Provider
</th>
</tr>
<td>
`AwsGlueJobSample`
</td>
<td>
`AwsGlueJob`
</td>
</tr>
| Entity |
|---|
| Job |
For more on how to use your data, see Understand and use integration data.
This integration collects AWS Glue data for Job.
<th style={{ width: "150px" }}>
Unit
</th>
<th>
Description
</th>
</tr>
<td>
Bytes
</td>
<td>
The number of bytes read from all data sources by all completed Spark tasks running in all executors.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.elapsedTime`
</td>
<td>
Milliseconds
</td>
<td>
The ETL elapsed time in milliseconds (does not include the job bootstrap times).
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.numCompletedStages`
</td>
<td>
Count
</td>
<td>
The number of completed stages in the job.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.numCompletedTasks`
</td>
<td>
Count
</td>
<td>
The number of completed tasks in the job.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.numFailedTasks`
</td>
<td>
Count
</td>
<td>
The number of failed tasks.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.numKilledTasks`
</td>
<td>
Count
</td>
<td>
The number of tasks killed.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.recordsRead`
</td>
<td>
Count
</td>
<td>
The number of records read from all data sources by all completed Spark tasks running in all executors.
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.shuffleBytesWritten`
</td>
<td>
Bytes
</td>
<td>
The number of bytes written by all executors to shuffle data between them since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes written for this purpose during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.driver.aggregate.shuffleLocalBytesRead`
</td>
<td>
Bytes
</td>
<td>
The number of bytes read by all executors to shuffle data between them since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes read for this purpose during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.driver.BlockManager.disk.diskSpaceUsed_MB`
</td>
<td>
Bytes
</td>
<td>
The number of megabytes of disk space used across all executors.
</td>
</tr>
<tr>
<td>
`glue.driver.ExecutorAllocationManager.executors.numberAllExecutors`
</td>
<td>
Count
</td>
<td>
The number of actively running job executors.
</td>
</tr>
<tr>
<td>
`glue.driver.ExecutorAllocationManager.executors.numberMaxNeededExecutors`
</td>
<td>
Count
</td>
<td>
The number of maximum (actively running and pending) job executors needed to satisfy the current load.
</td>
</tr>
<tr>
<td>
`glue.driver.jvm.heap.usage`
</td>
<td>
Percent
</td>
<td>
The fraction of memory used by the JVM heap for this driver (scale: 0-1) for driver.
</td>
</tr>
<tr>
<td>
`glue.ALL.jvm.heap.usage`
</td>
<td>
Percent
</td>
<td>
The fraction of memory used by the JVM heap for this driver (scale: 0-1) for ALL executors.
</td>
</tr>
<tr>
<td>
`glue.driver.jvm.heap.used`
</td>
<td>
Bytes
</td>
<td>
The number of memory bytes used by the JVM heap for the driver.
</td>
</tr>
<tr>
<td>
`glue.ALL.jvm.heap.used`
</td>
<td>
Bytes
</td>
<td>
The number of memory bytes used by the JVM heap for ALL executors.
</td>
</tr>
<tr>
<td>
`glue.driver.s3.filesystem.read_bytes`
</td>
<td>
Bytes
</td>
<td>
The number of bytes read from Amazon S3 by the driver since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes read during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.ALL.s3.filesystem.read_bytes`
</td>
<td>
Bytes
</td>
<td>
The number of bytes read from Amazon S3 by ALL executors since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes read during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.driver.s3.filesystem.write_bytes`
</td>
<td>
Bytes
</td>
<td>
The number of bytes written to Amazon S3 by the driver since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes written during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.ALL.s3.filesystem.write_bytes`
</td>
<td>
Bytes
</td>
<td>
The number of bytes written to Amazon S3 by ALL executors since the previous report (aggregated by the AWS Glue Metrics Dashboard as the number of bytes written during the previous minute).
</td>
</tr>
<tr>
<td>
`glue.driver.system.cpuSystemLoad`
</td>
<td>
Percent
</td>
<td>
The fraction of CPU system load used (scale: 0-1) by the driver.
</td>
</tr>
<tr>
<td>
`glue.ALL.system.cpuSystemLoad`
</td>
<td>
Percent
</td>
<td>
The fraction of CPU system load used (scale: 0-1) by the ALL executors.
</td>
</tr>
| Metric |
|---|
| `glue.driver.aggregate.bytesRead` |