[AWS] Add metric type to SNS, SQS and Billing data streams (#6319)

* Add metric type to SNS. Signed-off-by: constanca-m <constanca.manteigas@elastic.co> * Add metric type to SQS. Signed-off-by: constanca-m <constanca.manteigas@elastic.co> * Add metric type to Billing. Signed-off-by: constanca-m <constanca.manteigas@elastic.co> * Update changelog. Signed-off-by: constanca-m <constanca.manteigas@elastic.co> * Update files. Signed-off-by: constanca-m <constanca.manteigas@elastic.co> --------- Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
elastic · Jun 15, 2023 · 4dee550 · 4dee550
1 parent 854a638
commit 4dee550
Show file tree

Hide file tree

Showing 8 changed files with 227 additions and 196 deletions.
diff --git a/packages/aws/changelog.yml b/packages/aws/changelog.yml
@@ -1,4 +1,9 @@
 # newer versions go on top
+- version: "1.40.0"
+  changes:
+    - description: Add metric type to SNS, SQS and Billing data streams.
+      type: enhancement
+      link: https://github.com/elastic/integrations/pull/6319
 - version: "1.39.0"
   changes:
     - description: Add AWS API Gateway data stream.

diff --git a/packages/aws/data_stream/billing/fields/fields.yml b/packages/aws/data_stream/billing/fields/fields.yml
@@ -6,6 +6,7 @@
       fields:
         - name: EstimatedCharges
           type: long
+          metric_type: gauge
           description: Maximum estimated charges for AWS acccount.
         - name: Currency
           type: keyword
@@ -19,6 +20,7 @@
           fields:
             - name: amount
               type: double
+              metric_type: gauge
               description: Amortized cost amount.
             - name: unit
               type: keyword
@@ -28,6 +30,7 @@
           fields:
             - name: amount
               type: double
+              metric_type: gauge
               description: Blended cost amount.
             - name: unit
               type: keyword
@@ -37,6 +40,7 @@
           fields:
             - name: amount
               type: double
+              metric_type: gauge
               description: Normalized usage amount.
             - name: unit
               type: keyword
@@ -46,6 +50,7 @@
           fields:
             - name: amount
               type: double
+              metric_type: gauge
               description: Unblended cost amount.
             - name: unit
               type: keyword
@@ -54,6 +59,7 @@
           type: group
           fields:
             - name: amount
+              metric_type: gauge
               type: double
               description: Usage quantity amount.
             - name: unit

diff --git a/packages/aws/data_stream/sns/fields/fields.yml b/packages/aws/data_stream/sns/fields/fields.yml
@@ -32,35 +32,46 @@
           fields:
             - name: PublishSize.avg
               type: double
+              metric_type: gauge
               description: The size of messages published.
             - name: SMSSuccessRate.avg
               type: double
+              metric_type: gauge
               description: The rate of successful SMS message deliveries.
             - name: NumberOfMessagesPublished.sum
               type: long
+              metric_type: gauge
               description: The number of messages published to your Amazon SNS topics.
             - name: NumberOfNotificationsDelivered.sum
               type: long
+              metric_type: gauge
               description: The number of messages successfully delivered from your Amazon SNS topics to subscribing endpoints.
             - name: NumberOfNotificationsFailed.sum
               type: long
+              metric_type: gauge
               description: The number of messages that Amazon SNS failed to deliver.
             - name: NumberOfNotificationsFilteredOut.sum
               type: long
+              metric_type: gauge
               description: The number of messages that were rejected by subscription filter policies.
             - name: NumberOfNotificationsFilteredOut-InvalidAttributes.sum
               type: long
+              metric_type: gauge
               description: The number of messages that were rejected by subscription filter policies because the messages' attributes are invalid - for example, because the attribute JSON is incorrectly formatted.
             - name: NumberOfNotificationsFilteredOut-NoMessageAttributes.sum
               type: long
+              metric_type: gauge
               description: The number of messages that were rejected by subscription filter policies because the messages have no attributes.
             - name: NumberOfNotificationsRedrivenToDlq.sum
               type: long
+              metric_type: gauge
               description: The number of messages that have been moved to a dead-letter queue.
             - name: NumberOfNotificationsFailedToRedriveToDlq.sum
               type: long
+              metric_type: gauge
               description: The number of messages that couldn't be moved to a dead-letter queue.
             - name: SMSMonthToDateSpentUSD.sum
+              metric_type: gauge
               type: long
               description: The charges you have accrued since the start of the current calendar month for sending SMS messages.
     - name: cloudwatch

diff --git a/packages/aws/data_stream/sqs/fields/fields.yml b/packages/aws/data_stream/sqs/fields/fields.yml
@@ -13,39 +13,48 @@
       fields:
         - name: oldest_message_age.sec
           type: long
+          metric_type: gauge
           format: duration
           description: |
             The approximate age of the oldest non-deleted message in the queue.
         - name: messages.delayed
           type: long
+          metric_type: gauge
           description: |
             TThe number of messages in the queue that are delayed and not available for reading immediately.
         - name: messages.not_visible
           type: long
+          metric_type: gauge
           description: |
             The number of messages that are in flight.
         - name: messages.visible
           type: long
+          metric_type: gauge
           description: |
             The number of messages available for retrieval from the queue.
         - name: messages.deleted
           type: long
+          metric_type: gauge
           description: |
             The number of messages deleted from the queue.
         - name: messages.received
           type: long
+          metric_type: gauge
           description: |
             The number of messages returned by calls to the ReceiveMessage action.
         - name: messages.sent
           type: long
+          metric_type: gauge
           description: |
             The number of messages added to a queue.
         - name: empty_receives
           type: long
+          metric_type: gauge
           description: |
             The number of ReceiveMessage API calls that did not return a message.
         - name: sent_message_size.bytes
           type: long
+          metric_type: gauge
           format: bytes
           description: |
             The size of messages added to a queue.

diff --git a/packages/aws/docs/billing.md b/packages/aws/docs/billing.md
@@ -111,74 +111,74 @@ An example event for `billing` looks as following:
 
 **Exported fields**
 
-| Field | Description | Type |
-|---|---|---|
-| @timestamp | Event timestamp. | date |
-| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |
-| aws.\*.metrics.\*.\* | Metrics that returned from Cloudwatch API query. | object |
-| aws.billing.AmortizedCost.amount | Amortized cost amount. | double |
-| aws.billing.AmortizedCost.unit | Amortized cost unit. | keyword |
-| aws.billing.BlendedCost.amount | Blended cost amount. | double |
-| aws.billing.BlendedCost.unit | Blended cost unit. | keyword |
-| aws.billing.Currency | Currency name. | keyword |
-| aws.billing.EstimatedCharges | Maximum estimated charges for AWS acccount. | long |
-| aws.billing.NormalizedUsageAmount.amount | Normalized usage amount. | double |
-| aws.billing.NormalizedUsageAmount.unit | Normalized usage amount unit. | keyword |
-| aws.billing.ServiceName | AWS service name. | keyword |
-| aws.billing.UnblendedCost.amount | Unblended cost amount. | double |
-| aws.billing.UnblendedCost.unit | Unblended cost unit. | keyword |
-| aws.billing.UsageQuantity.amount | Usage quantity amount. | double |
-| aws.billing.UsageQuantity.unit | Usage quantity unit. | keyword |
-| aws.billing.end_date | End date for retrieving AWS costs. | keyword |
-| aws.billing.group_by | Cost explorer group by key values. | object |
-| aws.billing.group_definition.key | The string that represents a key for a specified group. | keyword |
-| aws.billing.group_definition.type | The string that represents the type of group. | keyword |
-| aws.billing.start_date | Start date for retrieving AWS costs. | keyword |
-| aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword |
-| aws.dimensions.\* | Metric dimensions. | object |
-| aws.linked_account.id | ID used to identify linked account. | keyword |
-| aws.linked_account.name | Name or alias used to identify linked account. | keyword |
-| aws.s3.bucket.name | Name of a S3 bucket. | keyword |
-| aws.tags.\* | Tag key value pairs from aws resources. | object |
-| cloud | Fields related to the cloud or infrastructure the events are coming from. | group |
-| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |
-| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword |
-| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword |
-| cloud.image.id | Image ID for the cloud instance. | keyword |
-| cloud.instance.id | Instance ID of the host machine. | keyword |
-| cloud.instance.name | Instance name of the host machine. | keyword |
-| cloud.machine.type | Machine type of the host machine. | keyword |
-| cloud.project.id | The cloud project identifier. Examples: Google Cloud Project id, Azure Project id. | keyword |
-| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |
-| cloud.region | Region in which this host, resource, or service is located. | keyword |
-| container.id | Unique container id. | keyword |
-| container.image.name | Name of the image the container was built on. | keyword |
-| container.labels | Image labels. | object |
-| container.name | Container name. | keyword |
-| data_stream.dataset | Data stream dataset. | constant_keyword |
-| data_stream.namespace | Data stream namespace. | constant_keyword |
-| data_stream.type | Data stream type. | constant_keyword |
-| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
-| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group |
-| error.message | Error message. | match_only_text |
-| event.dataset | Event dataset | constant_keyword |
-| event.module | Event module | constant_keyword |
-| host.architecture | Operating system architecture. | keyword |
-| host.containerized | If the host is a container. | boolean |
-| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |
-| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |
-| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |
-| host.ip | Host ip addresses. | ip |
-| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword |
-| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |
-| host.os.build | OS build information. | keyword |
-| host.os.codename | OS codename, if any. | keyword |
-| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
-| host.os.kernel | Operating system kernel version as a raw string. | keyword |
-| host.os.name | Operating system name, without the version. | keyword |
-| host.os.name.text | Multi-field of `host.os.name`. | match_only_text |
-| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |
-| host.os.version | Operating system version as a raw string. | keyword |
-| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |
-| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |
+| Field | Description | Type | Metric Type |
+|---|---|---|---|
+| @timestamp | Event timestamp. | date |  |
+| agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword |  |
+| aws.\*.metrics.\*.\* | Metrics that returned from Cloudwatch API query. | object |  |
+| aws.billing.AmortizedCost.amount | Amortized cost amount. | double | gauge |
+| aws.billing.AmortizedCost.unit | Amortized cost unit. | keyword |  |
+| aws.billing.BlendedCost.amount | Blended cost amount. | double | gauge |
+| aws.billing.BlendedCost.unit | Blended cost unit. | keyword |  |
+| aws.billing.Currency | Currency name. | keyword |  |
+| aws.billing.EstimatedCharges | Maximum estimated charges for AWS acccount. | long | gauge |
+| aws.billing.NormalizedUsageAmount.amount | Normalized usage amount. | double | gauge |
+| aws.billing.NormalizedUsageAmount.unit | Normalized usage amount unit. | keyword |  |
+| aws.billing.ServiceName | AWS service name. | keyword |  |
+| aws.billing.UnblendedCost.amount | Unblended cost amount. | double | gauge |
+| aws.billing.UnblendedCost.unit | Unblended cost unit. | keyword |  |
+| aws.billing.UsageQuantity.amount | Usage quantity amount. | double | gauge |
+| aws.billing.UsageQuantity.unit | Usage quantity unit. | keyword |  |
+| aws.billing.end_date | End date for retrieving AWS costs. | keyword |  |
+| aws.billing.group_by | Cost explorer group by key values. | object |  |
+| aws.billing.group_definition.key | The string that represents a key for a specified group. | keyword |  |
+| aws.billing.group_definition.type | The string that represents the type of group. | keyword |  |
+| aws.billing.start_date | Start date for retrieving AWS costs. | keyword |  |
+| aws.cloudwatch.namespace | The namespace specified when query cloudwatch api. | keyword |  |
+| aws.dimensions.\* | Metric dimensions. | object |  |
+| aws.linked_account.id | ID used to identify linked account. | keyword |  |
+| aws.linked_account.name | Name or alias used to identify linked account. | keyword |  |
+| aws.s3.bucket.name | Name of a S3 bucket. | keyword |  |
+| aws.tags.\* | Tag key value pairs from aws resources. | object |  |
+| cloud | Fields related to the cloud or infrastructure the events are coming from. | group |  |
+| cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword |  |
+| cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword |  |
+| cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword |  |
+| cloud.image.id | Image ID for the cloud instance. | keyword |  |
+| cloud.instance.id | Instance ID of the host machine. | keyword |  |
+| cloud.instance.name | Instance name of the host machine. | keyword |  |
+| cloud.machine.type | Machine type of the host machine. | keyword |  |
+| cloud.project.id | The cloud project identifier. Examples: Google Cloud Project id, Azure Project id. | keyword |  |
+| cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword |  |
+| cloud.region | Region in which this host, resource, or service is located. | keyword |  |
+| container.id | Unique container id. | keyword |  |
+| container.image.name | Name of the image the container was built on. | keyword |  |
+| container.labels | Image labels. | object |  |
+| container.name | Container name. | keyword |  |
+| data_stream.dataset | Data stream dataset. | constant_keyword |  |
+| data_stream.namespace | Data stream namespace. | constant_keyword |  |
+| data_stream.type | Data stream type. | constant_keyword |  |
+| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |  |
+| error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group |  |
+| error.message | Error message. | match_only_text |  |
+| event.dataset | Event dataset | constant_keyword |  |
+| event.module | Event module | constant_keyword |  |
+| host.architecture | Operating system architecture. | keyword |  |
+| host.containerized | If the host is a container. | boolean |  |
+| host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword |  |
+| host.hostname | Hostname of the host. It normally contains what the `hostname` command returns on the host machine. | keyword |  |
+| host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of `beat.name`. | keyword |  |
+| host.ip | Host ip addresses. | ip |  |
+| host.mac | Host MAC addresses. The notation format from RFC 7042 is suggested: Each octet (that is, 8-bit byte) is represented by two [uppercase] hexadecimal digits giving the value of the octet as an unsigned integer. Successive octets are separated by a hyphen. | keyword |  |
+| host.name | Name of the host. It can contain what `hostname` returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword |  |
+| host.os.build | OS build information. | keyword |  |
+| host.os.codename | OS codename, if any. | keyword |  |
+| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |  |
+| host.os.kernel | Operating system kernel version as a raw string. | keyword |  |
+| host.os.name | Operating system name, without the version. | keyword |  |
+| host.os.name.text | Multi-field of `host.os.name`. | match_only_text |  |
+| host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword |  |
+| host.os.version | Operating system version as a raw string. | keyword |  |
+| host.type | Type of host. For Cloud providers this can be the machine type like `t2.medium`. If vm, this could be the container, for example, or other information meaningful in your environment. | keyword |  |
+| service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, `service.type` would be `elasticsearch`. | keyword |  |