Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AWS] Set dimensions fields of Billing, SQS and SNS data streams #6311

Merged

Conversation

constanca-m
Copy link
Contributor

@constanca-m constanca-m commented May 24, 2023

What does this PR do?

Set dimensions fields of Billing, SQS and SNS data streams.

Billing:

  • The billing is split by the service for each AWS account. Region does not seem to take influence on this, as the value is the sum of all of them per AWS account. For that reason, cloud.account.id and ServiceName1 work as a unique combination.

SQS:

  • From AWS: "When you create a new queue, you must specify a queue name unique for your AWS account and region.", making the queue name unique per account and region.
  • There are two fields that hold the same value: aws.dimensions.QueueName and aws.sqs.queue.name. From my understanding, when one of them is available, the other is as well (they come from "cloudwatch:GetMetricData"), so only one of them was set as dimension (aws.dimensions.QueueName).

SNS:

  • Topic name is unique within AWS account.
  • SMSType can hold different values for the same timestamp, so it is set as dimension. It comes in combination with country, so that is a dimension as well.
  • An application has a Platform, and this combination should be unique. Both are set as dimension.

Regarding ECS fields, more info can be found here #5193 (comment).

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • No overwriting of documents when TSDB is enabled.

How to test this PR locally

Refer to #6293.

Related issues

Relates to #6293.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
@elasticmachine
Copy link

elasticmachine commented May 24, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-06-05T13:25:05.216+0000

  • Duration: 47 min 20 sec

Test stats 🧪

Test Results
Failed 0
Passed 188
Skipped 4
Total 192

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented May 24, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (15/15) 💚
Files 93.75% (15/16) 👎 -2.404
Classes 93.75% (15/16) 👎 -2.404
Methods 86.131% (236/274) 👎 -1.636
Lines 85.925% (7387/8597) 👎 -0.409
Conditionals 100.0% (0/0) 💚

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@@ -1,4 +1,9 @@
# newer versions go on top
- version: "1.36.9"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: in #6365 we bumped minor version, here only patch one
why the difference? :)

@constanca-m constanca-m merged commit d50a1f2 into elastic:main Jun 5, 2023
3 checks passed
@constanca-m constanca-m deleted the aws-add-dimensions-sqs-sns-billing branch June 5, 2023 14:20
@elasticmachine
Copy link

Package aws - 1.38.4 containing this change is available at https://epr.elastic.co/search?package=aws

sodhikirti07 pushed a commit that referenced this pull request Jun 15, 2023
* Add SQS dimensions.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Add SNS dimensions.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Add SNS dimensions.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Add Billing dimensions.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update changelog.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Add 2 ecs dimension fields.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Remove cloud.availability_zone as dimension (not present)

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

* Update md.

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>

---------

Signed-off-by: constanca-m <constanca.manteigas@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants