[AWS] Collect metrics from linked accounts #35540

kaiyan-sheng · 2023-05-22T23:18:12Z

What does this PR do?

This PR adds support for collecting metrics from linked accounts in AWS. IncludeLinkedAccounts is a new parameter offered in ListMetrics CloudWatch API for users to include metrics from source accounts when this API is called in a monitoring account. With this change, we also update the cloud.account.id to the account that the instance belongs to, either the monitoring account or one of the linked accounts.

Also I added dynamic label collection in this PR as well to collect the actual cloud.account.name for each metric. Dynamic label is supported as a part of the get-metric-data API call for CloudWatch. With adding ${PROP('AccountLabel')} as a part of the label for collecting cloudwatch metrics, we can get the label specified for the source account that owns the metric.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

Use cases

Config 1: Collect metrics from a given namespace.

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  credential_profile_name: cross-account
  regions:
    - ap-northeast-1
  metrics:
    - namespace: AWS/EC2

Expected behavior: We should collect metrics from all instances both from the monitoring account and linked accounts.

Config 2: Collect a specific metric from a given instance.

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  credential_profile_name: cross-account
  regions:
    - ap-northeast-1
  metrics:
    - namespace: AWS/EC2
      name: [ "CPUUtilization" ]
      dimensions:
        - name: InstanceId
          value: i-041ee56b355341288
      statistic: [ "Average" ]

Expected behavior: We should find the instance from one of the linked accounts and collect the metric.

Config3: collect a specific metric from all instances both from the linked accounts and the monitoring account.

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  credential_profile_name: cross-account
  regions:
    - ap-northeast-1
  metrics:
    - namespace: AWS/EC2
      name: [ "CPUUtilization" ]
      statistic: [ "Average" ]

Expected behavior: We should see CPUUtilization metric for all instances from both the linked accounts and the monitoring acco

unt.

Sample Output

{
  "_index": ".ds-metricbeat-8.9.0-2023.05.24-000001",
  "_id": "MdCkT4gBdTd3f6iY6Eme",
  "_version": 1,
  "_score": 0,
  "_source": {
    "@timestamp": "2023-05-24T21:00:00.000Z",
    "metricset": {
      "name": "cloudwatch",
      "period": 300000
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "region": "ap-northeast-1",
      "account": {
        "name": "Sxxxxxa",
        "id": "80xxxxxxxx66"
      },
      "provider": "aws"
    },
    "ecs": {
      "version": "8.0.0"
    },
    "host": {
      "hostname": "Kaiyans-MacBook-Pro.local",
      "architecture": "x86_64",
      "name": "Kaiyans-MacBook-Pro.local"
    },
    "aws": {
      "cloudwatch": {
        "namespace": "AWS/EC2"
      },
      "dimensions": {
        "InstanceId": "i-087df6035a52dc9fe"
      },
      "ec2": {
        "metrics": {
          "CPUUtilization": {
            "avg": 0.23333333333332956
          }
        }
      }
    },
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 1811546083
    }
  }
}

mergify · 2023-05-22T23:18:47Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @kaiyan-sheng? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

elasticmachine · 2023-05-23T02:20:04Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-05-25T19:42:29.235+0000
Duration: 144 min 56 sec

Test stats 🧪

Test	Results
Failed	0
Passed	26132
Skipped	1964
Total	28096

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

…unts

tommyers-elastic

looking good kaiyan - thanks for the quick turnaround on this. couple of small questions. no blockers.

x-pack/metricbeat/module/aws/billing/billing.go

tommyers-elastic · 2023-05-25T07:29:35Z

x-pack/metricbeat/module/aws/billing/billing.go

@@ -195,11 +199,16 @@ func (m *MetricSet) getCloudWatchBillingMetrics(
 		}
 		for valI, metricDataResultValue := range output.Values {
 			labels := strings.Split(*output.Label, labelSeparator)
+			event := mb.Event{}
+			if labels[accountIdIdx] != "" {


under what circumstances is this empty when we pass includeLinkedAccounts?

tommyers-elastic · 2023-05-25T07:41:50Z

x-pack/metricbeat/module/aws/billing/billing.go


-	label := metricName + labelSeparator
+	label := metric.AccountID + labelSeparator + "${PROP('AccountLabel')}" + labelSeparator + metricName + labelSeparator


nit not related to this PR - it would be better if we could avoid the manual work of keeping this label in sync with the index constants used to parse it. one way would just be to wrap the label creation and parsing in a type; at least that way the indexing concerns don't bleed out into the code. let's not change it now, just a thought though.

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go

x-pack/metricbeat/module/aws/utils_test.go

girodav · 2023-05-25T12:43:19Z

x-pack/metricbeat/module/aws/billing/billing.go

@@ -338,7 +347,7 @@ func (m *MetricSet) addCostMetrics(metrics map[string]costexplorertypes.MetricVa
 	return event
 }

-func constructMetricQueries(listMetricsOutput []types.Metric, dataGranularity time.Duration) []types.MetricDataQuery {
+func constructMetricQueries(listMetricsOutput []aws.MetricWithID, dataGranularity time.Duration) []types.MetricDataQuery {


Nit, not required for this PR

We should rename constructMetricQueries and createMetricDataQuery with something clearer (e.g createQueries vs createSingleQuery)

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go

girodav

Thanks for this PR Kaiyan! I put some comments but none of the are deal breakers.

cmacknz

Adding codeowner approval for the go.mod changes. Did not look closely at the other parts of the PR.

kaiyan-sheng · 2023-05-25T20:28:16Z

Thank you all for the review! I made another commit for some of the comments, and I will address the rest in a separate PR.

(cherry picked from commit 217e658)

(cherry picked from commit 217e658) # Conflicts: # NOTICE.txt # go.mod # go.sum # metricbeat/docs/modules/aws.asciidoc # x-pack/metricbeat/module/aws/_meta/docs.asciidoc # x-pack/metricbeat/module/aws/billing/billing.go # x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go # x-pack/metricbeat/module/aws/cloudwatch/cloudwatch_test.go # x-pack/metricbeat/module/aws/utils.go # x-pack/metricbeat/module/aws/utils_test.go

(cherry picked from commit 217e658) Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

(cherry picked from commit 217e658)

…5587) * [AWS] Collect metrics from linked accounts (#35540) (cherry picked from commit 217e658) Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

[AWS] Collect metrics from linked accounts

e02cffc

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 22, 2023

mergify bot assigned kaiyan-sheng May 22, 2023

kaiyan-sheng added 3 commits May 23, 2023 15:53

remove config parameter

6491654

Merge remote-tracking branch 'upstream/main' into include_linked_acco…

cc72bbc

…unts

Add changelog

2018920

kaiyan-sheng added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label May 24, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 24, 2023

kaiyan-sheng marked this pull request as ready for review May 24, 2023 03:42

kaiyan-sheng requested review from a team as code owners May 24, 2023 03:42

kaiyan-sheng requested review from pierrehilbert and cmacknz May 24, 2023 03:42

girodav self-requested a review May 24, 2023 08:32

kaiyan-sheng removed request for pierrehilbert and cmacknz May 24, 2023 16:06

fix billing metricset

ec5662e

tommyers-elastic requested review from zmoog and tommyers-elastic May 24, 2023 20:00

kaiyan-sheng added 4 commits May 24, 2023 15:44

add dynamic label for account label

3373931

update doc

20fb82f

Merge remote-tracking branch 'upstream/main' into include_linked_acco…

56d567d

…unts

fix unit tests

34586ee

tommyers-elastic approved these changes May 25, 2023

View reviewed changes

girodav reviewed May 25, 2023

View reviewed changes

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Show resolved Hide resolved

girodav reviewed May 25, 2023

View reviewed changes

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved

girodav reviewed May 25, 2023

View reviewed changes

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved

girodav reviewed May 25, 2023

View reviewed changes

x-pack/metricbeat/module/aws/cloudwatch/cloudwatch.go Outdated Show resolved Hide resolved

girodav approved these changes May 25, 2023

View reviewed changes

cmacknz approved these changes May 25, 2023

View reviewed changes

kaiyan-sheng added 2 commits May 25, 2023 13:39

use strings.Join for creating label

6640de8

Merge branch 'main' into include_linked_accounts

57da06b

kaiyan-sheng added backport-v8.8.0 Automated backport with mergify and removed backport-v8.8.0 Automated backport with mergify labels May 25, 2023

kaiyan-sheng merged commit 217e658 into elastic:main May 26, 2023
116 checks passed

kaiyan-sheng deleted the include_linked_accounts branch May 26, 2023 03:53

zmoog added backport-v7.17.0 Automated backport with mergify backport-v8.7.0 Automated backport with mergify backport-v8.8.0 Automated backport with mergify labels May 26, 2023

mergify bot pushed a commit that referenced this pull request May 26, 2023

[AWS] Collect metrics from linked accounts (#35540)

3db37a7

(cherry picked from commit 217e658)

mergify bot pushed a commit that referenced this pull request May 26, 2023

[AWS] Collect metrics from linked accounts (#35540)

81830e0

(cherry picked from commit 217e658)

mergify bot mentioned this pull request May 26, 2023

[7.17](backport #35540) [AWS] Collect metrics from linked accounts #35585

Closed

This was referenced May 26, 2023

[8.7](backport #35540) [AWS] Collect metrics from linked accounts #35586

Merged

[8.8](backport #35540) [AWS] Collect metrics from linked accounts #35587

Merged

zmoog pushed a commit that referenced this pull request May 26, 2023

[AWS] Collect metrics from linked accounts (#35540) (#35586)

67e61d3

(cherry picked from commit 217e658) Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

zmoog pushed a commit that referenced this pull request May 30, 2023

[AWS] Collect metrics from linked accounts (#35540)

23972a7

(cherry picked from commit 217e658)

chrisberkhout pushed a commit that referenced this pull request Jun 1, 2023

[AWS] Collect metrics from linked accounts (#35540)

4c4d658

kaiyan-sheng mentioned this pull request Jun 5, 2023

[AWS] Support collecting dynamic labels for account ID and account name elastic/integrations#6232

Closed

reakaleek mentioned this pull request Jul 19, 2023

Fix ironbank validation in 8.8 #36115

Closed

6 tasks

heisenbergye mentioned this pull request Jan 21, 2024

[AWS] Requests include loggroups from linked accounts #37681

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWS] Collect metrics from linked accounts #35540

[AWS] Collect metrics from linked accounts #35540

kaiyan-sheng commented May 22, 2023 •

edited

mergify bot commented May 22, 2023

elasticmachine commented May 23, 2023 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

tommyers-elastic left a comment

tommyers-elastic May 25, 2023

tommyers-elastic May 25, 2023

girodav May 25, 2023

girodav left a comment

cmacknz left a comment

kaiyan-sheng commented May 25, 2023


		label := metricName + labelSeparator
		label := metric.AccountID + labelSeparator + "${PROP('AccountLabel')}" + labelSeparator + metricName + labelSeparator

[AWS] Collect metrics from linked accounts #35540

[AWS] Collect metrics from linked accounts #35540

Conversation

kaiyan-sheng commented May 22, 2023 • edited

What does this PR do?

Checklist

Related issues

Use cases

Config 1: Collect metrics from a given namespace.

Config 2: Collect a specific metric from a given instance.

Config3: collect a specific metric from all instances both from the linked accounts and the monitoring account.

Sample Output

mergify bot commented May 22, 2023

elasticmachine commented May 23, 2023 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

tommyers-elastic left a comment

Choose a reason for hiding this comment

tommyers-elastic May 25, 2023

Choose a reason for hiding this comment

tommyers-elastic May 25, 2023

Choose a reason for hiding this comment

girodav May 25, 2023

Choose a reason for hiding this comment

girodav left a comment

Choose a reason for hiding this comment

cmacknz left a comment

Choose a reason for hiding this comment

kaiyan-sheng commented May 25, 2023

kaiyan-sheng commented May 22, 2023 •

edited

elasticmachine commented May 23, 2023 •

edited by jenkins-beats-ci bot