[AWS] Support owning account for cross account monitoring #40691

Kavindu-Dodan · 2024-09-04T16:48:15Z

Proposed commit message

Adds owning_account configuration option for AWS where it accepts a source account ID. Internally, this maps to the ListMetrics API's ¹ OwningAccount parameter. Note that include_linked_accounts should be set to true to use the account filtering.

include_linked_accounts: true # default to true, hence can be ignored in configuration yaml
owning_account: 111111111111

See the screenshot below from an actual run with owning account filtering,

In comparison, consider the screenshot below where both monitoring account and linked accounts are visible as there is no filtering for owning_account,

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

None :owning_account is a new configuration

How to test this PR locally

Build x-pack metricsbeat
Enable AWS monitoring with include_linked_accounts set to true (default is already true) & owning_account account ID
Observe metrics data and validate that cloud.account.id match with owning_account ID

Related issues

Closes #40570

https://docs.aws.amazon.com/AmazonCloudWatch/latest/APIReference/API_ListMetrics.html#API_ListMetrics_RequestParameters ↩

elasticmachine · 2024-09-04T16:50:29Z

Pinging @elastic/obs-ds-hosted-services (Team:obs-ds-hosted-services)

CHANGELOG.next.asciidoc

x-pack/metricbeat/modules.d/aws.yml.disabled

kaiyan-sheng

LGTM!

mergify · 2024-09-11T11:54:02Z

backport-8.x has been added to help with the transition to the new branch 8.x.

kaiyan-sheng · 2024-09-12T21:53:19Z

@elastic/elastic-agent-data-plane Could we get a review for this PR please? Thanks!!

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

pierrehilbert

Approving for the Elastic Agent Data Plane part as this is only adding new configuration entries in the yml file.

Kavindu-Dodan · 2024-09-17T15:04:56Z

Approving for the Elastic Agent Data Plane part as this is only adding new configuration entries in the yml file.

Thank you @pierrehilbert Yes, this adds owning_account option which works alongside include_linked_accounts property.

* support OwniningAccount filtering Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the new paramter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * Update CHANGELOG.next.asciidoc Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update x-pack/metricbeat/modules.d/aws.yml.disabled Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * add missing docuemntation of owning_account parameter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 03e7699)

pierrehilbert · 2024-09-17T15:08:40Z

I just remembered that you mentioned the target is 8.15.2: you should add the backport-8.15 label if this is the case.

* support OwniningAccount filtering Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the new paramter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * Update CHANGELOG.next.asciidoc Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update x-pack/metricbeat/modules.d/aws.yml.disabled Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * add missing docuemntation of owning_account parameter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 03e7699)

…40860) * support OwniningAccount filtering Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the new paramter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * Update CHANGELOG.next.asciidoc Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update x-pack/metricbeat/modules.d/aws.yml.disabled Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * add missing docuemntation of owning_account parameter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 03e7699) Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

…t monitoring (#40864) * [AWS] Support owning account for cross account monitoring (#40691) * support OwniningAccount filtering Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * document the new paramter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * add changelog Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> * Update CHANGELOG.next.asciidoc Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * Update x-pack/metricbeat/modules.d/aws.yml.disabled Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> * add missing docuemntation of owning_account parameter Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> --------- Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co> Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co> (cherry picked from commit 03e7699) * Update CHANGELOG.next.asciidoc for the backport --------- Co-authored-by: Kavindu Dodanduwa <Kavindu-Dodan@users.noreply.github.com>

Kavindu-Dodan added enhancement backport-skip Skip notification from the automated backport with mergify labels Sep 4, 2024

Kavindu-Dodan requested review from kaiyan-sheng and a team September 4, 2024 16:48

Kavindu-Dodan requested a review from a team as a code owner September 4, 2024 16:48

Kavindu-Dodan requested review from faec and VihasMakwana September 4, 2024 16:48

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2024

mergify bot assigned Kavindu-Dodan Sep 4, 2024

Kavindu-Dodan added the Team:obs-ds-hosted-services Label for the Observability Hosted Services team label Sep 4, 2024

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Sep 4, 2024

Kavindu-Dodan force-pushed the feat/support-owning-account-in-metrics branch 3 times, most recently from 69c5245 to a5e9003 Compare September 6, 2024 14:29

kaiyan-sheng reviewed Sep 9, 2024

View reviewed changes

CHANGELOG.next.asciidoc Outdated Show resolved Hide resolved

kaiyan-sheng reviewed Sep 9, 2024

View reviewed changes

x-pack/metricbeat/modules.d/aws.yml.disabled Outdated Show resolved Hide resolved

Kavindu-Dodan force-pushed the feat/support-owning-account-in-metrics branch 3 times, most recently from 41b36f2 to 580d9ed Compare September 9, 2024 16:45

kaiyan-sheng approved these changes Sep 9, 2024

View reviewed changes

Kavindu-Dodan force-pushed the feat/support-owning-account-in-metrics branch from 580d9ed to b9ed778 Compare September 10, 2024 16:55

v1v added the backport-v8.x label Sep 10, 2024

mergify bot added the backport-8.x Automated backport to the 8.x branch with mergify label Sep 11, 2024

v1v removed the backport-v8.x label Sep 11, 2024

Kavindu-Dodan removed the backport-skip Skip notification from the automated backport with mergify label Sep 12, 2024

Kavindu-Dodan added 2 commits September 13, 2024 09:15

support OwniningAccount filtering

60c229b

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

document the new paramter

32688ed

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan and others added 4 commits September 13, 2024 09:15

add changelog

903ce38

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Update CHANGELOG.next.asciidoc

4eb164d

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

Update x-pack/metricbeat/modules.d/aws.yml.disabled

f379b34

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>

add missing docuemntation of owning_account parameter

104fdb7

Signed-off-by: Kavindu Dodanduwa <kavindu.dodanduwa@elastic.co>

Kavindu-Dodan force-pushed the feat/support-owning-account-in-metrics branch from b9ed778 to 104fdb7 Compare September 13, 2024 16:15

pierrehilbert added the Team:Elastic-Agent-Data-Plane Label for the Agent Data Plane team label Sep 16, 2024

pierrehilbert approved these changes Sep 17, 2024

View reviewed changes

Kavindu-Dodan merged commit 03e7699 into elastic:main Sep 17, 2024
29 checks passed

mergify bot mentioned this pull request Sep 17, 2024

[8.x](backport #40691) [AWS] Support owning account for cross account monitoring #40860

Merged

6 tasks

kaiyan-sheng added the backport-8.15 Automated backport to the 8.15 branch with mergify label Sep 17, 2024

mergify bot mentioned this pull request Sep 17, 2024

[8.15](backport #40691) [AWS] Support owning account for cross account monitoring #40864

Merged

6 tasks

kaiyan-sheng mentioned this pull request Sep 26, 2024

[AWS] Support owning account for cross account monitoring elastic/integrations#11260

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[AWS] Support owning account for cross account monitoring #40691

[AWS] Support owning account for cross account monitoring #40691

Kavindu-Dodan commented Sep 4, 2024 •

edited

Loading

elasticmachine commented Sep 4, 2024

kaiyan-sheng left a comment

mergify bot commented Sep 11, 2024

kaiyan-sheng commented Sep 12, 2024

pierrehilbert left a comment

Kavindu-Dodan commented Sep 17, 2024

pierrehilbert commented Sep 17, 2024

[AWS] Support owning account for cross account monitoring #40691

[AWS] Support owning account for cross account monitoring #40691

Conversation

Kavindu-Dodan commented Sep 4, 2024 • edited Loading

Proposed commit message

Checklist

Disruptive User Impact

How to test this PR locally

Related issues

Footnotes

elasticmachine commented Sep 4, 2024

kaiyan-sheng left a comment

Choose a reason for hiding this comment

mergify bot commented Sep 11, 2024

kaiyan-sheng commented Sep 12, 2024

pierrehilbert left a comment

Choose a reason for hiding this comment

Kavindu-Dodan commented Sep 17, 2024

pierrehilbert commented Sep 17, 2024

Kavindu-Dodan commented Sep 4, 2024 •

edited

Loading