[Azure] [Billing] Add Azure resource tags from Usage Details API #36428

zmoog · 2023-08-28T08:44:41Z

Proposed commit message

With this change, we are adding support for Azure resource tags from the response from the Usage Details API. Both legacy and modern usage details data formats include tags from resources.

We take tag values from the usage details response and store them in the event as azure.resource.tags field. We can leverage the existing module level mapping for these field.

Multiple users asked for including resource tags to make the most of usage details data.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Fetch data from an Enterprise Agreement (EA) account
Fetch data from an Microsoft Customer Agreement (MCA) account

How to test this PR locally

Related issues

Closes [Azure - Billing Metrics] - Fails to ingest tags resource field #36373

Use cases

Screenshots

Logs

mergify · 2023-08-28T08:45:17Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

zmoog · 2023-08-28T08:51:59Z

I added a simple mapping of tag values from legacy (EA) and modern (MCA) billing account types.

I added a few tags to Azure resources, but I am not getting those tags from Usage Details API; I would like to know if this kind of data needs some time to propagate across Azure systems.

I am waiting some time before trying again.

elasticmachine · 2023-08-28T09:40:22Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2023-08-30T06:10:18.768+0000
Duration: 51 min 7 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1478
Skipped	96
Total	1574

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

Both legacy and modern data formats support resource tags.

The Azure module already has an existing mapping for the `azure.resource.tags` field.

x-pack/metricbeat/module/azure/billing/data.go

muthu-mps · 2023-08-30T06:12:55Z

I have tried enabling the billing module with this PR and the tags are added to the resources fields in the metrics. The change looks good!

devamanv

Changes look good!

mergify · 2023-08-30T09:07:44Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)

To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

zmoog · 2023-08-30T09:08:22Z

Removed the backport labels: I prefer to merge now and take some time and think about if and which version backport to.

) * Add resource tags from Usage Details API Both legacy and modern data formats support resource tags. * Store tags in the azure.resource.tags field The Azure module already has an existing mapping for the `azure.resource.tags` field. * Add a unit test to check the tags field mapping * Update changelog --------- Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com> (cherry picked from commit 55edac5)

zmoog · 2023-08-30T10:44:24Z

Since the Azure module already includes mapping for azure.resource.tags, the lack of tag values can be considered a bug. So, I am backporting the fix to 8.9 and 8.10.

… Usage Details API (#36463) * [Azure] [Billing] Add Azure resource tags from Usage Details API (#36428) * Add resource tags from Usage Details API Both legacy and modern data formats support resource tags. * Store tags in the azure.resource.tags field The Azure module already has an existing mapping for the `azure.resource.tags` field. * Add a unit test to check the tags field mapping * Update changelog --------- Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com> (cherry picked from commit 55edac5) * Remove extra changelog comments --------- Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

) (#36464) * Add resource tags from Usage Details API Both legacy and modern data formats support resource tags. * Store tags in the azure.resource.tags field The Azure module already has an existing mapping for the `azure.resource.tags` field. * Add a unit test to check the tags field mapping * Update changelog --------- Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com> (cherry picked from commit 55edac5) Co-authored-by: Maurizio Branca <maurizio.branca@elastic.co>

…stic#36428) * Add resource tags from Usage Details API Both legacy and modern data formats support resource tags. * Store tags in the azure.resource.tags field The Azure module already has an existing mapping for the `azure.resource.tags` field. * Add a unit test to check the tags field mapping * Update changelog --------- Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

zmoog self-assigned this Aug 28, 2023

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 28, 2023

zmoog added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Aug 28, 2023

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 28, 2023

zmoog force-pushed the zmoog/azure-billing-add-tags branch from 149fc56 to f066926 Compare August 28, 2023 10:10

zmoog changed the title ~~[Azure] [Billing] Add resource tags from Usage Details API~~ [Azure] [Billing] Add Azure resource tags from Usage Details API Aug 28, 2023

zmoog added backport-v8.9.0 Automated backport with mergify backport-v8.10.0 Automated backport with mergify labels Aug 28, 2023

zmoog marked this pull request as ready for review August 28, 2023 15:26

zmoog requested a review from a team as a code owner August 28, 2023 15:26

zmoog requested review from a team August 28, 2023 15:26

zmoog force-pushed the zmoog/azure-billing-add-tags branch from cde009c to 7445190 Compare August 28, 2023 22:36

girodav approved these changes Aug 29, 2023

View reviewed changes

zmoog added 4 commits August 29, 2023 17:46

Add resource tags from Usage Details API

216a42b

Both legacy and modern data formats support resource tags.

Store tags in the azure.resource.tags field

16387cc

The Azure module already has an existing mapping for the `azure.resource.tags` field.

Add a unit test to check the tags field mapping

0d558c0

Update changelog

2f5365a

zmoog force-pushed the zmoog/azure-billing-add-tags branch from 7445190 to 2f5365a Compare August 29, 2023 15:47

devamanv reviewed Aug 29, 2023

View reviewed changes

x-pack/metricbeat/module/azure/billing/data.go Show resolved Hide resolved

Merge branch 'main' into zmoog/azure-billing-add-tags

93ca4a1

lalit-satapathy mentioned this pull request Aug 30, 2023

[Azure - Billing Metrics] - Fails to ingest tags resource field #36373

Closed

devamanv approved these changes Aug 30, 2023

View reviewed changes

zmoog removed backport-v8.9.0 Automated backport with mergify backport-v8.10.0 Automated backport with mergify labels Aug 30, 2023

zmoog merged commit 55edac5 into elastic:main Aug 30, 2023
21 checks passed

zmoog deleted the zmoog/azure-billing-add-tags branch August 30, 2023 09:09

zmoog added the backport-v8.9.0 Automated backport with mergify label Aug 30, 2023

mergify bot mentioned this pull request Aug 30, 2023

[8.9](backport #36428) [Azure] [Billing] Add Azure resource tags from Usage Details API #36463

Merged

zmoog added backport-v8.10.0 Automated backport with mergify bug labels Aug 30, 2023

mergify bot mentioned this pull request Aug 30, 2023

[8.10](backport #36428) [Azure] [Billing] Add Azure resource tags from Usage Details API #36464

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure] [Billing] Add Azure resource tags from Usage Details API #36428

[Azure] [Billing] Add Azure resource tags from Usage Details API #36428

zmoog commented Aug 28, 2023 •

edited

mergify bot commented Aug 28, 2023

zmoog commented Aug 28, 2023

elasticmachine commented Aug 28, 2023 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

muthu-mps commented Aug 30, 2023

devamanv left a comment

mergify bot commented Aug 30, 2023

zmoog commented Aug 30, 2023 •

edited

zmoog commented Aug 30, 2023

[Azure] [Billing] Add Azure resource tags from Usage Details API #36428

[Azure] [Billing] Add Azure resource tags from Usage Details API #36428

Conversation

zmoog commented Aug 28, 2023 • edited

Proposed commit message

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

mergify bot commented Aug 28, 2023

zmoog commented Aug 28, 2023

elasticmachine commented Aug 28, 2023 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

muthu-mps commented Aug 30, 2023

devamanv left a comment

Choose a reason for hiding this comment

mergify bot commented Aug 30, 2023

zmoog commented Aug 30, 2023 • edited

zmoog commented Aug 30, 2023

zmoog commented Aug 28, 2023 •

edited

elasticmachine commented Aug 28, 2023 •

edited by jenkins-beats-ci bot

zmoog commented Aug 30, 2023 •

edited