[Azure Billing] Switch to Cost Management API for forecast data #32589

zmoog · 2022-08-03T11:07:09Z

What does this PR do?

Switch the fetching of forecast data from the Azure Consumption API to the Cost Management API.

Why is it important?

The previous forecast endpoint from Consumption API is no longer maintained and does not support scope.

Additional information

The Consumption API version of the forecast endpoint was simple: it offered a simple filter with limited options. Instead, the Cost Management API version is a much richer endpoint with a dozen of options. The current implementation focuses on offering the same functionality using the new API.

The Azure Portal uses the Cost Management API (version 2021-10-01) to implement the Cost Management > Cost analysis page. We used the Azuire Portal as a reference to build the request for the Forecast data in this module.

This Azure Billing module is using an older version of the Cost Management API: 2019-10-01, it is the latest working ¹ version offered in the official Azure SDK. If we need to jump to a later version, we built a fallback solution: the github.com/zmoog/azure-sdk-clients repo contains an unofficial Azure SDK client generated from the OpenAPI spec using the same toolchain Microsoft uses.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

On Azure:

Create a new App registration.
Take note of the "Application (client) ID"; it will be your CLIENT_ID.
Visit App Registrations > App > Certificates & secrets > New client secret.
Take note of the secret "Value"; it will be your CLIENT_SECRET.
Visit Cost Management + Billing > Access control (IAM) > Add role assignment.
Select role "Billing account reader" and the name of your app in "Users, groups, or app".

On your local environment:

Enable the Azure Billing module and use the following configuration:

- module: azure
  metricsets:
    - billing
  enabled: true
  period: 24h
  client_id: '<CLIENT_ID>'
  client_secret: '<CLIENT_SECRET>'
  tenant_id: '<TENANT_ID>'
  subscription_id: '<SUBSCRIPTION_ID>'
  refresh_list_interval: 600s

Build and run Metricbeat

$ cd x-pack/metricbeat
$ mage build 
>> build: Building metricbeat

$ ./metricbeat -e -E cloud.id=<CLOUD_ID> -E cloud.auth=<CLOUD_AUTH> -E output.elasticsearch.allow_older_versions=true

You may need to use output.elasticsearch.allow_older_versions=true if you built Metricbeat from sources and not using the latest version of ES.

Related issues

Closes [Azure] switch to cost management API for forecast data #32033

Version 2020-06-01 does not work for Forecast. ↩

elasticmachine · 2022-08-03T11:22:28Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2022-09-08T10:07:55.477+0000
Duration: 53 min 3 sec

Test stats 🧪

Test	Results
Failed	0
Passed	1244
Skipped	87
Total	1331

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.
run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

aspacca · 2022-08-12T01:48:46Z

x-pack/metricbeat/module/azure/billing/client.go

 	//

-	actualCosts, err := client.BillingService.GetForecast(fmt.Sprintf("properties/chargeType eq '%s'", "Actual"))


~~what was of usage.actualCosts?~~

https://github.com/elastic/beats/pull/32589/files#diff-4cdd545c5b3b0cfc2223812bc91592664236894dc885646b9fad7608c1c384e9R76-R77

@aspacca, we probably should add what actual/forecast data are, WDYT?

aspacca · 2022-08-12T01:49:51Z

x-pack/metricbeat/module/azure/billing/client.go

 	)
 	if err != nil {
 		return usage, fmt.Errorf("retrieving usage details failed in client: %w", err)
 	}

-	usage.UsageDetails = usageDetails.Values()
+	for paginator.NotDone() {


you didn't change the behaviour here, but does it make sense to change and fetch forecasts even if we don't have usage details?

The module collects Usage Details data and forecast data at different time intervals:

usage details: previous day (24 hours)

forecast: current month (30 days)

Suppose we have zero usage in the last 24h and some resource usage in the previous days. In this case, usage details would be empty, but we could have some forecasts based on the days before.

WDYT? I have no previous knowledge of this topic, I reconstructed it from the codebase and Microsoft docs, so any question like this one is more than welcome!

Suppose we have zero usage in the last 24h and some resource usage in the previous days. In this case, usage details would be empty, but we could have some forecasts based on the days before.

WDYT? I have no previous knowledge of this topic, I reconstructed it from the codebase and Microsoft docs, so any question like this one is more than welcome!

I indeed didn't think about the fact that the call to the usage details could fail because there are zero usage in the last 24h :)

Mine was a more generic doubt about the fact that they are two different API calls, that can fail each for different reasons, even transient (ie: by the time we call the first we have a network outage that will be solved by the time of the next call, or just one upstream service is down while the other is up). Does it make sense to keep fetching the second metrics if fetching the first fails?

I would look the answer to this in the dashboards or the way the two metrics are related to each others.
Taking the example you made, and assuming having zero usage data is the reason for a possible failure: I see it can make sense to see in a dashboard that I have zero usage and the forecast is horizontal, rather than missing at all any information. At the same time is the usage is zero and the forecast increase this can be an hint that something is not working properly.

Probably @ravikesarwani might have an opinion here as well.

Does it make sense to keep fetching the second metrics if fetching the first fails?

Yep, I think it makes sense.

In the dashboard, usage details and forecasts have independent visualizations: we don't have one visualization that uses data from both. However, the sum of the usage details matches the amount of actual data in the forecast for the same day.

I'm in favor of changing the logic and continuing with the forecast even if the call for usage details fails for some reason.

@ravikesarwani, let us know what's on your mind.

x-pack/metricbeat/module/azure/billing/data.go

tommyers-elastic

great work. LGTM

x-pack/metricbeat/module/azure/billing/billing_test.go

x-pack/metricbeat/module/azure/billing/billing.go

x-pack/metricbeat/module/azure/billing/client.go

x-pack/metricbeat/module/azure/billing/data.go

x-pack/metricbeat/module/azure/billing/service.go

mergify · 2022-08-16T15:28:29Z

This pull request does not have a backport label.
If this is a bug or security fix, could you label this PR @zmoog? 🙏.
For such, you'll need to label your PR with:

The upcoming major version of the Elastic Stack
The upcoming minor version of the Elastic Stack (if you're not pushing a breaking change)
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:
backport-v8./d.0 is the label to automatically backport to the 8./d branch. /d is the digit

mergify · 2022-08-24T21:04:29Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b switch-azure-forecast-to-cost-management-api upstream/switch-azure-forecast-to-cost-management-api
git merge upstream/main
git push upstream switch-azure-forecast-to-cost-management-api

The previous endpoint from Consumption API is no longer maintained and does not support scope.

After testing with different values of includeFreshPartialCost, the best trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data.

Thanks golangci-lint!

Add a few comments, rename a couple of variables, and fix a typo.

Stopping the processing of the whole set of rows when we detect a wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones

* Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones (cherry picked from commit 86b111d) # Conflicts: # x-pack/metricbeat/module/azure/billing/billing.go # x-pack/metricbeat/module/azure/billing/billing_test.go # x-pack/metricbeat/module/azure/billing/client.go # x-pack/metricbeat/module/azure/billing/client_test.go # x-pack/metricbeat/module/azure/billing/data.go # x-pack/metricbeat/module/azure/billing/data_test.go # x-pack/metricbeat/module/azure/billing/mock_service.go # x-pack/metricbeat/module/azure/billing/service.go

* Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones (cherry picked from commit 86b111d)

…) (#33024) * Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones (cherry picked from commit 86b111d) Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>

* Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones (cherry picked from commit 86b111d)

… for forecast data (#33023) * [Azure Billing] Switch to Cost Management API for forecast data (#32589) * Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones (cherry picked from commit 86b111d) * Fix conflicts * Update NOTICE.txt * Fix linter objections * Fix NOTICE * Fix missing log initialization Co-authored-by: Maurizio Branca <maurizio.branca@gmail.com>

* Switch forecast to Cost Management API The previous endpoint from Consumption API is no longer maintained and does not support scope. * Add comments with findings from tests After testing with different values of includeFreshPartialCost, the best the trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data. * Drop rows with an unexpected format and continue Stopping the processing of the whole set of rows when we detect the wrong row is too hard. With this change, when something is wrong with a row with it we: - log the circumstances - stop processing the current row - continue with the next ones

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 3, 2022

mergify bot assigned zmoog Aug 3, 2022

zmoog added the Team:Cloud-Monitoring Label for the Cloud Monitoring team label Aug 3, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 3, 2022

zmoog added enhancement needs_team Indicates that the issue/PR needs a Team:* label labels Aug 3, 2022

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Aug 3, 2022

zmoog changed the title ~~[Azure Billing] Switch forecast to Cost Management API~~ [Azure Billing] Switch to Cost Management API for forecast data Aug 3, 2022

zmoog mentioned this pull request Aug 3, 2022

[meta] Azure Billing: upgrade Azure SDK clients elastic/integrations#3570

Closed

zmoog force-pushed the switch-azure-forecast-to-cost-management-api branch 2 times, most recently from e06bf12 to 5389aa1 Compare August 9, 2022 15:12

zmoog marked this pull request as ready for review August 10, 2022 14:29

zmoog requested a review from a team as a code owner August 10, 2022 14:29

zmoog requested a review from a team August 10, 2022 14:30

aspacca reviewed Aug 12, 2022

View reviewed changes

tommyers-elastic approved these changes Aug 12, 2022

View reviewed changes

zmoog added backport-v7.17.0 Automated backport with mergify backport-v8.4.0 Automated backport with mergify labels Aug 19, 2022

zmoog force-pushed the switch-azure-forecast-to-cost-management-api branch from a236100 to ab0eb25 Compare August 22, 2022 09:49

zmoog added 7 commits September 8, 2022 11:57

Switch forecast to Cost Management API

54971ad

The previous endpoint from Consumption API is no longer maintained and does not support scope.

Add comments with findings from tests

4e8f704

After testing with different values of includeFreshPartialCost, the best trade-off with the current design of the dashboard is to set it to false and only use consolidated/final data.

Fix goimports

72cc091

Thanks golangci-lint!

Clean up

7697ae2

Add a few comments, rename a couple of variables, and fix a typo.

Fix linter objections

fc894df

Update CHANGELOG

66590a4

zmoog force-pushed the switch-azure-forecast-to-cost-management-api branch from ab0eb25 to 66590a4 Compare September 8, 2022 10:07

zmoog merged commit 86b111d into elastic:main Sep 8, 2022

zmoog deleted the switch-azure-forecast-to-cost-management-api branch September 8, 2022 14:16

This was referenced Sep 8, 2022

[7.17](backport #32589) [Azure Billing] Switch to Cost Management API for forecast data #33023

Merged

[8.4](backport #32589) [Azure Billing] Switch to Cost Management API for forecast data #33024

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Azure Billing] Switch to Cost Management API for forecast data #32589

[Azure Billing] Switch to Cost Management API for forecast data #32589

zmoog commented Aug 3, 2022 •

edited

elasticmachine commented Aug 3, 2022 •

edited by jenkins-beats-ci bot

Build stats

Test stats 🧪

aspacca Aug 12, 2022 •

edited

zmoog Aug 17, 2022

aspacca Aug 12, 2022

zmoog Aug 17, 2022

aspacca Aug 17, 2022

zmoog Aug 17, 2022

zmoog Aug 17, 2022

tommyers-elastic left a comment

mergify bot commented Aug 16, 2022

mergify bot commented Aug 24, 2022

		//

		actualCosts, err := client.BillingService.GetForecast(fmt.Sprintf("properties/chargeType eq '%s'", "Actual"))

[Azure Billing] Switch to Cost Management API for forecast data #32589

[Azure Billing] Switch to Cost Management API for forecast data #32589

Conversation

zmoog commented Aug 3, 2022 • edited

What does this PR do?

Why is it important?

Additional information

Checklist

How to test this PR locally

Related issues

Footnotes

elasticmachine commented Aug 3, 2022 • edited by jenkins-beats-ci bot

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

aspacca Aug 12, 2022 • edited

Choose a reason for hiding this comment

zmoog Aug 17, 2022

Choose a reason for hiding this comment

aspacca Aug 12, 2022

Choose a reason for hiding this comment

zmoog Aug 17, 2022

Choose a reason for hiding this comment

aspacca Aug 17, 2022

Choose a reason for hiding this comment

zmoog Aug 17, 2022

Choose a reason for hiding this comment

zmoog Aug 17, 2022

Choose a reason for hiding this comment

tommyers-elastic left a comment

Choose a reason for hiding this comment

mergify bot commented Aug 16, 2022

mergify bot commented Aug 24, 2022

zmoog commented Aug 3, 2022 •

edited

elasticmachine commented Aug 3, 2022 •

edited by jenkins-beats-ci bot

aspacca Aug 12, 2022 •

edited