Azure Billing Integration omit API version

[Gill-Bates]

Elasticsearch Version

Version: 8.9.0, Build: deb/8aa461beb06aa0417a231c345a1b8c38fb498a0d/2023-07-19T14:43:58.555259655Z

Installed Plugins

No response

Java Version

JVM: 20.0.2

OS Version

Linux vmelastic 6.1.0-10-cloud-amd64 elastic/elasticsearch#1 SMP PREEMPT_DYNAMIC Debian 6.1.38-2 (2023-07-27) x86_64 GNU/Linu

Problem Description

When the integration makes a request against the Billing API, it must include the currently valid API version. It does not do that. Therefore, the integration is not functional.

Wrong (current behavior):
https://management.azure.com/providers/Microsoft.Billing/billingAccounts/1234567/providers/Microsoft.Consumption/usageDetails

Correct (expected behavior):
https://management.azure.com/providers/Microsoft.Billing/billingAccounts/1234567/providers/Microsoft.Consumption/usageDetails?api-version=2023-05-01

Steps to Reproduce

1. Install Plugin
2. Connect with Azure

Logs (if relevant)

[elastic_agent.metricbeat][error] Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/providers/Microsoft.Billing/billingAccounts/1234567/providers/Microsoft.Consumption/usageDetails

{
  "error": {
    "code": "MissingApiVersionParameter",
    "message": "The api-version query parameter (?api-version=) is required for all requests."
  }
}

[elasticsearchmachine]

Pinging @elastic/es-distributed (Team:Distributed)

[zmoog]

@Gill-Bates, from your report, I see you are using the Elastic Agent version 8.9.0 to collect billing data using the Azure Billing integration.

• Which version of the Azure Billing integration are you using?
• Can you share the configuration setting you are using with the private and authentication information redacted?

I think you set the "Billing Scope Account ID" from the error message.

Is it correct?

[Gill-Bates]

@zmoog Yes, this is correct. I want to fetch the costs from the Billing Account Scope.

[zmoog]

I tried to reproduce this issue but failed. I need help getting this error.

I tried using stack version 8.9.0 on two different Azure account types:

• Enterprise Agreement (EA) account
• Microsoft Customer Agreement (MCA) account

The call to Usage Detail API succeeded in both cases.

I mention the account types EA and MCA because they have different behaviors in the billing area.

Which Azure account type is your account?

---

You also reported another issue with the billing account ID. At first glance, this problem may happen if the values contain only digits, but in my experience, the billing account IDs have characters other than digits, for example:

9bb0998a-65ab-4c38-91ac-8073a63f191f:f3ba39d8-9b8d-41f1-8aa1-b3fedebc1429_2019-05-31

The billing account ID format looks like this <UUID>:<UUID>_YYYY-MM-DD.

What is the format of your billing account ID?

[Gill-Bates]

@zmoog I have re-installed my entire Elasticstack. I summarize my environment:

Azure

Elastic

• Elasticsearch v. 8.9.1
• Elastic-Agent v. 8.9.1
• Azure Billing Integration v 1.2.0

My Azure Environment:

• Enterprise Agreement
• Billing Account Id 8 Digit Number (e.g. 12345678)

grep "error" /var/log/elastic-agent/elastic-agent-20230825.ndjson

{
    "log.level": "warn",
    "@timestamp": "2023-08-25T12:36:26.773+0200",
    "message": "error fetching EC2 Identity Document: operation error ec2imds: GetInstanceIdentityDocument, failed to get API token, cannot get API token, operation error ec2imds: getToken, http response error StatusCode: 400, request to EC2 IMDS failed.",
    "component": {
        "binary": "filebeat",
        "dataset": "elastic_agent.filebeat",
        "id": "filestream-monitoring",
        "type": "filestream"
    },
    "log": {
        "source": "filestream-monitoring"
    },
    "service.name": "filebeat",
    "ecs.version": "1.6.0",
    "log.logger": "add_cloud_metadata",
    "log.origin": {
        "file.line": 91,
        "file.name": "add_cloud_metadata/provider_aws_ec2.go"
    },
    "ecs.version": "1.6.0"
}
{
    "log.level": "error",
    "@timestamp": "2023-08-25T12:36:40.329+0200",
    "message": "Error fetching data for metricset azure.billing: error retrieving usage information: retrieving usage details failed in client: GET https://management.azure.com/providers/Microsoft.Billing/billingAccounts/12345678/providers/Microsoft.Consumption/usageDetails\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: MissingApiVersionParameter\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"MissingApiVersionParameter\",\n    \"message\": \"The api-version query parameter (?api-version=) is required for all requests.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
    "component": {
        "binary": "metricbeat",
        "dataset": "elastic_agent.metricbeat",
        "id": "azure/metrics-default",
        "type": "azure/metrics"
    },
    "log": {
        "source": "azure/metrics-default"
    },
    "ecs.version": "1.6.0",
    "log.origin": {
        "file.line": 256,
        "file.name": "module/wrapper.go"
    },
    "service.name": "metricbeat",
    "ecs.version": "1.6.0"
}

[zmoog]

@Gill-Bates, let me double-check one more thing. All the accounts I can access, including my private one, have the <UUID>:<UUID>_YYYY-MM-DD billing account ID format.

If you open the following page, do you see the same digit-only codes?

Cost Management + Billing > Billing Account > Settings > Properties > ID

[zmoog]

I was wrong in one of my previous comments. Both Azure accounts I used for my tests are Microsoft Customer Agreement (MCA) accounts.

Cost management, billing data, and behavior can vary depending on the Azure account type.

I need to get access to the billing account level data of the Enterprise Agreement (EA) account. I get back to you as soon as I can try on the same account type.

[zmoog]

Quick update: I am about to enable my app registration to access an EA account. I confirm EA accounts have a different Billing Account ID format; in my case, it's an 8-digit number, as you reported @Gill-Bates.

[zmoog]

We were able to reproduce this issue.

It is not related to EA accounts; it also happens to MCA accounts. The problem occurs when there are more than 1000 usage details items in 24 hours, and the Consumption SDK needs to paginate the results.

Root cause

When there are more than 1000 usage details items, the Consumption SDK paginates the response. Each response page contains a 'next link' to the following page.

The $filter query parameters contain a date range expression like the following:

properties/usageStart eq '2023-11-16' and properties/usageEnd eq '2023-11-16'

Unfortunately, the Azure service adds the query parameters value to the 'next link' without encoding them. Since one of the parameters, $filter contains space, it breaks the URL, causing the following parameters to get lost.

Possible solutions

The problem is in Azure SDK for Go (Consumption SDK) so it seems we can't fix it directly. However, we are already maintaining a fork of the Azure SDK for Go to address another SDK problem. See elastic/beats#37151 to learn more about our for of Azure SDK for Go.

We added a small patch to our fork that URL encodes the next link value and avoids the 400 error.

[gbocchini]

Thanks for the updates @zmoog ! Awesome! Have a great week!

[zmoog]

We are planning to release a fix for this bug in 8.12.0 and backport it to 8.11.2.

[gbocchini]

Thanks for keeping me updated @zmoog ! Awesome job bro! Thanks!