SPIKE: Azure Cost Details API Migration

[4upz]

Related issue with context: #1175

The purpose of this spike is to research necessary implementation details needed for implementing support for the Azure Cost Details API. This API will be replacing the Usage Details API.

it should cover the following:

• Verifying process of enabling Cost Details API access (if any)
  • Azure subscription types supported (For documentation)
  • Permissions needed (for documentation)
• Verify structure of Cost Details API response contains data needed for CCF estimates
• Implementation steps for querying Cost Details API for a given date range
  • Any pagination logic needed (i.e. does it return an async iterable response)
  • Libraries/packages needed
• Potential migration steps
  • What parts of the Azure package can be converted vs what needs to be rewritten
• Scalability for large usage
  • Maximum boundary before needing to recommend Exports method

[4upz]

After looking into this, I found the following:

• The Cost Details API is the official replacement for the previous Consumption Management’s “Usage Details List” API – the main endpoint that we used to query usage data for a subscription.
• Using this API will require installing the “@azure/arm-csotmanagement” library to replace the “@azure/arm-consumption” library.
• It is recommended to chunk calls into small date ranges for more manageable files. We will need to decide what an appropriate maximum should be (and provide a config to customize this maximum)

Important Limitations:

• Maximum time range of one month per report is supported
• Historical data is only supported for up to 13 months prior
  • I.e. seeding 13 months of historical data is recommended to be done using 13 calls for one month datasets

The Cost Details api serves as a job creator via a polling endpoint that you can interact with via the cost management API. Using the sdk-for-js, you can use the costManagementClient.generateCostDetailsReport.beginCreateOperation to initiate a report for a given start and end date. You can have this kick off the task in the background and check back for results later using costManagementClient.generateCostDetailsReport.beginGetOperationrResults to fetch the status/results or by simply waiting for a response by calling beginCreateOperationAndWait instead.

The reports can be scoped at the billing level using a direct billing account ID (more performant, less requests) or at the subscription level as we do now (less performant, more requests/reports). The subscription level would require gathering a list of subscriptions as we currently do and then creating a report for each scoped subscription.

The report is confirmed to have all of the same properties that CCF needs for calculating estimates and parsing usage data. The following steps can be taken after a report is generated:

• The Response will include a blobs property with an array of links (typically one unless response needs to be split) to download the generated csv file with the cost & usage data
• This response can be fetched (using node-fetch or axios) to get the csv file’s content using response.text(). This content can then be parsed to JSON using csvtojson to convert it synchronously or streams for reading content line-by-line
• The JSON response can be used to create new “ConsumptionRow” objects (probably renamed to CostDetailsRow)

Some future improvements after first implementation (out of scope for now):

• Storing historical rows either within the user’s own storage account or within the cache (i.e. MongoDB)
  • Important to note that reports are only available for 24 hours. Azure recommends storing them instead of repeating requests for the same start/end date

Note on permissions and scale:

• Cost Details and the Cost Management API equires Azure Enterprise Customer account
• Recommended for small cost datasets (2GB or 2million rows maximum)
• Azure recommends Cost Exports for larger scopes
• Not recommended for ongoing data ingestion
• Azure recommends Cost Exports for recurring datasets
• Rate limits have the same retry-after field to implement retry logic when they’re encountered.
• API only supports Enterprise or Customer Agreements
  • Pay-As-You-Go is not supported – recommended to use Exports
  • Cost Management Reader (Read Only) access is required
    • RBAC - “Cost Analysis / Forecast / Query / Cost Details API”