SPIKE: Azure Cost Export Support

[4upz]

Related Issue with full context: #1175

This spike should serve as research for implementing support for Cost Exports through Azure's daily export feature.

It should cover the following:

• Verifying process of setting up exports
  • Azure subscription types supported
  • Permissions needed (for documentation purposes)
• Verify structure of exported data contains columns needed for CCF estimates
• Implementation steps for parsing Azure Export Data
  • Libraries needed
  • Azure permissions and related sdk libraries needed
• Scalability for large implementations
  • Example: Potential for support large csv files with multiple subscriptions, or handling multiple csv files at a per-subscription level

[4upz]

After looking into this, I found the following:

Cost data exports are available to Enterprise Agreement (EA), Microsoft Customer Agreement customers, as well as alternative payment accounts that include:

• MSDN
• Microsoft Online Service Program (MOSP) pay-as-you-go
• Visual Studio Azure subscription

Azure does mention that the Usage Details API can be used for the above listed alternative account types. It is only deprecated for EA or MCA users. If we wish to include a low-overhead option for these users, it may be worth implementing logic that uses the account type to determine the method for fetching usage data – retaining the current implementation for small users. However, they still warn that the Usage Details will still eventually be turned off, and that the Cost Details API will turn on support for these accounts before then.

Setting up Exports through the Azure Portal

Azure’s documentation provides detailed instructions on how to schedule exports to a storage account using the azure portal. Those instructions can be found here.

There are some important things to call out:

• The same permissions regarding Cost Management access applies here as it does for the Cost Details API.
• Azure Storage write permissions are required for configuring the storage account destination and reading the stored exports.
• Exports support daily (month-to-date aggregated), weekly (last seven days), and monthly (last month) export configurations
  • Daily aggregates the latest data to previous daily exports
  • There is also an option for one-time exports that is useful for historical data
    • Can take up to 2 hours
    • Maximum of 90 days of historical data is available
• Subscription cost information can be aggregated using management groups. It will be important to call this out in the documentation in case there are issues getting usage data for all desired accounts.
• Export data can be chunked into smaller files if large data ingestion becomes an issue

Setting up Exports through the Exports API

This is currently supported and may be worth implementing as a CLI command to ease and automate setup for users. Similar to the Cost Details API, it is available through the @azure/arm-costmanagement package.

See examples for creating exports by billing accounts and other methods.

The results are similar to setting up exports traditionally, in which csv files will be recurrently generated and stored in the configured storage account.

Implementing Exports

Exports serve as the more scalable option for persisting historical estimates and fetching large amounts of usage data. The resulting CSV files can be parsed/streamed similar to the Cost Details API implementation.

It is recommended to implement #1222 first.

The @azure/storage-blobs package will need to be used to fetch cost export data from the configured storage container.

It does bring into question how using Exports will affect estimation requests, and how to handle requests for data ranges that are not available or have yet to be exported. We may want to consider a config that tells CCF which method to use, and if using Exports that it is only usable with data that has already been exported. Alternatively, we could simply use the API to create new exports of the given date range. We would need to be aware of the 90-day limit, and provide additional parameters for chunking data and informing the user on the creation of new exports due to the amount of time it may take for large data.