Async fetching of cost data

[spaelling]

I have built and maintained, and rewritten, the logic for fetching and processing cost data for a few years, so I am quite intimate with these APIs, and one major issue is that getting larger amounts of data is painfully slow.

This is both due to using pagination, and that JSON is very inefficient as a large portion of the raw data fetched are json properties.

There is a much less known and poorly documented (I have had the cost management team confirm this is officially supported) way of getting large amounts of data. The endpoint looks something like below. basically just appending /download

/subscriptions/000-000-000-000-000/providers/Microsoft.Consumption/usageDetails/download?api-version=2019-10-01&metric=AmortizedCost&ln=en&startDate=2023/01/01&endDate=2023/05/01

What happens is that you get an immediate response and in the response headers the one called location contains a uri that keeps track of an async operation that is generating a csv file. You can poll this every few seconds, and eventually it will provide a link to a the csv (it uses a SAS token in a storage account).

(you could try it out in Postman or similar to get a feel of the speed)

You would then have to download the csv file and parse the data.

You would probably need some sort of logic to determine if this method is faster than getting it as json and using pagination, or maybe just a switch in the cli call.

My C# is very rusty, but would be happy to assist otherwise.

[mivano]

Thanks for the suggestion. At the moment, the calls are performant enough for me. But I can imagine I or someone else will easily run into this issue. I do notice some rate limiting once in a while though.

Might be a nice option to offer both API calls as an option. I will have a play with it soon.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.