feat: calculate cost of Kinesis Data Streams using provisioned mode

[jbleduigou]

Problem

Add the cost calculation for Kinesis Data Streams using provisioned mode.

Solution

We first need to retrieve the prices by using the pricing API of AWS.
Then we have two parts in the cost : a cost per shard and then a cost depending on the number of events ingested by the stream.

Changes Made

• Added the cost calculation logic in providers/aws/kinesis/streams.go
• Added unit tests accordingly in providers/aws/kinesis/streams_test.go
• Added two mocks to support the unit tests in mocks/cloudwatch.go and mocks/pricing.go

How to Test

You will need to create a Kinesis Data Steam in the AWS Console.
You might need to wait a few hours before the cost calculation start to kick in.

Screenshots

Notes

n/a

Checklist

• Code follows the contributing guidelines
• Changes have been thoroughly tested
• Documentation has been updated, if necessary
• Any dependencies have been added to the project, if necessary

Reviewers

@mlabouardy

[Azanul]

Could you also attach the pricing output received, I'd like to check the price map keys

[jbleduigou]

Here is the pricing:

{
    "PriceList": [
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of GBs of data retrieved by consumers using enhanced fan-out\",\"usagetype\":\"ReadBytes\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"EnhancedFanoutDataRetrieval\",\"group\":\"Enhanced fan-out GB of data retrieved\"},\"sku\":\"4V6WBV2SGNWG4M9M\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"4V6WBV2SGNWG4M9M.JRTCKXETXF\":{\"priceDimensions\":{\"4V6WBV2SGNWG4M9M.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB\",\"endRange\":\"Inf\",\"description\":\"$0.013 per GB of data retrieved by consumers using enhanced fan-out\",\"appliesTo\":[],\"rateCode\":\"4V6WBV2SGNWG4M9M.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0130000000\"}}},\"sku\":\"4V6WBV2SGNWG4M9M\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-ExtendedRetention-ByteHrs\",\"description\":\"per byte hour data stored within 1-7 days\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandExtendedRetentionByteHrs\",\"group\":\"OnDemand\"},\"sku\":\"59DG9NNVUBGN3C5M\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"59DG9NNVUBGN3C5M.JRTCKXETXF\":{\"priceDimensions\":{\"59DG9NNVUBGN3C5M.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB-month\",\"endRange\":\"Inf\",\"description\":\"0.10 per GB-month for retention within 1-7 days\",\"appliesTo\":[],\"rateCode\":\"59DG9NNVUBGN3C5M.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.1000000000\"}}},\"sku\":\"59DG9NNVUBGN3C5M\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-BilledOutgoingEFOBytes\",\"description\":\"Per GB of data retrieved by consumers using enhanced fan-out\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandEFODataRetrieval\",\"group\":\"OnDemand\"},\"sku\":\"5KE89N55PB48GC9E\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"5KE89N55PB48GC9E.JRTCKXETXF\":{\"priceDimensions\":{\"5KE89N55PB48GC9E.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB\",\"endRange\":\"Inf\",\"description\":\"0.05 per GB of data read using EFO Consumer\",\"appliesTo\":[],\"rateCode\":\"5KE89N55PB48GC9E.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0500000000\"}}},\"sku\":\"5KE89N55PB48GC9E\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"maximumExtendedStorage\":\"7 days\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of standard shard hours\",\"usagetype\":\"Storage-ShardHour\",\"standardStorageRetentionIncluded\":\"1 day\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"shardHourStorage\",\"group\":\"Provisioned shard hour\"},\"sku\":\"7RXVY4V5SG8NQC38\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"7RXVY4V5SG8NQC38.JRTCKXETXF\":{\"priceDimensions\":{\"7RXVY4V5SG8NQC38.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"ShardHour\",\"endRange\":\"Inf\",\"description\":\"$0.015 per provisioned shard-hour\",\"appliesTo\":[],\"rateCode\":\"7RXVY4V5SG8NQC38.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0150000000\"}}},\"sku\":\"7RXVY4V5SG8NQC38\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-BilledIncomingBytes\",\"description\":\"Per GB of data ingested\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandDataIngested\",\"group\":\"OnDemand\"},\"sku\":\"826E7FJQWVKN3CBH\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"826E7FJQWVKN3CBH.JRTCKXETXF\":{\"priceDimensions\":{\"826E7FJQWVKN3CBH.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB\",\"endRange\":\"Inf\",\"description\":\"0.08 per GB of data written\",\"appliesTo\":[],\"rateCode\":\"826E7FJQWVKN3CBH.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0800000000\"}}},\"sku\":\"826E7FJQWVKN3CBH\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"maximumExtendedStorage\":\"7 days\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of extended shard hours\",\"usagetype\":\"Extended-ShardHour\",\"standardStorageRetentionIncluded\":\"1 day\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"shardHourStorage\",\"group\":\"Addon shard hour\"},\"sku\":\"9DT967X6YQX3AYMB\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"9DT967X6YQX3AYMB.JRTCKXETXF\":{\"priceDimensions\":{\"9DT967X6YQX3AYMB.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"ShardHour\",\"endRange\":\"Inf\",\"description\":\"$0.020 per extended shard-hour\",\"appliesTo\":[],\"rateCode\":\"9DT967X6YQX3AYMB.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0200000000\"}}},\"sku\":\"9DT967X6YQX3AYMB\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"maximumExtendedStorage\":\"365 Days\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of GB-month long-term data storage\",\"usagetype\":\"LongTermRetention-ByteHrs\",\"standardStorageRetentionIncluded\":\"1 day\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"LongTermRetentionByteHrs\",\"group\":\"Long-term Data Retention GB-month\"},\"sku\":\"ECQFCZZ6DTWE83YB\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"ECQFCZZ6DTWE83YB.JRTCKXETXF\":{\"priceDimensions\":{\"ECQFCZZ6DTWE83YB.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB-month\",\"endRange\":\"Inf\",\"description\":\"$0.023 per GB-month for Long-term data storage (data older than 7 days) in US East (N. Virginia)\",\"appliesTo\":[],\"rateCode\":\"ECQFCZZ6DTWE83YB.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0230000000\"}}},\"sku\":\"ECQFCZZ6DTWE83YB\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"maximumExtendedStorage\":\"365 Days\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of GB long-term data retrieval\",\"usagetype\":\"LongTermRetention-ReadBytes\",\"standardStorageRetentionIncluded\":\"1 day\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"LongTermRetentionDataRetrieval\",\"group\":\"Long-term Data Retrieval GB\"},\"sku\":\"GXPWS67USXVJ3QQB\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"GXPWS67USXVJ3QQB.JRTCKXETXF\":{\"priceDimensions\":{\"GXPWS67USXVJ3QQB.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB\",\"endRange\":\"Inf\",\"description\":\"$0.021 per GB for Long-term data retrieval (data older than 7 days) in US East (N. Virginia)\",\"appliesTo\":[],\"rateCode\":\"GXPWS67USXVJ3QQB.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0210000000\"}}},\"sku\":\"GXPWS67USXVJ3QQB\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of payload units generated by producers\",\"usagetype\":\"PutRequestPayloadUnits\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"PutRequest\",\"group\":\"Payload Units\"},\"sku\":\"MAFXKQJ2YAQM6UDY\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"MAFXKQJ2YAQM6UDY.JRTCKXETXF\":{\"priceDimensions\":{\"MAFXKQJ2YAQM6UDY.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"PutRequest\",\"endRange\":\"Inf\",\"description\":\"$0.014 per 1 million payload units\",\"appliesTo\":[],\"rateCode\":\"MAFXKQJ2YAQM6UDY.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0000000140\"}}},\"sku\":\"MAFXKQJ2YAQM6UDY\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-BilledOutgoingBytes\",\"description\":\"Per GB of data retrieved using GetRecords consumers\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandDataRetrieval\",\"group\":\"OnDemand\"},\"sku\":\"PNHYMNK9TM59MTU8\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"PNHYMNK9TM59MTU8.JRTCKXETXF\":{\"priceDimensions\":{\"PNHYMNK9TM59MTU8.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB\",\"endRange\":\"Inf\",\"description\":\"0.04 per GB of data read using GetRecords Consumer\",\"appliesTo\":[],\"rateCode\":\"PNHYMNK9TM59MTU8.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0400000000\"}}},\"sku\":\"PNHYMNK9TM59MTU8\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"groupDescription\":\"Number of enhanced fan-out consumer-shard hours generated by consumers\",\"usagetype\":\"EnhancedFanoutHour\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"ConsumerHour\",\"group\":\"Enhanced fan-out consumer-shard hour\"},\"sku\":\"QFHADPJ42UJZ4EUK\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"QFHADPJ42UJZ4EUK.JRTCKXETXF\":{\"priceDimensions\":{\"QFHADPJ42UJZ4EUK.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"ConsumerShardHour\",\"endRange\":\"Inf\",\"description\":\"$0.015 per enhanced fan-out consumer-shard hour\",\"appliesTo\":[],\"rateCode\":\"QFHADPJ42UJZ4EUK.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0150000000\"}}},\"sku\":\"QFHADPJ42UJZ4EUK\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-LongTermRetention-ByteHrs\",\"description\":\"per byte hours data is stored beyond 7 days\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandLongTermRetentionByteHrs\",\"group\":\"OnDemand\"},\"sku\":\"QSF9ZM5MQMT76T99\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"QSF9ZM5MQMT76T99.JRTCKXETXF\":{\"priceDimensions\":{\"QSF9ZM5MQMT76T99.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"GB-month\",\"endRange\":\"Inf\",\"description\":\"0.023 per GB-month for retention beyond 7 days\",\"appliesTo\":[],\"rateCode\":\"QSF9ZM5MQMT76T99.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0230000000\"}}},\"sku\":\"QSF9ZM5MQMT76T99\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}",
        "{\"product\":{\"productFamily\":\"Kinesis Streams\",\"attributes\":{\"regionCode\":\"us-east-1\",\"servicecode\":\"AmazonKinesis\",\"usagetype\":\"OnDemand-StreamHour\",\"description\":\"per stream hour\",\"locationType\":\"AWS Region\",\"location\":\"US East (N. Virginia)\",\"servicename\":\"Amazon Kinesis\",\"operation\":\"OnDemandStreamHr\",\"group\":\"OnDemand\"},\"sku\":\"S69RFKAH58MYJ6TF\"},\"serviceCode\":\"AmazonKinesis\",\"terms\":{\"OnDemand\":{\"S69RFKAH58MYJ6TF.JRTCKXETXF\":{\"priceDimensions\":{\"S69RFKAH58MYJ6TF.JRTCKXETXF.6YS6EN2CT7\":{\"unit\":\"StreamHr\",\"endRange\":\"Inf\",\"description\":\"0.04 per Stream Hour Used\",\"appliesTo\":[],\"rateCode\":\"S69RFKAH58MYJ6TF.JRTCKXETXF.6YS6EN2CT7\",\"beginRange\":\"0\",\"pricePerUnit\":{\"USD\":\"0.0400000000\"}}},\"sku\":\"S69RFKAH58MYJ6TF\",\"effectiveDate\":\"2023-06-01T00:00:00Z\",\"offerTermCode\":\"JRTCKXETXF\",\"termAttributes\":{}}}},\"version\":\"20230801012600\",\"publicationDate\":\"2023-08-01T01:26:00Z\"}"
    ],
    "FormatVersion": "aws_v1"
}