Dataflow cost metrics

[fpeter8]

Dataflow job detail page (https://console.cloud.google.com/dataflow?project=projectId -> go to any job), on the right hand panel, under "Resource metrics", add two new rows:

• Current cost: how much this job is costing us with the current machine number
• Total cost: how much it has already cost us since the beginning

Note that the page looks different for batch/streaming jobs (and possibli for SDK1/2 jobs as well)

[gsipos]

https://cloudpricingcalculator.appspot.com/static/data/pricelist.json

[fpeter8]

table is updated regularly for running jobs (you cannot calculate the cost once, you have to subscribe to the changes / update regularly)

[fpeter8]

json is obsolete, you have to use this api: https://cloud.google.com/billing/v1/how-tos/catalog-api

[fpeter8]

usage:

• list all services: https://cloudbilling.googleapis.com/v1/services?key=API_KEY
• Search for "Cloud Dataflow", note the service id
• list all prices for a service: https://cloudbilling.googleapis.com/v1/services/SERVICE_ID/skus?key=API_KEY

Relevant entries:

• "vCPU Time Batch"
• "vCPU Time Streaming"
• "Local Disk Time PD Standard"
• "Local Disk Time PD SSD"
• "RAM Time"

each region might have different prices, the API returns the regions in the "serviceRegions" field. Dataflow region is in the table below the "Resource metrics" table.

For "Current" fields the value is {price} * {current value} / {time period}
e.g. "Current vCPU" = 1
"vCPU Time Streaming" = 0.00054 USD / hour
calculated value: 0.00054 USD / hour

For "Total" fields the value is {price} * {total value}
e.g. "Total vCPU" = 5.14 vCPU h
"vCPU Time Streaming" = 0.00054 USD / hour
calculated value: 0,0027756 USD

For development, you can use API key 'AIzaSyBZVfVwDKpduSuNOJlvWildIeQ5AsNtnWM' (not sure if we should use this for prod, or get it from the user via config)

[gsipos]

✅ Currency code can be specified.
❓ pricingInfo array is a chronological ordering of price history.

https://cloud.google.com/billing/reference/rest/v1/services.skus/list

[gsipos]

The cost metrics should be displayed in a user preferred currency format.
The user should be able to define what is his preferreed currency, and should be able to define project specific currency format as well.

[csabakassai]

I think the project specific configuration is over-engineering. I believe a global GCPimp level configuration is more than enough

[fpeter8]

Description of the SKU api response: https://cloud.google.com/billing/reference/rest/v1/services.skus/list#PricingExpression