support all node types

[sholzhauer-es]

All nodes that carry a cost should be accounted for in the end. Need investigation.

We want to provide a cost breakdown per data stream accounting for the cost of the data nodes but also provide insight into how much the other nodes/data transfers is costing per cluster. E.g an ML job is costing x ECU

[sholzhauer-es]

Started work on this.

@JohannesMahne I am taking the billing_cluster_cost as starting point. It will allow us to take both "node type" and "meta" cost down the line.

TLDR: instead of summing deployment cost and then dividing over the storage tiers I am doing the split first then start adding back up where needed.

At least that is my thought process 😉

[sholzhauer-es]

Figured our cost types, updating the billing transform to take the sku into the group_by allows the ES|QL to use that:

FROM billing_cluster_cost_nodes_lookup
| grok sku "%{WORD}.es.%{WORD:es_node_type}"
| GROK sku "%{WORD}.%{WORD:non_es_node_type}"
| eval cost_type = CASE(
    es_node_type is not null, es_node_type,
    non_es_node_type  
)
| eval cost_type = CASE(
    cost_type == "data", "data-transfer",
    cost_type == "datahot", "datahot/datacontent",
    cost_type
)
| DROP es_node_type, non_es_node_type

Azure:

AWS:

GCP:

[sholzhauer-es]

Status update.

We are making progress on this one. We are splitting the dashboard into two

• overview
• datatiers

The overview shows cost per sku type (node roles, snapshots and data transfer) based on the conversation rate.
The datatiers will show cost per memory and storage, as well as how "well" you are utilizing both of these metrics.

TODO

• Use the cost of memory and storage, with utilization score to calculate the realized cost.
• Take the realized cost together with the query time and indexing time per datastream to calculate % of cost per datastream
• take realized cost and store size of datastreams to calculate % of cost per stream
• Take the realized cost together with the query time and indexing time per tier to calculate % of cost per tier
• take realized cost and store size of tiers to calculate % of cost per tier

Screenshots