Skip to content

Combine merge_metrics and condense_metrics into one #152

@naved001

Description

@naved001

MetricsProcessor provides couple of important methods.

merge_metrics: We get separate metrics for CPU, GPU and memory usage. This method can combine those metrics into one. Additionally, this method can combine multiple time-series as well, which is how we are able to stitch together metrics collected over several days into one giant metric list.

condense_metrics: It takes the merged metrics list,, and then goes by each timestamp to calculate the start and end time of a pod and gives us a dictionary from which we can bill from.

The problem we may run into is that, when we load metrics from 30 days, merge_metrics will create a massive dictionary in-memory before the condensing happens. And we plan to increase the frequency at which we collect metrics, which means we'll have up to 15 more data to process.

My proposed solution is that if we merge and condense metric one file at a time then we don't build a giant dictionary in memory.

Alternative solutions:

  1. Pre-process the metrics as we collect them i.e. merge and condense every day metrics. This will save us storage space as well. However, we will need to write a method to merge this new condensed metrics from multiple days.
  2. Instead of just collecting daily metrics, we immediately condense the metrics, and produce a report. And then at the end of the month we will combine all those little reports to produce the monthly report.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions