-
Notifications
You must be signed in to change notification settings - Fork 6
Description
MetricsProcessor provides couple of important methods.
merge_metrics: We get separate metrics for CPU, GPU and memory usage. This method can combine those metrics into one. Additionally, this method can combine multiple time-series as well, which is how we are able to stitch together metrics collected over several days into one giant metric list.
condense_metrics: It takes the merged metrics list,, and then goes by each timestamp to calculate the start and end time of a pod and gives us a dictionary from which we can bill from.
The problem we may run into is that, when we load metrics from 30 days, merge_metrics will create a massive dictionary in-memory before the condensing happens. And we plan to increase the frequency at which we collect metrics, which means we'll have up to 15 more data to process.
My proposed solution is that if we merge and condense metric one file at a time then we don't build a giant dictionary in memory.
Alternative solutions:
- Pre-process the metrics as we collect them i.e. merge and condense every day metrics. This will save us storage space as well. However, we will need to write a method to merge this new condensed metrics from multiple days.
- Instead of just collecting daily metrics, we immediately condense the metrics, and produce a report. And then at the end of the month we will combine all those little reports to produce the monthly report.