High Churn from avalanche even though series-interval and metric-interval parameters are set high

[sharath-sri-chellappa]

Hi Team,

I wanted to understand why Avalanche produces such a high churn even though the series-interval and metric-interval is set high. The following are the parameters I have set in my kubernetes yaml which deploys 2 replicas of the avalanche:

`

• "--metric-count=20"
• "--series-count=100"
• "--label-count=10"
• "--port=9001"
• "--value-interval=1"
• "--series-interval=316000"
• "--metric-interval=316000"
  `

This basically ensures that there are 20 metrics with 100 series per metric. These exist without changing/cycling for 316000 seconds. What changes is the values for each of the metric-series combination which does so every second.

So essentially my understanding from this is that the churn should not be high as there is no creation of a new series every single scrape. as the series-metric combination remains the same, its just that the values change. Now what I seem to notice is that the churn is 2000 which means that every scrape has 2000 series added which is not what is the case.

I calculate the churn by querying Promethues with the following query -
topk(200, sum without(instance)(sum_over_time(scrape_series_added[1h])))
This will give a per-target churn value over the past hour, and aggregate it up ignoring the instance label and then find the biggest 200 churners
Reference Link- https://www.robustperception.io/finding-churning-targets-in-prometheus-with-scrape_series_added

My questions now are -

1. Why are the avalanche replicas creating this level of churn even though series and metrics are not created newly for every scrape. I really hope that this is not a limitation of Avalanche
2. What is the number of series per chunk in the head block in the case of Avalanche. I am not sure if that makes any difference because I seem to notice that on an average there is 1 chunk created for every 2 series (I have seen that for any other metrics generated by a non-avalanche load its about 1 chunk for every series). Not sure if this makes a difference to churn but I guess its going to make a difference to the samples/second of the remoteWrite to Thanos from Prometheus. Again not sure if this is the right forum to ask.

Please let me know if you need any further information from me on the same.