Add a gauge for "number of open chunks".

[discordianfish]

We have a metric for rushed mode, but non for throttled ingestion which is at least as important.

Additionally to that, it might be also useful to add a metric about the currently active timeseries which, I think, is different from prometheus_local_storage_memory_series because that shows how many Prometheus could fit into memory, not necessary how many would be active if it could fit more.
Or, this doesn't make sense but then the documentation could be a bit cleaner in that regard.

[beorn7]

• "Active timeseries" is somewhat vague and therefore cannot have a metric. (What would be the threshold of not considering a time series active anymore? See also prometheus/docs#442 – my proposed definition there would be exact but it would be excessively expensive to track.) What could be helpful to answer your question might be the number of open head chunks. These are chunks that are still mutable and cannot be persisted and thus cannot be evicted. Would that satisfy your use case?
• There cannot be an on/off metric for throttled ingestion because it is not a state (like rushed mode). Each time a scrape or rule evaluation starts, the storage layer is asked if it is OK to do so. Essentially, the answer is "no" if the (unbumped) persistence urgency score would be 1. As an approximation, you could use prometheus_local_storage_persistence_urgency_score, but note that it is calculated at different times and then obviously sampled by scraping in intervals, but that's the best answer you can get: You are "in throttled mode" (which does not exist) if the urgency score is 1.
• WRT to documentation: I have troubles going into all the detail in the docs because the sheer amount of text will render it useless for quick troubleshooting. I'm happy to review PRs that give it a try, see also prometheus/docs#442 as mentioned above (where the original author hasn't come back in a while).

[discordianfish]

What could be helpful to answer your question might be the number of open head chunks. These are chunks that are still mutable and cannot be persisted and thus cannot be evicted. Would that satisfy your use case?

Yes, that sounds exactly what I'm looking for and would be super helpful.

Each time a scrape or rule evaluation starts, the storage layer is asked if it is OK to do so. Essentially, the answer is "no" if the (unbumped) persistence urgency score would be 1.

What do you think about an 'error' counter which would be incremented if the storage layer answers "no"? But I'm good with using prometheus_local_storage_persistence_urgency_score.

WRT to documentation..

Let's discuss that over in prometheus/docs#442, looks like this is about the same confusion.

[beorn7]

A "throttling_needed_total" counter sounds like a good idea. Easy to implement, and easy to reason with. Can alert nicely on increase.

Title adjusted.

[juliusv]

@beorn7 Both cases already have separate counters for the throttled cases:

prometheus/rules/manager.go

Line 149 in 0938661

iterationsSkipped.Inc()

prometheus/retrieval/scrape.go

Line 430 in b4660a5

targetSkippedScrapes.WithLabelValues(interval.String()).Inc()

Can this issue be considered done?

[beorn7]

You are right about the counts for "throttling needed". But we still need a Gauge for "open chunks". That's seems to me a really useful metric to have.

[ghost]

@beorn7 @juliusv @discordianfish Would this be a good first issue to take a stab at? Or is there something else that'd be better suited?

[beorn7]

@copyconstructor A metric for open chunks sounds like a really good starting point: Note that the chunking code was just moved to its own package, which made the metrics collection a bit more complicated. Just have a look how it is done right now, and feel free to ask me or @juliusv if you have questions.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.