Proposal: limit the amount of information stored on dropped targets

[bboreham]

Proposal

As described at #12301, SD may use a lot of memory recording all the labels of any target dropped by rules.
Suppose you have 100,000 Kubernetes pods and have a rule to keep only 2 of them, then Prometheus will store 99,998 sets of information, including all the Kubernetes labels, annotations and other metadata available to relabeling rules.

As far as I can see, the only place this information is used is in rendering the /service-discovery page.
However, your browser will crash before it finishes rendering 100,000 targets, so this is of limited use.

My proposal is to change SD to record a limited number of dropped targets, say 20.
Change the UI to add "..." to show that more have been dropped but no details saved.

Maybe, add more advanced tooling to help admins troubleshoot their rules.

[liguozhong]

/service-discovery should be a switch flag, which is turned off by default and turned on by the user in the test environment, so that this problem can be solved. I have never used /service-discovery, I usually only need the /targets page for work.

but limit the amount of information stored on active targets still a great idea.

[bwplotka]

I would love that 👍🏽