Feature idea: Lazy scrapes

[RichiH]

This is more a discussion that an issue, but I didn't find a better place to toss this.

Basically, lazy scrapes would scrape a target every X time, but only record into the time series if

• the value changed
• timeout Y was reached
• Prometheus shuts down gracefully.

As a specific example, I will scrape UPS systems quite aggressively to have near-instant alarms when power fails or fluctuates. Yet, in a normal year, I will have between one or two incidents at max.
Given a timeout of, say, four minutes, would allow me to scrape every five seconds, but only record every four minutes.
While I know that Prometheus already compresses static values quite efficiently, this scheme would allow another gain of up to 1:300.

[fabxc]

I understand why that idea seems attractive for your use case, yet I think implementing it this way leads to a great deal of complexity through all layers.
Currently, our staleness handling will get in the way anyway.

How about running two Prometheus servers? One scrapes slowly, the other one in 1s intervals. The fast one alerts but has a very low retention, while the slower one keeps data for longer.
Alternatively, the slower can also federate from the faster one.

[RichiH]

I can't judge the implementation cost and complexity, so I trust your call on this.

Having two parallel servers, or scraping one with the other, means I will lose blips, though. If not for that, I would already be doing so.

This is similar to what I meant with the exporting to long-term storage: Sometimes, you will need averages, sometimes an aggregate, sometimes min or max, and sometimes you need to catch blips.

Matter of fact, typing the above I just realized that the idea of lazy scrapes is motivated by the lack of well-defined degradation of data and/or its granularity in the long term.

[beorn7]

With the current encoding, a non-changing value takes 1 byte of space. With planned changes, it might boil down to 1 bit or even less. So there is really not much to gain by all those fundamental changes and complexities @fabxc mentioned.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.