Implement Stateless Ruler Proposal

[bwplotka]

Tracking ticket for implementing https://github.com/thanos-io/thanos/blob/master/docs/proposals/202005_scalable-rule-storage.md#proposal

Open Question about this proposal: Is push required by ruler really remote-write? Remote write is for batches vs here we kind of write samples, no? More of pushgateway API etc 🤔 cc @brancz

[bwplotka]

Ongoing work: #3743

[metalmatze]

The way this is implemented in my current PR is that the samples are added to a list of series and they are only sent in a batch once commit is called. So it's not that every series is sent on its own.
Not sure if that's what you're asking for or how we want to have this done in the end.

[brancz]

I don't see how this is more like the pushgateway API. I agree this should be remote-write, as @metalmatze reiterated. (not necessarily a good argument but Cortex does the same)

[jaybatra26]

Hi! Can I take this up as a part of LFX programme?

[bwplotka]

Anyone can help! (: By for LFX program we have @idoqo this spring. Still there is a lot of work so anyone can work on a piece.

[jaybatra26]

Thanks, @bwplotka I will then start with good first issues.

[idoqo]

After 1:1 discussions, it seems like the implementation for this is sort of a choice between full and partial statelessness.

Full stateless is based on Frederic’s comment (#3743 (comment)) and will have us writing to the remote storage as soon as they are evaluated. We could then record the last successful remote write and back-fill the failed ones (say, by ruler asking “when was the last time I successfully wrote data?”). It also means no persistence (be it local storage or a WAL for buffering data). This makes sense as we don’t know exactly how much data rule evaluations will produce and that makes it hard(er) to scale the WAL/local storage.

One issue with full statelessness though, is that we might end up spamming the remote-write storage with requests since we are sending the request for each sample that ruler produces.

The alternative would be partial statelessness where we send the remote-write requests in batches. To batch though, we do need some kind of WAL that persists the data for a while and flushes after they have been successfully sent to the remote storage.

[brancz]

Whether it's batches or not, what I meant by "record" is in fact a sort of persistence, but just a marker of where we left off, as opposed to the data itself. As I mentioned in my last comment on the issue linked, I don't think there will be one remote-write request per sample, but rather per rule evaluation.

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[stale]

Closing for now as promised, let us know if you need this to be reopened! 🤗

[stale]

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

[m-messiah]

Let's continue with it? I think remote-write from thanos ruler could be a great thing

[GiedriusS]

Implemented in #4731 👍 thank you all for your work!