Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once #64774
Conversation
alexweav
added
area/alerting
alexweav
requested review from
sakjur,
papagian and
suntala
and removed request for
a team
|
Backport is product-approved. |
|
Hello @armandgrillet!
Please, if the current pull request addresses a bug fix, label it with the |
alexweav
added
product-approved
alexweav
force-pushed
the
alexweav/fanout-backend
branch
from
ad0214d
to
0e59784
Compare
There was a problem hiding this comment.
LGTM.
I wonder if we should consider a situation when on write one of the inputs could behave slower than the other (this is also applicable to the single input case), and probably cancel the operation by a timeout. Currently, historian uses the same context as the rule evaluation routine but operates asynchronously, so requests are will be piling up.
| BackendTypeLoki BackendType = "loki" | ||
| BackendTypeNoop BackendType = "noop" | ||
| BackendTypeSQL BackendType = "sql" | ||
| ) |
There was a problem hiding this comment.
I thought you would put this enum to the settings package and will validate the mode when it parses settings which will make Grafana crash faster, during settings parsing. But I am ok with this too.
There was a problem hiding this comment.
Other consistency checks (are required keys for each backend provided? are urls valid? etc) also already happen in ngalert package. I added it there to keep all validation in one place.
I'm not opposed to moving the validation as a whole to config package, though! I'd just rather not split validation for now
|
This pull request was removed from the 9.4.6 milestone because 9.4.6 is currently being released. |
…o multiple other backends at once (#64774) * Rename RecordStatesAsync to Record * Rename QueryStates to Query * Implement fanout writes * Implement primary queries * Simplify error joining * Add test for query path * Add tests for writes and error propagation * Allow fanout backend to be configured * Touch up log messages and config validation * Consistent documentation for all backend structs * Parse and normalize backend names more consistently against an enum * Touch-ups to documentation * Improve clarity around multi-record blocking * Keep primary and secondaries more distinct * Rename fanout backend to multiple backend * Simplify config keys for multi backend mode (cherry picked from commit a31672f)
…patches to multiple other backends at once (#64983) Alerting: Create new state history "fanout" backend that dispatches to multiple other backends at once (#64774) * Rename RecordStatesAsync to Record * Rename QueryStates to Query * Implement fanout writes * Implement primary queries * Simplify error joining * Add test for query path * Add tests for writes and error propagation * Allow fanout backend to be configured * Touch up log messages and config validation * Consistent documentation for all backend structs * Parse and normalize backend names more consistently against an enum * Touch-ups to documentation * Improve clarity around multi-record blocking * Keep primary and secondaries more distinct * Rename fanout backend to multiple backend * Simplify config keys for multi backend mode (cherry picked from commit a31672f) Co-authored-by: Alexander Weaver <weaver.alex.d@gmail.com>
What is this feature?
This PR adds a new state history backend called
fanout, that composes multiple other backends.It allows you to write to multiple state history backends at once, and select one of them for reads. In this PR, the backend selected for read traffic is called the
primary, all others are calledsecondaries. Write traffic is dispatched to all configured backends.Why do we need this feature?
This makes it much easier for operators to enable the new Loki-based state history on an active system. They can move to
fanout, verify that Loki is operating correctly, then switch toloki.Which issue(s) does this PR fix?:
Fixes #64363
Fixes #64364
Special notes for your reviewer: