-
Notifications
You must be signed in to change notification settings - Fork 792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow notifiers to receive metrics regardless of consumer group status #341
Comments
In Kubernetes, using the Custom Metrics Adapter for Prometheus means we can horizontally autoscale consumers on arbitrary metrics that we already collect with Prometheus, such as consumer lag using Burrow. I'm not sure if anyone has already done this, but I'm looking for examples, and this is the closest mention I've been able to find. |
Yeah, I'd like to instrument the app itself with Prometheus and it would
overlap with the http API likely, but also provide operational metrics.
…On Feb 8, 2018 6:23 PM, "Steve Coffman" ***@***.***> wrote:
In Kubernetes, using the Custom Metrics Adapter for Prometheus means we
can horizontally autoscale consumers on arbitrary metrics that we already
collect with Prometheus, such as consumer lag using Burrow. I'm not sure if
anyone has already done this, but I'm looking for examples, and this is the
closest mention I've been able to find.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#341 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AKgsUHQzYz7erswgc5YeQ1LEVqBtF-QKks5tS6wagaJpZM4RzGAl>
.
|
This seems like a bug instead of an enhancement. Notifier configuration has a 'threshold' property, which already indicates if StatusOK messages should be sent out. Shouldn't the threshold property be used in the evaluation request 'ShowAll' flag? IE: ShowAll = (threshold == StatusOK) |
We currently have a working fork where we implemented two additional notifiers: one to push metrics to AWS CloudWatch [1] and another one to push metrics to dogstatsd 2.
Long story short, here at Vend we have a containerised driven architecture leveraged on AWS ECS and we wanted a way to be able to autoscale our consumers based on consumer lag. AWS Application autoscaling does not work well with sparse metric, which means that we need to push every single metric to cloudwatch regardless of the consumer status. As of [2], we push all metrics to dogstatsd so we can plot nice consumer lag graphs and also to create some monitors that may eventually trigger pagerduty if a consumer lag goes over a defined threshold.
Having said that ^^, at the moment there is no elegant way to enable
ShowAll
per notifier. Since we don't care about any other notifier we simply addedShowAll: True
to https://github.com/linkedin/Burrow/blob/master/core/internal/notifier/coordinator.go#L375The aforementioned approach is not ideal should we wish to push those two custom notifiers to upstream and on top of that it makes merging off upstream a bit cumbersome sometimes.
Please advise on what the best approach would be to address the above scenario.
The text was updated successfully, but these errors were encountered: