Add an alerting engine

[jd]

There's no alerting engine in Gnocchi. It should offer an easy solution to trigger actions on e.g. threshold.

The Aodh project from OpenStack supports Gnocchi, but it does that mainly by polling the API regularly. It's pretty slow at the end of the day and OpenStack specific.

We might need to add some other features first, but this issue should be a placeholder to discuss a design.

[chungg]

random idea: if we mark a metric as 'has alert', when we aggregate, we would know this, and we can broadcast to an alert engine to evaluate rule? although this might be a bit aggressive and create a lot of extra rule evaluations at the beginning of new metric... not sure how we schedule triggers for alert evaluations either.

[jd]

@sileht has exactly the same idea. Since metricd loads all the data to do aggregation, it seems it's the best place to also do the alerting evaluation.

[sileht]

Ceph have an interresting API to monitor objects: http://docs.ceph.com/docs/master/rados/api/librados/#rados_watch3

But I have no idea how it scales.

[chungg]

watch seems interesting. i'm not sure what the notification it's sending is... if we read the object, will it notify as well?

it does create a watch object so it will bump up the storage size (not sure how much)

[lucasponce]

Hi @jd,

From Hawkular Alerting we are working in a 2.x of lightweight components that can be used separately and independently.
We have started to work on the Federated Alerting concept adding support to several sources like Prometheus, Elasticsearch, Kafka.
I have also thought about Gnocchi and I wonder if it could be interesting to add a plugin for it.
We have written a blog about it,
http://www.hawkular.org/blog/2017/08/alerts-multiple-sources.html
And a demo to see how it would look like
https://www.youtube.com/watch?v=mM1mwJneKO4

The aim is to complement a gap where today an integration between systems is mandatory.

I would be interested to know your feedback and if this could be useful for the Gnocchi ecosystem.

Thanks !

[chungg]

@lucasponce is hawkular polling prometheus/elasticsearch with the query defined in 'expression'?

[lucasponce]

@chungg Yes. Hawkular defines Triggers, which contains Conditions, on this objects Hawkular can read data from multiple sources, as Prometheus, Apache Kafka or Elasticsearch (or your own source of data using an API).
Also we can define several modes like polling or streaming and we can add value playing with more complex scenarios like:
http://www.hawkular.org/blog/2017/08/alerts-nelson-rules.html
http://www.hawkular.org/blog/2017/01/13/events-aggregation-extension.html
An intro of this component can be found here
https://www.slideshare.net/ponceballesteros/hawkular-alerting
(in the 2.x version we have eliminated the Cassandra dependency, and hawkular alerting is just a single process).
Please let us know what do you think.

[chungg]

i see. i imagine it'd behave similar to how Aodh interacts with Gnocchi currently (ie. create an alarm definition with query, run query periodically, trigger action, repeat).

i don't see having Hawkular support Gnocchi as a bad thing; a good thing actually, since it removes barriers to adoption. that said, i think the hope is that we don't need to define another polling interval and instead leverage the fact that when Gnocchi stores data, it already has time series loaded, and therefore it can check and trigger alarms as data is entered into Gnocchi rather than be dependent on a polling cycle. if we can do that, i still think we can connect to Hawkular by possibly 'streaming' the triggered alarm to Hawkular rather than have Hawkular poll Gnocchi.

[lucasponce]

Yes, the streaming approach was the primary integration for Hawkular Metrics as well.
When a metric is inserted in the system, it queries a cache to validate if a metrics has a trigger defined, then is sent into the alerting engine for evaluation.
Also, we introduced the dual 'polling' approach as there are also different scenarios where polling could be more interesting.
For example, as described in nelson rules defined, to detect advanced behaviours around a metric, a streaming polling could be good; but for other scenarios, like when you want to compare periods (average of today vs yesterday) perhaps a polling could be appropiate.
Well, the interesting about Hawkular Alerting is that the alerting engine was born in an independent way (it is not linked with Hawkular Metrics), so that let us to send data from different sources without being linked to any specific format of an implementation.
Auto-resolution of problems is also a nice feature where we think it adds additional value: detect when something went wrong (i.e. a bad response time of an application) and detect when it's back. So dealing with lifecycle states can help and enrich the model.
I agree that alerting on thresholds doesn't require a complex alerting engine (even this can be done outside with simple scripts polling regularly) but when collecting a high number of metrics complexity grows and then this type of engines can be very interesting.

Perhaps we can explore the streaming approach in Gnocchi, then perhaps a plugin would be necessary and Gnocchi could get the info related what are the metrics that have definitions and send the data into the alerting engine.

[chungg]

are there existing 'streaming' plugins? i'm not sure what plans the other contributors have for implementing this functionality. i don't have it queued up so not sure when we can move forward with this.

[lucasponce]

I think we can try two ways.
One that wouldn't need gnocchi development, where we can create a polling plugin in Hawkular Alerting (similar as we have with Elasticsearch/Kafka/Prometheus) where we can get data and process them in streaming. So it is not a pure "streaming" approach, but this perhaps could help to add a primary Gnocchi/Hawkular Alerting integration.
And a second approach, we could implement a similar pattern like we did in Hawkular Metrics, meaning that it should be in Gnocchi side to push data into Hawkular Alerting for evaluation filtering for those metrics that have triggers defined.
This last proposal would be a pure streaming, but it would need some kind of plugin in gnocchi side.
I am not familiar with the Gnocchi architecture, this kind of logic is something doable that can be implemented as something optional ?

[chungg]

seems like a sane approach to me 👍

[seuf]

influxdb has the same approach, you can create "subscribers" on time series, and each time a new value arrive for a serie, it is streamed to the subscriber (for example kapacitor for alerting).
Maybe the metricd should implement a streaming method to forward all the metrics to an other process which will chose to process or not the data.