Monitoring

[HonzaMac]

We would like to start monitor of circuit breakers and retry's and timeouts in production.

I like this library the most from others but I am missing a way how to monitor current state.

I've tried to get at least breaker.stats and some onRetry and other, but would be better to have more holistic way how to get this events and state out of cockatiel.

Is there a way how to get this statistics?

i.e. here is some inspiration in Levee project.

Example how such output could one day look.
Article from Hystrix with video.

This image actually aggregates all instances of circuit breaker per concrete service.

I can image taking this information with every request (or batching can be done by another tool).
I am thinking now about this metrics:

• time to process request
• state - open, closed, half-open
• end state of request - timedout/sucess/failed
• 1,2,3 retry of request

As a side note: I want to integrate it into Grafana using InfluxDb (but this is not important)
Example of Grafana dashboard

[HonzaMac]

What do you think @connor4312 ?

[connor4312]

Apologies, been busy. I will look into this more deeply this weekend.

[connor4312]

The circuit breaker now includes a general onStateChange callback, and all policies include onSuccess and onFailure events, which are called with information about the function run duration. I think this should be sufficient to get the monitoring you're after.

Levee actually includes a basic statistics store, but I think this is out of scope for the library--there are numerous, differently configurable ways to collect call duration, for instance, which can be handled better by a specific telemetry library.

[HonzaMac]

Thank you @connor4312 for this improvement 👏. This will enable monitoring what I am looking for, will try it in upcoming week.

Great work 💪🏼