Calculating lag and consumption metrics

[danthegoodman1]

I see that recently #1275 was merged in, and was wondering if there are (or plan to be) suggestions on how to monitor consumer/producer behavior in redpanda?

I would really want to monitor things like consumer lag (per topic, or per partition, time would be ideal), consume rate, and produce rate. Going through the metrics currently available on docker do not seem to provide these metrics (at least not from cmd+f for some common terms from the metrics endpoint output).

Edit:
Thinking about this I guess it is possible to use custom instrumetation to get produce and consume rates for a topic, but that requires the capability to do so - some tools using a Kafka api might not be custom made.

[emaxerrno]

@danthegoodman1 - totally, we need to update our prom query+grafana json/dashboard. Good eye on catching #1275

you are welcome to play around w/ it atm, currently is not yet on the release notes, so not ready for wide consumption, but close!

we have to make it easy like you mentioned. if you don't want to play around w/ it, feel free to use kminion project to monitor lag.

[emaxerrno]

cc: @coral-waters

there will be :) docs is a priority for us

also try rpk for showing in your terminal the consumer lag.

[danthegoodman1]

👏 Yeah rpk is great for quick checks but being able to monitor that in Grafana dashboards and setup alerts is the goal 😄

thanks for all the help!

[coral-waters]

@danthegoodman1 - Thanks for the feedback! We're working on getting the doc up to speed and we definitely want to know what info will be helpful to users. I created an issue to make sure we get this documented - #3430

[danthegoodman1]

Oh great, then I don’t have to feel bad about being annoying with all these discussion topics :)

[coral-waters]

Haha. Not at all. Please ping me for anything that should be in the doc. We're still filling out the doc team so it might take a minute to get up on the website, but it's really helpful to know what kinds of things people are looking for. It will eventually get there! :)