Instrument batches in each_message?

[michaelsauter]

We use the process_batch.consumer instrumentation to monitor consumer offset health (e.g. growing offset lag, constant offset lag, very large offset lag, and offset progression). We also have some consumers that use each_message, but for those I'm not sure how to monitor offset lag.

There is an offset lag in process_message.consumer but I don't want to call monitoring for each message, plus it is tricky to monitor offset progression (meaning when I receive a batch in one partition its offsets should start after the previous batch from that partition).

What do you think about instrumenting the batch fetch for each_message as well?

[dasch]

I'd be open to that.

[michaelsauter]

Cool. What name doe you suggest for the instrumentation? Seems weird to use process_batch.consumer as we are processing messages (even though I would instrument exactly the same as process_batch.consumer.

I'll come up with a PR then.

[dasch]

One option would be to instrument the offset commits rather than the processing itself...

Otherwise, perhaps it's better to name this fetch_batch.consumer, since that's really what you're aiming for. I don't think you need to capture timing (batch fetches are multiplexed anyway) so perhaps just instrument("fetch_batch.consumer", ...).

[michaelsauter]

Okay, I'll go with that and see how that works out.

[michaelsauter]

Merged via #501.