Skip to content

Conversation

@ucarion
Copy link
Contributor

@ucarion ucarion commented Jul 7, 2015

I am trying to collect data about how long I'm waiting on Kafka; I want to answer questions like "how much latency is there for getting messages out of Kafka?". To this end, I've put together this PR, which adds an option to enable metrics collection in KafkaClient.

This uses scales, and would let me write stuff like this:

from kafka import KafkaConsumer

consumer = KafkaConsumer('foobar',
                         bootstrap_servers=['localhost:9092'],
                         group_id='kafka_test',
                         enable_metrics=True)

# interesting logic would go here ...
for _ in xrange(10):
    for message in consumer.fetch_messages():
        process(message)

# outputs the 95th percentile of how long it takes to perform send_fetch_request
print consumer.metrics.fetch_request_timer['95percentile']

I've only added support for metrics in KafkaConsumer, but I can add other consumers if you agree with this design.

kafka/client.py Outdated
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check out functools.wraps.

@ecanzonieri
Copy link
Contributor

It may also be useful to get errors count as part of these metrics

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the upstream client configuration here is metric.reporters. I try to stay consistent, so perhaps metric_reporters and support a list of reporter interfaces?

@dpkp
Copy link
Owner

dpkp commented Dec 2, 2015

Metrics support would be great -- if you have time, take a look at my comments re: consistency w/ upstream client approach. Otherwise I'll probably put this off for a bit and clean up w/ a larger 0.9 client refactor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants