Delay in consumer after message is send using simple producer

[attodorov]

Hi

I'm sending a message using this python code:

client = KafkaClient("localhost:9092")
producer = SimpleProducer(client)
producer.send_messages("mytopic", bytes("some msg", 'utf-8'))

Then i am consuming it using kafka-net in the following way:

        var options = new KafkaOptions(new Uri("http://10.0.1.106:9092"), new Uri("http://10.0.1.106:9092"));
        var router = new BrokerRouter(options);
        ConsumerOptions opts = new ConsumerOptions("mytopic", router);
        var consumer = new Consumer(opts);

foreach (var message in consumer.Consume())
{
// do something with the message
}

My problem is that once a message is sent, it takes between 1-2 up to 10 seconds until it is consumed in jafka-net. Is this normal? I'm using default Kafka /zookeeper configs.

My expectation is that messages are received immediatelly.

Thanks
Angel

[pcajamie]

I've had similar problems - the best I've managed so far is to change DefaultMaxBlockingWaitTime in the FetchRequest class from 5000ms right down to 100ms, that definitely helps but it's not instant. I'm sure there must be some other settings which I'm missing.

[Jroland]

The consumer is supposed to be connecting to the brokers and using a polling query that is waiting for a particular amount of data to show up in the topic before returning the results. This might not be set with the best default. Ill have a look at the consumer tomorrow. There are a few bug fixes in the Tcp connection handling that I have queued that should go out which might help.

[Jroland]

I think the two settings you are looking for are this (unfortunately I have not made these readily available to change in the code). Ill add that.

    public int MaxWaitTime = DefaultMaxBlockingWaitTime;

The max wait time is the maximum amount of time in milliseconds to block waiting if insufficient data is available at the time the request is issued.

    public int MinBytes = DefaultMinBlockingByteBufferSize;

This is the minimum number of bytes of messages that must be available to give a response. If the client sets this to 0 the server will always respond immediately, however if there is no new data since their last request they will just get back empty message sets. If this is set to 1, the server will respond as soon as at least one partition has at least 1 byte of data or the specified timeout occurs. By setting higher values in combination with the timeout the consumer can tune for throughput and trade a little additional latency for reading only large chunks of data (e.g. setting MaxWaitTime to 100 ms and setting MinBytes to 64k would allow the server to wait up to 100ms to try to accumulate 64k of data before responding).

[Jroland]

You can now tune the values. Grab the latest nuget.
To get the minimum delay, use MinimumBytes = 1 in the ConsumerOptions class.

[attodorov]

Great, thanks !

[pcajamie]

Perfect - I'll give it a try!