Way to configure Consumer to match Producer throughput

[CROprogrammer]

Hi,

I have a case in which I want to send as many messages from Producer to Topic in a short period, and to read messages from that topic in Consumer. I can achive high throughput in Producer and number of messages I've defined are sent, but Consumer that runs parallel to Producer skips a lot of messages from topic or it does not consume them.

How can I achive that my consumer does not skip messages and that it consumes all messages produced in producer?

When I use Constant Timer then Consumer can process message, but then too few messages can be send to Topic in short period of time.

My test plans:
Test Plans.zip

{
  "type": "record",
  "namespace": "com.example",
  "name": "Student",
  "fields": [
    { "name": "name", "type": "string" },
    { "name": "age", "type": "int", "default" : -1 }
  ]
}

[CROprogrammer]

Does anyone know why this could be?

[CROprogrammer]

I've noticed that lag on Kafka Topic is high when sending lots of messages in short period of time, and Consumer at one point stops consuming messages and then a lot of them stay unconsumed and Consumer shuts down. How can Consumer be configured so that no messages become unconsumed?

I would like to send for example a million messages in short period of time, and I would expect for it to be Produced and Consumed in a few seconds.

[jemacineiras]

Hi, we need to check this behaviour, Consumer is designed to wait until receive one single message and then deliver to JMeter. An idea we have in mind is to allow sort of "number of messages to consume" so we will consuming messages until this value is reached.
About the lag in Kafka topics... you mean when you produce messages? What kind of setup are you using to run the tests?

[CROprogrammer]

Even when there is no lag - assuming that all messages have been consumed, they do not appear in JMeter. I've tried to send 10 000 messages with 1 Producer thread, and cosuming it with 10 Consumer threads, but only 24 messages appeared in JMeter.

[CROprogrammer]

[jemacineiras]

That is , can you please open an issue so I can check it and see If I should reimplement the consumer?

[CROprogrammer]

#429

[jemacineiras]

I will be on it asap.