How to span multiple consumers from single server instance

[naresh417]

I have topic with T with 20 partitions but I have only one sever to deploy my consumer . I.e. single consumer in my consumer group. To overcome this I created some 5 threads and calling the consume method thinking that it’s going to read from 5 different partitions assuming each thread as consumer . However my 5 threads are reading from one partition and why so ? . How the librdkafka determine single consumer vs multiple consumer . How can I achieve the parallelism without increasing my infrastructure?
by the way I didn’t change my consumer portion assignment I.e by default it’s range,round robin . Is it anything related to this partition assignment . Some one please look into this and provide some solution ?

[edenhill]

https://github.com/edenhill/librdkafka/wiki/FAQ#how-are-partitions-fetched

I recommend you create 5 separate consumer instances, one for each processing thread, and have them use the same group.id.

[naresh417]

Hi Edenhill thank you very much for your quick response . As you said In each processing thread am creating instance already with same groupid . My bad, instance in the discussion thread heading " How to span multiple consumers from single instance", i actually meant it's single server instance not the single object instance of consumer . Also i observed in the code we are caching the consumer instance for 60 seconds to avoid to create multiple consumer objects, is it causing the issue ? . Also with the default range consumer partition setting, when there is consumer lag and producing messages to the partitions only consumer reads from few partitions always it never picked other partitions . Thanks In advance