GH-2381 : Support kafka parallel-consumer

[chickenchickenlove]

Introduction

Hi, Spring Team.
This is chickenchickenlove, huge fan of you guys.
I saw this issue (Support kafka parallel-consumer), and this comment as well!

No plans currently; but contributions are welcome!

So, i'd like to implement that integrate Parallel Consumer to spring-kafka
This is skeleton code.

Please take a look when you have some free time. 🙇‍♂️
Thank you for your time!

Background

• Sometimes, Confluent Parallel consumer will be more helpful than kafka consumer, because parallel consumer have good parallel processing (If you want to know more, See this).
• Thus, i think it will be good option to spring-kafka as well.

How To use

@EnableKafka
@EnableParallelConsumer
@SpringBootApplication
public class SpringkafkasampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringkafkasampleApplication.class, args);
	}
}

• Annotated @EnableParallelConsumer to your Spring application. then, some spring bean related with parallel consumer, will be registered to ApplicationContext

@Component
public class MyTestClass implements ParallelConsumerCallback<String, String> {

    private final List<String> topics;
    public MyTestClass(List<String> topics) {
        this.topics = topics;
    }

    @Override
    public void accept(PollContext<String, String> context) {
        context.getConsumerRecordsFlattened().forEach(kvConsumerRecord -> System.out.println(kvConsumerRecord));
    }

    @Override
    public List<String> getTopics() {
        return this.topics;
    }
}

• and then, you should implement interface ParallelConsumerCallback and register their concrete class as spring bean by annotating @Component or @Configuration.
• then, ParallelConsumerFactory will start ParallelConsumer and ParallelConsumer process records with ParallelConsumerCallback concrete class.

ETC

• How about follow the way like @KafkaListener?
  • IMHO, it is unnecessary. because AFAIK, ParallelConsumer can allow only one callback. thus, it is not suitable for ParallelConsumer.
• Can we remove ParallelConsumerImportSelector?
  • Yes. if org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration.java class support to register spring bean, ParallelConsumerImportSelector can be removed.

Update from Apr 27.

• I added some interfaces for parallelConsumer.

Thank you for your time, Again.

[pivotal-cla]

@chickenchickenlove Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-cla]

@chickenchickenlove Thank you for signing the Contributor License Agreement!

[sobychacko]

@chickenchickenlove Thank you for the PR and the kind words. Next month, we will release the first RC version of 3.2.0. Since this is an entirely new feature and involves dependencies from io.confluent (as opposed to org.apache for the regular consumer), this needs a detailed review to come up with a plan for supporting parallel consumers. I think you are in the right direction, but we will have an internal discussion on this matter. Depending on how things go, we may have to defer this feature to the 3.3.x release, but we will let you know.

[chickenchickenlove]

Hello, @sobychacko !
Thanks for your time 🙇‍♂️

I see! however, that's sounds great. 👍
If after internal discussions, spring-kafka team decides to introduce ParallelConsumer to spring-kafka, and there is anything more to do, Please let me know. I'm more than willing to help.

Also, Please let me know in case of a decision not to introduce ParallelConsumer to spring-kafka!"

Thanks for your time, again 🙇‍♂️

[tagorenathv]

Any update on this please ?

[sobychacko]

@tagorenathv We will look at this after the GA for 3.2.0 (this month) as part of the next minor version.