Issue with virtual threads when total concurrency > 256

[andrewscode]

In what version(s) of Spring for Apache Kafka are you seeing this issue?

3.1.1

Describe the bug

Spring fails to start when the total number partitions / concurrency > 256 and virtual threads are enabled.

To Reproduce

Attached is a reproducer.

1. Run the docker compose file first.
2. Visit localhost:9000
3. Create a topic called "example-topic" with 300 partitions
4. Run the TestKafka.brokenStartup test

The test will never succeed.

If you kill the test and run it again, you'll see the following:

2024-02-02T07:00:14.544-05:00  INFO 32392 --- [      kafka-256] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-exampl-256, groupId=exampl] Request joining group due to: need to re-join with the given member-id: consumer-exampl-256-a32b6d30-4d37-4ab1-9d93-bdad18f834a5
2024-02-02T07:00:14.544-05:00  INFO 32392 --- [      kafka-256] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-exampl-256, groupId=exampl] Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberIdRequiredException)
2024-02-02T07:00:14.544-05:00  INFO 32392 --- [      kafka-256] o.a.k.c.c.internals.ConsumerCoordinator  : [Consumer clientId=consumer-exampl-256, groupId=exampl] (Re-)joining group
2024-02-02T07:00:44.548-05:00 ERROR 32392 --- [           main] o.s.k.l.KafkaMessageListenerContainer    : Consumer thread failed to start - does the configured task executor have enough threads to support all containers and concurrency?

The thread id gets stuck at kafka-256 which is the default limit of platform threads for virtual threads.

Expected behavior

The Test should finish / succeed.

Sample

reproducer.zip

[artembilan]

So, don't do that concurrency = "300" if there is indeed a limit for virtual threads number.

Or what do you propose supposed to be as a fix on the Framework side?

Thanks

[andrewscode]

With Virtual threads enabled, concurrency shouldn't be limited to some value. The whole point of virtual threads is you can have millions. But it's not the # of virtual threads that's the problem. Spring Kafka must be using something (synchronized?) that's causing it's consumers to be left on platform threads where there is a limit of 256

Or what do you propose supposed to be as a fix on the Framework side?

Well it shouldn't block the application from starting.

[artembilan]

Yeah... I think the problem is really in the Kafka Client by itself.
I see too many synchronized in their.
So, probably it is safe to assume that Kafka Client is not ready for virtual threads.
Just wild guess because it is not what blocking those threads in the Spring code.

[andrewscode]

If that's the case, does it make sense to use virtual threads at all for spring-kafka even if the spring property (spring.threads.virtual.enabled) is set?

[artembilan]

Agreed, it might not, but we need to make strong confirmation that the problem is there even without Spring Kafka.
What I meant before that we did all the changes in our code to support virtual threads.
The Kafka Client is out of our scope, so, there has to be a separate discussion exactly with Confluent.
And then we need to provide a proof with plain Kafka Client configuration where virtual threads are used and no Spring at all.

[artembilan]

Duplicate of #3074