OSB unable to consume message from kafka with multiple host

[coemee]

Hi,

Help needed.
OSB 12.1.3 and Kafka_2.11-0.8.2.1

On test-env which has 1 Kafka host. OSB proxy is able to consume the message.

But on pre-prod env which has 3 Kafka host, OSB is unable to consume the message.
And from Kafka server can see many "Connection reset by peer" errors, and can see that OSB managed to subscribe to Kafka successfully BUT number of group consumer ids is keep growing.

Anyone can help to advise, what could the problem and how to resolve it?

Following is the dump thread stacks error:
"Keep-Alive-Timer" Id=553 TIMED_WAITING
at java.lang.Thread.sleep(Native Method)
at sun.net.www.http.KeepAliveCache.run(KeepAliveCache.java:172)
at java.lang.Thread.run(Thread.java:745)
"[STANDBY] ExecuteThread: '32' for queue: 'weblogic.kernel.Default (self-tuning)'" Id=384 WAITING on weblogic.work.ExecuteThread@6f537875
at java.lang.Object.wait(Native Method)
- waiting on weblogic.work.ExecuteThread@6f537875
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:281)
"[STANDBY] ExecuteThread: '31' for queue: 'weblogic.kernel.Default (self-tuning)'" Id=363 WAITING on weblogic.work.ExecuteThread@76fff562
at java.lang.Object.wait(Native Method)
- waiting on weblogic.work.ExecuteThread@76fff562
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:281)
"[STANDBY] ExecuteThread: '30' for queue: 'weblogic.kernel.Default (self-tuning)'" Id=352 WAITING on weblogic.work.ExecuteThread@2b5b9ecf
at java.lang.Object.wait(Native Method)
- waiting on weblogic.work.ExecuteThread@2b5b9ecf
at java.lang.Object.wait(Object.java:503)
at weblogic.work.ExecuteThread.waitForRequest(ExecuteThread.java:247)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:281)
"pool-16-thread-5" Id=346 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33e7d9e9
at sun.misc.Unsafe.park(Native Method)
- waiting on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33e7d9e9
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1085)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
"pool-16-thread-4" Id=335 WAITING on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@33e7d9e9
at sun.misc.Unsafe.park(Native Method)