Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Multiple WebSocket threads getting created for every single client connection #10850

Open
Spandana806 opened this issue Nov 3, 2023 · 3 comments
Open

Multiple WebSocket threads getting created for every single client connection #10850

Spandana806 opened this issue Nov 3, 2023 · 3 comments
Labels
Question

Comments

@Spandana806
Copy link

10.0.13

Multiple websocket threads being instantiated for every WebSocketClient.start() call

WebSocketClient client = new WebSocketClient(); client.start(); client.connect(endpoint, URI.create("ws://host:port/endpoint/"));

Below is the thread dump with websocket threads:
"WebSocket@88584176-46" #46 prio=5 os_prio=0 cpu=0.00ms elapsed=41.18s tid=0x000001a262077b00 nid=0x2fd8 runnable [0x0000009ee36fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b193b10> (a sun.nio.ch.Util$2)
- locked <0x000000061b193998> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$636/0x000001a21a543730.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-47" #47 prio=5 os_prio=0 cpu=46.88ms elapsed=41.18s tid=0x000001a262072a00 nid=0x4124 waiting on condition [0x0000009ee3efe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.8.1/Native Method)
- parking to wait for <0x0000000603fde450> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.8.1/LockSupport.java:252)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@17.0.8.1/AbstractQueuedSynchronizer.java:1672)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:219)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.idleJobPoll(QueuedThreadPool.java:1015)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1065)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-48" #48 prio=5 os_prio=0 cpu=0.00ms elapsed=41.18s tid=0x000001a262073420 nid=0x6b64 runnable [0x0000009ee40ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b19d758> (a sun.nio.ch.Util$2)
- locked <0x000000061b19d5e0> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$636/0x000001a21a543730.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-49" #49 prio=5 os_prio=0 cpu=0.00ms elapsed=41.18s tid=0x000001a262073930 nid=0x5140 runnable [0x0000009ee41ff000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b19ec50> (a sun.nio.ch.Util$2)
- locked <0x000000061b19ead8> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$636/0x000001a21a543730.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-50" #50 prio=5 os_prio=0 cpu=0.00ms elapsed=41.18s tid=0x000001a262074860 nid=0x69b4 runnable [0x0000009ee42fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b1a0148> (a sun.nio.ch.Util$2)
- locked <0x000000061b19ffd0> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$636/0x000001a21a543730.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-51" #51 prio=5 os_prio=0 cpu=0.00ms elapsed=41.18s tid=0x000001a262073e40 nid=0x6178 runnable [0x0000009ee43fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b1c75c8> (a sun.nio.ch.Util$2)
- locked <0x000000061b1c7450> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produce(AdaptiveExecutionStrategy.java:194)
at org.eclipse.jetty.io.ManagedSelector$$Lambda$636/0x000001a21a543730.run(Unknown Source)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-52" #52 prio=5 os_prio=0 cpu=15.62ms elapsed=41.18s tid=0x000001a262078520 nid=0x60a0 runnable [0x0000009ee44fe000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WEPoll.wait(java.base@17.0.8.1/Native Method)
at sun.nio.ch.WEPollSelectorImpl.doSelect(java.base@17.0.8.1/WEPollSelectorImpl.java:111)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(java.base@17.0.8.1/SelectorImpl.java:129)
- locked <0x000000061b19c260> (a sun.nio.ch.Util$2)
- locked <0x000000061b19c0e8> (a sun.nio.ch.WEPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(java.base@17.0.8.1/SelectorImpl.java:146)
at org.eclipse.jetty.io.ManagedSelector.nioSelect(ManagedSelector.java:180)
at org.eclipse.jetty.io.ManagedSelector.select(ManagedSelector.java:187)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:604)
at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:541)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.produceTask(AdaptiveExecutionStrategy.java:450)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:243)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy$$Lambda$635/0x000001a21a5430e0.run(Unknown Source)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None

"WebSocket@88584176-53" #53 prio=5 os_prio=0 cpu=437.50ms elapsed=41.18s tid=0x000001a281277510 nid=0x69b8 waiting on condition [0x0000009ee45ff000]
java.lang.Thread.State: TIMED_WAITING (parking)
at jdk.internal.misc.Unsafe.park(java.base@17.0.8.1/Native Method)
- parking to wait for <0x0000000603fe49c0> (a java.util.concurrent.SynchronousQueue$TransferStack)
at java.util.concurrent.locks.LockSupport.parkNanos(java.base@17.0.8.1/LockSupport.java:252)
at java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@17.0.8.1/SynchronousQueue.java:401)
at java.util.concurrent.SynchronousQueue.poll(java.base@17.0.8.1/SynchronousQueue.java:903)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:325)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:401)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:934)
at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1078)
at java.lang.Thread.run(java.base@17.0.8.1/Thread.java:833)

Locked ownable synchronizers:
- None
@Spandana806 Spandana806 changed the title Multiple WebSocket threads getting created for every single client conenction Multiple WebSocket threads getting created for every single client connection Nov 3, 2023
@joakime
Copy link
Contributor

joakime commented Nov 3, 2023

This appears to be a the standard java nio selector allocation based on your hardware.
And it's not being created for every single client connection, it's being created for every instance of HttpClient or WebSocketClient.
A single HttpClient or WebSocketClient can open and manage hundreds of thousands of client connections (there is no need to be creating / starting multiple clients)
All connections will share those selector threads.

Ultimately this is harmless and normal if you use the HttpClient and WebSocketClient properly.

But, If you are not going to be using thousands of websocket connections simultaneously, dial down your selector calculation down to 1.
Just keep in mind that you should still use HttpClient and WebSocketClient properly (only 1 instance for however many connections you are going to create)
A single selector should be good for around 1000 active connections. (depending on hardware)

Eg:

ClientConnector connector = new ClientConnector();
connector.setSelectors(1);
HttpClient http = new HttpClient(new HttpClientTransportOverHTTP(connector));
http.start();

WebSocketClient ws = new WebSocketClient(http);
ws.start();

@Spandana806
Copy link
Author

Thank you for the quick response.

I am trying to create a websocket proxy server based on discussion in this thread.

I have uploaded the code here

Based on your suggestion, I am creating single instance of WebSocketContainer here.

After which I see the below threads getting created and will stay through out the period my server runs:
one - Connector-Scheduler-6e1ae763-1
one - HttpClient@a94a452-scheduler-1
nine - WebSocket

My next set of questions are:

  • Why are these threads in running state even when there are no websocket connections active?
  • What will be their lifetime?
  • How many client connections can they serve?
  • Do we need to close the WebSocketContainer? If yes, when?

@sbordet
Copy link
Contributor

sbordet commented Nov 8, 2023

Why are you worried about those threads?
They are necessary for the WebSocket implementation to work.

The scheduler threads are necessary and you see one for the server and one for the client.

The other runnable threads are likely selector threads waiting for connections to be established or readable.

As @joakime you can reduce the number of selectors in both the server and client via configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@joakime @sbordet @Spandana806