EventLoopException at client side on load

[Amitunreal]

I have worked on Netty 3.x.x and after encountering EPOLL Exceptions I decided to upgrade to Netty 4.0.0 alpha 5. So I am pretty new to this version.

I have a test setup involving a client and server which are using WebSocket Protocol of Netty 4.0.0 alpha 5 and running on

java version "1.6.0_35"
Java(TM) SE Runtime Environment (build 1.6.0_35-b10)
Java HotSpot(TM) 64-Bit Server VM (build 20.10-b01, mixed mode)

I have also applied following workaround as suggested by Netty developer to tackle EPOLL Exception,
-Dio.netty.epollBugWorkaround=true

With above setup, I am trying to download a 1mb file from server to client with the load of 50 request(files) per second and after 2 to 5 minutes I am getting following Exception,

The Exception block of main
io.netty.channel.EventLoopException: failed to create a child event loop
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:53)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:24)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:32)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:28)
at com.rancore.netty.Netty4Client.run(Netty4Client.java:88)
at com.rancore.netty.Netty4Client$2.run(Netty4Client.java:122)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:90)
at io.netty.channel.socket.nio.NioEventLoop.(NioEventLoop.java:83)
at io.netty.channel.socket.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:53)
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:50)
... 6 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:88)
... 9 more
The Exception block of main
io.netty.channel.EventLoopException: failed to create a child event loop
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:53)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:24)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:32)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:28)
at com.rancore.netty.Netty4Client.run(Netty4Client.java:88)
at com.rancore.netty.Netty4Client$2.run(Netty4Client.java:122)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:90)
at io.netty.channel.socket.nio.NioEventLoop.(NioEventLoop.java:83)
at io.netty.channel.socket.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:53)
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:50)
... 6 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.EPollArrayWrapper.epollCreate(Native Method)
at sun.nio.ch.EPollArrayWrapper.(EPollArrayWrapper.java:69)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:52)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:88)
... 9 more

The Scenario at client side is

1. Client creates a Bootstrap and initializes it.
2. Client connects to the Server
3. With the returned channel it writes a Http Post request to server and wait for 200 OK.
4. After receiving 200 OK it starts the Handshake process.
5. As soon as the handshake process completes the client receives a TextWebSocketFrame with metadata of requested file.
6. After receiving above, Client receives a BinaryWebSocketFrame having binary data of requested file.

There is no Error reported on the Server Side at all.

Please provide insight for above problem.

[trustin]

Hi! Thanks for trying Netty 4 first of all. I really appreciate your feed back.

Could you ensure that you are not creating too many NioEventLoopGroups? Usually, it's just fine to create a single event loop group instance and reuse it for all channels. If not, would you mind if you write a simple test case (or test app) that demonstrates the problem?

[Amitunreal]

Hey!! Thanks for replying,
Actually the test case on which I am working is as follows,

1. I have created a server which is able to handle requests for file upload.
2. Of course one client can upload many files using single channel, But I am considering a test case in which I want to create multiple clients and to do so I should create Multiple Bootstraps, EventLoopGroup and Channel that means,

One Bootstraps, EventLoopGroup and Channel per client and I am creating almost 100 client per second that means 100 Bootstraps, EventLoopGroup and Channel per second, Though I Shutdown each and every Bootstrap at the end of call flow (explained above), I am getting the above Exception.

If creating multiple EventLoopGroups is not advisable then how to replicate above scenario?
Because at real time there will be millions of clients and each client will have its own Bootstrap, EventLoopGroup and Channel and the server should serve all those clients.

And if test app is necessary to understand the problem then I'll surely upload the code snippet of test case.

[normanmaurer]

The clients should share the EventLoopGroup

Sent from my iPhone. Excuse any typos....

Am 05.10.2012 um 18:25 schrieb Amitunreal notifications@github.com:

Hey!! Thanks for replying,
Actually the test case on which I am working is as follows,

I have created a server which is able to handle requests for file upload.
Of course one client can upload many files using single channel, But I am considering a test case in which I want to create multiple clients and to do so I should create Multiple Bootstraps, EventLoopGroup and Channel that means,
One Bootstraps, EventLoopGroup and Channel per client and I am creating almost 100 client per second that means 100 Bootstraps, EventLoopGroup and Channel per second, Though I Shutdown each and every Bootstrap at the end of call flow (explained above), I am getting the above Exception.

If creating multiple EventLoopGroups is not advisable then how to replicate above scenario?
Because at real time there will be millions of clients and each client will have its own Bootstrap, EventLoopGroup and Channel and the server should serve all those clients.

And if test app is necessary to understand the problem then I'll surely upload the code snippet of test case.

—
Reply to this email directly or view it on GitHub.

[Amitunreal]

1.OK, that means for testing purpose I should create only One EventLoopGroup and share it among all the clients?

And I should do this because I am creating all the clients in one code and a single software can't handle so amny EventLoopGroups isn't it?

Because you see normally there will be millions of clients using their own cellphones for file uploading so I am trying to replicate the exact scenario.

-- Now I am using only one client and putting it inside a loop because of which it is creating so many EventLoopGroup therefore you are suggesting me to share single EventLoopGroup among all clients, is it so?

I am currently running following scenario :-
while(true)
{
  if(logic to allow only 100 time in 1 second)
  {
    Create Bootstrap;
    Create EventLoopGroup;
    Create Channel Initializer;
    Initialize Bootstrap using above;
    Connect to server;
    Perform handshake;
    File transfer;
  }
}

[normanmaurer]

Yes

Sent from my iPhone. Excuse any typos....

Am 05.10.2012 um 19:42 schrieb Amitunreal notifications@github.com:

1.OK, that means for testing purpose I should create only One EventLoopGroup and share it among all the clients?

And I should do this because I am creating all the clients in one code and a single software can't handle so amny EventLoopGroups isn't it?

Because you see normally there will be millions of clients using their own cellphones for file uploading so I am trying to replicate the exact scenario.

-- Now I am using only one client and putting it inside a loop because of which it is creating so many EventLoopGroup therefore you are suggesting me to share single EventLoopGroup among all clients, is it so?

I am currently running following scenario :-
while(true)
{
if(logic to allow only 100 time in 1 second)
{
Create Bootstrap;
Create EventLoopGroup;
Create Channel Initializer;
Initialize Bootstrap using above;
Connect to server;
Perform handshake;
File transfer;
}
}

—
Reply to this email directly or view it on GitHub.

[Amitunreal]

Ok Thanks for your quikck response.
I'll try my load test scenario with your suggestion and update about it here ASAP
till then ZZZZZ.....

[slandelle]

Just out of curiosity, why don't you use a stress tool for this?

[Amitunreal]

Nice suggestion thanks, But still is there any stress tool available? From where can I get the stress tool, Because I am badly in need of such kind of tool.

[slandelle]

Self promotion: http://gatling-tool.org

[Amitunreal]

Thanx buddy......I'll see if that tool is helpful for me. But anyways Thanks.

[Amitunreal]

The problem still persists,

I am getting following Exception when I am running the client in Multi threaded environment: -

io.netty.channel.EventLoopException: failed to create a child event loop
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:53)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:24)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at com.rancore.netty.Test.(Test.java:49)
at com.rancore.netty.Test$2.run(Test.java:128)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:90)
at io.netty.channel.socket.nio.NioEventLoop.(NioEventLoop.java:83)
at io.netty.channel.socket.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:53)
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:50)
... 4 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:88)
... 7 more

Below is the link for Client Code on which I am working,

https://gist.github.com/3851329

Please review and revert with some solution.

Regards

Amit

[ghost]

this might be a silly suggestion but can you try using jdk7 ?

[normanmaurer]

Can you show me the limit for the user that is running the application:

ulimit -n

Thanks

Am 08.10.2012 um 10:13 schrieb Amitunreal notifications@github.com:

The problem still persists,

I am getting following Exception when I am running the client in Multi threaded environment: -

io.netty.channel.EventLoopException: failed to create a child event loop
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:53)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:24)
at io.netty.channel.socket.nio.NioEventLoopGroup.(NioEventLoopGroup.java:36)
at com.rancore.netty.Test.(Test.java:49)
at com.rancore.netty.Test$2.run(Test.java:128)
Caused by: io.netty.channel.ChannelException: failed to open a new selector
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:90)
at io.netty.channel.socket.nio.NioEventLoop.(NioEventLoop.java:83)
at io.netty.channel.socket.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:53)
at io.netty.channel.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:50)
... 4 more
Caused by: java.io.IOException: Too many open files
at sun.nio.ch.IOUtil.initPipe(Native Method)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:49)
at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at io.netty.channel.socket.nio.NioEventLoop.openSelector(NioEventLoop.java:88)
... 7 more

Below is the link for Client Code on which I am working,

https://gist.github.com/3851329

Please review and revert with some solution.

Regards

Amit

—
Reply to this email directly or view it on GitHub.

[Amitunreal]

The limit for the user that is running the application

# ulimit -n
1024

Hard Limit

# ulimit -H -n
1024

Soft Limit

# ulimit -S -n
1024

The current system limit for file descriptors

# cat /proc/sys/fs/file-max
6518084

Did you have a look at the code? I am running that client with 2 threads therefore I am getting above exception.

While, Multiple Instances of the same client with single thread is running fine.

[trustin]

You are still not sharing EventGroup. You create it every time you instantiate the class Test. Also, your code does not compile. Closing.

[Amitunreal]

I have corrected the code and made some changes for EventLoopGroup to make it shared.

Below is the link for Client Code on which I have corrected,

https://gist.github.com/3851329

Now the EventLoopGroup is static so that it will get created only once and it will be shared irrespective of the number of times I instantiate the Test class.

Ok.

Please reopen the issue.

[Amitunreal]

Please contribute in this thread....any suggestions?

[ghost]

try compiler compliance level 1.7, generated class files compatibilty 1.7, source compatibility 1.7,
then run it in jdk7

I personally cannot compile that, due to missing some:
RateLimitManager
RateLimitCachingManager
WebSocketClientHandler

I did manage to get CPSThreadFactory though.

[Amitunreal]

Sir I appreciate your suggestion that I should try it on JDK7, but Its an industry level project and we work on JDK6 here,

so.....Changing JDK is the only solution?

Because I have shared the exact scenario in dummy replica.....and can you tell me why shouldn't I try it on JDK6?
Why it is not workin on JDK6?

[ghost]

sorry, it's just something that I would try if I were you. You don't have to do it, chances are there will be no difference and you'll see the same errors. It's just something that I would personally try, well mainly because I cannot compile netty4 with jdk6 I get compile errors (about 160 in netty-transport alone) ie. this import "import java.net.StandardSocketOptions;" cannot be resolved in io.netty.channel.socket.DatagramChannelConfig ; but even so I chose jdk7 and it compiles, the ccl, gcfc and sc are all 1.6 still.

so it's probably not a solution at all, changing the JDK. But I figured since you wanted any suggestions, I wanted to tell you mine.

PS: I'm not a netty dev.

[Amitunreal]

Thanks for your suggestion I really appreciate it......but I have compiled Netty 4 (My shared code) on JDK6 it works fine but on load and in multi threaded environment I am getting above exception.

regards
Amit

[ghost]

Maybe it's related to this: #650

[normanmaurer]

@Amitunreal is it still exactly the same stacktrace now that you share the EventLoopGroup ?

[Amitunreal]

Maybe but i didn't explore much about the file descriptor usage in my client.....
I think now I should look into the FD usage of client so that I can confirm my problem is related to issue 650....

[Amitunreal]

Norman Sir, I didn't check the client after making the changes.
So I'll surely put that client on load and come up with some conclusion after some hours.
Ok.

Thanks & Regards

[lizuojing]

@Amitunreal I also get this Exception when i run my application for about two hours. I am try a few times and then i get the same exception.I want to know whether your solution is working,and whether you finally solve the problem.how to do it?