While running a suite of remote tests, tests starts to fail open new webdriver instance with error "failed to create a child event loop"

[saritapol13]

🐛 Bug Report

When the test suite is run after 40-50 tests using selenium4.0.0-alpha04 jar, the test to open new webdriver instance starts failing with below exception.

This suite works fine with selenium-server-standalone-3.141.59.jar

The code to spawn the new browser is:
new RemoteWebDriver(new URL(remoteServerUrl), getEdgeChromiumOptions());

java.lang.IllegalStateException: failed to create a child event loop at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:88) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:47) at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:59) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:86) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:81) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:68) at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:32) at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:21) at org.asynchttpclient.netty.channel.ChannelManager.<init>(ChannelManager.java:130) at org.asynchttpclient.DefaultAsyncHttpClient.<init>(DefaultAsyncHttpClient.java:89) at org.asynchttpclient.DefaultAsyncHttpClient.<init>(DefaultAsyncHttpClient.java:71) at org.asynchttpclient.Dsl.asyncHttpClient(Dsl.java:28) at org.openqa.selenium.remote.http.netty.CreateNettyClient.apply(CreateNettyClient.java:34) at org.openqa.selenium.remote.http.netty.NettyHttpHandler.<init>(NettyHttpHandler.java:39) at org.openqa.selenium.remote.http.netty.NettyClient$Factory.createClient(NettyClient.java:68) at org.openqa.selenium.remote.http.HttpClient$Factory.createClient(HttpClient.java:71) at org.openqa.selenium.remote.HttpCommandExecutor.<init>(HttpCommandExecutor.java:90) at org.openqa.selenium.remote.HttpCommandExecutor.<init>(HttpCommandExecutor.java:73) at org.openqa.selenium.remote.HttpCommandExecutor.<init>(HttpCommandExecutor.java:60) at org.openqa.selenium.remote.RemoteWebDriver.<init>(RemoteWebDriver.java:149) at gaugecore.utils.CustomWebDriver.browserFor(CustomWebDriver.java:189) at gaugecore.utils.CustomWebDriver.getBrowser(CustomWebDriver.java:79) at gaugecore.utils.MultiBrowser.openNewBrowser(MultiBrowser.java:383) at gaugecore.utils.ScenarioContext.setUp(ScenarioContext.java:174) at jdk.internal.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at com.thoughtworks.gauge.execution.MethodExecutor.execute(MethodExecutor.java:38) at com.thoughtworks.gauge.execution.HooksExecutor$TaggedHookExecutor.executeHook(HooksExecutor.java:100) at com.thoughtworks.gauge.execution.HooksExecutor$TaggedHookExecutor.execute(HooksExecutor.java:88) at com.thoughtworks.gauge.execution.HooksExecutor.execute(HooksExecutor.java:45) at com.thoughtworks.gauge.processor.MethodExecutionMessageProcessor.executeHooks(MethodExecutionMessageProcessor.java:65) at com.thoughtworks.gauge.processor.ScenarioExecutionStartingProcessor.process(ScenarioExecutionStartingProcessor.java:34) at com.thoughtworks.gauge.connection.MessageDispatcher.dispatchMessages(MessageDispatcher.java:92) at com.thoughtworks.gauge.GaugeRuntime.dispatchMessages(GaugeRuntime.java:98) at com.thoughtworks.gauge.GaugeRuntime.lambda$connectInParallel$0(GaugeRuntime.java:74) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: io.netty.channel.ChannelException: failed to open a new selector at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:175) at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:142) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:146) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)

To Reproduce

Detailed steps to reproduce the behavior:

Expected behavior

A new browser should be spawned.

Test script or set of commands reproducing this issue

Environment

OS:
Browser:
Browser version:
Browser Driver version:
Language Bindings version:
Selenium Grid version (if applicable):

OSX configuration
Processor: 4x Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz
Kernel: 3.10.0-1062.4.1.el7.x86_64
Memory: 7.63 GB
Puppet: 3.8.7
Facter: 2.4.6

ulimit unlimited

[diemol]

@saritapol13 could you please help us to reproduce the issue?
For example, what does getEdgeChromiumOptions() do? What browser version and driver are you using?

[saritapol13]

@diemol The reproducible sample project code is committed here
ChromiumEdge browser version 84.0.522.49 and Starting MSEdgeDriver 84.0.522.49. The issue is reproducible even if you run the tests on chrome 84.0.4147.105. I'm running tests on the selenium grid setup.

Run the suite as:
gradlew clean test
Pass value for parameters as below:
browserName=chromiumEdge
gridServerUrl=gridserverurl

With the sample project code this is the exception stacktrace:

`product1.RemoteBrowserTests2 > about_tab_is_on_home_page2()[40] FAILED
java.lang.IllegalStateException: failed to create a child event loop
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:88)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:58)
at io.netty.util.concurrent.MultithreadEventExecutorGroup.(MultithreadEventExecutorGroup.java:47)
at io.netty.channel.MultithreadEventLoopGroup.(MultithreadEventLoopGroup.java:59)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:78)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:73)
at io.netty.channel.nio.NioEventLoopGroup.(NioEventLoopGroup.java:60)
at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:32)
at org.asynchttpclient.netty.channel.NioTransportFactory.newEventLoopGroup(NioTransportFactory.java:21)
at org.asynchttpclient.netty.channel.ChannelManager.(ChannelManager.java:130)
at org.asynchttpclient.DefaultAsyncHttpClient.(DefaultAsyncHttpClient.java:89)
at org.asynchttpclient.DefaultAsyncHttpClient.(DefaultAsyncHttpClient.java:71)
at org.asynchttpclient.Dsl.asyncHttpClient(Dsl.java:28)
at org.openqa.selenium.remote.http.netty.CreateNettyClient.apply(CreateNettyClient.java:34)
at org.openqa.selenium.remote.http.netty.NettyHttpHandler.(NettyHttpHandler.java:39)
at org.openqa.selenium.remote.http.netty.NettyClient$Factory.createClient(NettyClient.java:68)
at org.openqa.selenium.remote.http.HttpClient$Factory.createClient(HttpClient.java:71)
at org.openqa.selenium.remote.HttpCommandExecutor.(HttpCommandExecutor.java:90)
at org.openqa.selenium.remote.HttpCommandExecutor.(HttpCommandExecutor.java:73)
at org.openqa.selenium.remote.HttpCommandExecutor.(HttpCommandExecutor.java:60)
at org.openqa.selenium.remote.RemoteWebDriver.(RemoteWebDriver.java:149)
at utils.BrowserOperations.browserFor(BrowserOperations.java:44)
at utils.BrowserOperations.openNewBrowser(BrowserOperations.java:23)
at product1.RemoteBrowserTests2.about_tab_is_on_home_page2(RemoteBrowserTests2.java:22)

    Caused by:
    io.netty.channel.ChannelException: failed to open a new selector
        at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:181)
        at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:147)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:138)
        at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:37)
        at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84)
        ... 23 more

        Caused by:
        java.io.IOException: Too many open files
            at java.base/sun.nio.ch.IOUtil.makePipe(Native Method)
            at java.base/sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:83)
            at java.base/sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36)
            at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:179)
            ... 27 more

`

[saritapol13]

@diemol What I've seen is the method from package org.openqa.selenium.remote.http.netty;

 @Override
  public AsyncHttpClient apply(config) {
    Objects.requireNonNull(config, "Client config to use must be set.");

    return asyncHttpClient();
  }

always tries to create a new asynchttpclient which internally creates a new EventLoopGroup consuming lot of resources. Can we share the EventLoopGroup here?

[saritapol13]

Sorry forgot to add the Java version:
openjdk 11.0.5 2019-10-15 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.5+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.5+10-LTS, mixed mode, sharing)

[pujagani]

@diemol What I've seen is the method from package org.openqa.selenium.remote.http.netty;

 @Override
  public AsyncHttpClient apply(config) {
    Objects.requireNonNull(config, "Client config to use must be set.");

    return asyncHttpClient();
  }

always tries to create a new asynchttpclient which internally creates a new EventLoopGroup consuming a lot of resources. Can we share the EventLoopGroup here?

Selenium-4.0.0-Alpha-7 reuses the HttpClient https://github.com/SeleniumHQ/selenium/blob/selenium-4.0.0-alpha-7/java/client/src/org/openqa/selenium/remote/http/netty/NettyClient.java#L41. Please upgrade to that version. Thank you!

[marcosybarraa]

Same problem here with 4.0.0-Beta-1 and ChromeDriver

[pujagani]

I believe this fix a5d5af2 by @diemol should fix it. It is currently merged in the trunk. It will be released as part of Beta 2 release. It uses a single async http client instance.

[marcosybarraa]

Thanks Pujagani,
When is expected the beta2 to be available?

kind regards

[diemol]

Yes, this was fixed in that commit.
Beta2 should come soon, my guess is by the end of March 2021.
Closing for now but feel free to comment if the issue is present after beta 2 is released.