New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Node registration is occasionally delayed many minutes #14

Closed
c0nstructer opened this Issue Dec 1, 2014 · 39 comments

Comments

Projects
None yet
10 participants
@c0nstructer

c0nstructer commented Dec 1, 2014

Status 13 keeps popping on selenium grid after some time of usage. Been using webdriverio, server throws error that timeout expired. Help, anyone?

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Can you provide more detail please? Perhaps either logs, a stack trace, screen shot. What do you mean by popping on selenium grid?

Member

mtscout6 commented Dec 1, 2014

Can you provide more detail please? Perhaps either logs, a stack trace, screen shot. What do you mean by popping on selenium grid?

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

Yes of course. But first i must describe following situtation. At first, i used your standard hub and node-chrome, node firefox images. I spawned 1 request at time, worked fine. I spawned 5 requests, worked fine, i spawned 10 request, i got timeout. After that, i also tried only single request, i got also timeout. I wanted more parallel request, so I've overriden config.json in node chome and firefox to maxInstances 25, max Session 25.
I spawned 10 request to chrome, 10 request to firefox. Also crashed.

Also I am reporting slow register time, even with your default images, sometimes it takes 1 minutes to node to register on hub, sometimes 40 minutes.

Here is docker log on creating a request.

15:47:54.768 INFO - Trying to create a new session on test slot {platform=LINUX, seleniumProtocol=WebDriver, browserName=firefox, maxInstances=25}
15:47:54.775 INFO - Available nodes: [host :http://172.17.0.71:5555, host :http://172.17.0.70:5555]
15:47:54.775 INFO - Trying to create a new session on node host :http://172.17.0.71:5555
15:47:54.776 INFO - Trying to create a new session on test slot {platform=LINUX, seleniumProtocol=WebDriver, browserName=chrome, maxInstances=25}
15:48:40.204 WARN - IOException thrown while closing Closeable.
org.seleniumhq.jetty7.io.EofException
        at org.seleniumhq.jetty7.http.HttpGenerator.flushBuffer(HttpGenerator.java:939)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.flushResponse(AbstractHttpConnection.java:612)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection$Output.close(AbstractHttpConnection.java:1001)
        at com.google.common.io.Closeables.close(Closeables.java:79)
        at org.openqa.selenium.io.IOUtils.closeQuietly(IOUtils.java:45)
        at org.openqa.grid.internal.TestSession.writeRawBody(TestSession.java:440)
        at org.openqa.grid.internal.TestSession.forward(TestSession.java:266)
        at org.openqa.grid.web.servlet.handler.RequestHandler.forwardNewSessionRequestAndUpdateRegistry(RequestHandler.java:88)
        at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:111)
        at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:83)
        at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:67)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.seleniumhq.jetty7.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doHandle(ServletHandler.java:479)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doHandle(SessionHandler.java:225)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.seleniumhq.jetty7.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.seleniumhq.jetty7.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.seleniumhq.jetty7.server.Server.handle(Server.java:349)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
        at org.seleniumhq.jetty7.http.HttpParser.parseNext(HttpParser.java:857)
        at org.seleniumhq.jetty7.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
        at org.seleniumhq.jetty7.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at org.seleniumhq.jetty7.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:354)
        at org.seleniumhq.jetty7.io.bio.StreamEndPoint.flush(StreamEndPoint.java:159)
        at org.seleniumhq.jetty7.io.bio.StreamEndPoint.flush(StreamEndPoint.java:177)
        at org.seleniumhq.jetty7.http.HttpGenerator.flushBuffer(HttpGenerator.java:863)
        ... 33 more

On my side of script i get

RuntimeError: RuntimeError
     (UnknownError:13) An unknown server-side error occurred while processing th
e command.
     Problem: Process timed out after waiting for 45000 ms.←[1;30m

     Callstack:
     -> init()

Somethimes i get error (only spawning firefox e.g 2 requests)

RuntimeError: RuntimeError
     (UnknownError:13) An unknown server-side error occurred while processing th
e command.
     Problem: Unable to bind to locking port 7054 within 45000 ms
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch:
 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: FirefoxDriver←[1;30m

     Callstack:
     -> init()

Note that chrome is performing fine under 5 requests even instance and session are 25.

node firefox shows:

Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: FirefoxDriver
15:48:40.185 WARN - Exception thrown
java.util.concurrent.ExecutionException: org.openqa.selenium.WebDriverException: java.lang.reflect.InvocationTargetException
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: unknown
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.openqa.selenium.remote.server.DefaultSession.execute(DefaultSession.java:175)
        at org.openqa.selenium.remote.server.DefaultSession.<init>(DefaultSession.java:111)
        at org.openqa.selenium.remote.server.DefaultSession.createSession(DefaultSession.java:88)
        at org.openqa.selenium.remote.server.DefaultDriverSessions.newSession(DefaultDriverSessions.java:108)
        at org.openqa.selenium.remote.server.handler.NewSession.handle(NewSession.java:57)
        at org.openqa.selenium.remote.server.handler.NewSession.handle(NewSession.java:1)
        at org.openqa.selenium.remote.server.rest.ResultConfig.handle(ResultConfig.java:112)
        at org.openqa.selenium.remote.server.JsonHttpCommandHandler.handleRequest(JsonHttpCommandHandler.java:172)
        at org.openqa.selenium.remote.server.DriverServlet.handleRequest(DriverServlet.java:201)
        at org.openqa.selenium.remote.server.DriverServlet.doPost(DriverServlet.java:163)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.openqa.selenium.remote.server.DriverServlet.service(DriverServlet.java:129)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.openqa.jetty.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
        at org.openqa.jetty.jetty.servlet.ServletHandler.dispatch(ServletHandler.java:680)
        at org.openqa.jetty.jetty.servlet.ServletHandler.handle(ServletHandler.java:571)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1526)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1479)
        at org.openqa.jetty.http.HttpServer.service(HttpServer.java:920)
        at org.openqa.jetty.http.HttpConnection.service(HttpConnection.java:820)
        at org.openqa.jetty.http.HttpConnection.handleNext(HttpConnection.java:986)
        at org.openqa.jetty.http.HttpConnection.handle(HttpConnection.java:837)
        at org.openqa.jetty.http.SocketListener.handleConnection(SocketListener.java:243)
        at org.openqa.jetty.util.ThreadedServer.handle(ThreadedServer.java:358)
        at org.openqa.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:537)

c0nstructer commented Dec 1, 2014

Yes of course. But first i must describe following situtation. At first, i used your standard hub and node-chrome, node firefox images. I spawned 1 request at time, worked fine. I spawned 5 requests, worked fine, i spawned 10 request, i got timeout. After that, i also tried only single request, i got also timeout. I wanted more parallel request, so I've overriden config.json in node chome and firefox to maxInstances 25, max Session 25.
I spawned 10 request to chrome, 10 request to firefox. Also crashed.

Also I am reporting slow register time, even with your default images, sometimes it takes 1 minutes to node to register on hub, sometimes 40 minutes.

Here is docker log on creating a request.

15:47:54.768 INFO - Trying to create a new session on test slot {platform=LINUX, seleniumProtocol=WebDriver, browserName=firefox, maxInstances=25}
15:47:54.775 INFO - Available nodes: [host :http://172.17.0.71:5555, host :http://172.17.0.70:5555]
15:47:54.775 INFO - Trying to create a new session on node host :http://172.17.0.71:5555
15:47:54.776 INFO - Trying to create a new session on test slot {platform=LINUX, seleniumProtocol=WebDriver, browserName=chrome, maxInstances=25}
15:48:40.204 WARN - IOException thrown while closing Closeable.
org.seleniumhq.jetty7.io.EofException
        at org.seleniumhq.jetty7.http.HttpGenerator.flushBuffer(HttpGenerator.java:939)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.flushResponse(AbstractHttpConnection.java:612)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection$Output.close(AbstractHttpConnection.java:1001)
        at com.google.common.io.Closeables.close(Closeables.java:79)
        at org.openqa.selenium.io.IOUtils.closeQuietly(IOUtils.java:45)
        at org.openqa.grid.internal.TestSession.writeRawBody(TestSession.java:440)
        at org.openqa.grid.internal.TestSession.forward(TestSession.java:266)
        at org.openqa.grid.web.servlet.handler.RequestHandler.forwardNewSessionRequestAndUpdateRegistry(RequestHandler.java:88)
        at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:111)
        at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:83)
        at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:67)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.seleniumhq.jetty7.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doHandle(ServletHandler.java:479)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doHandle(SessionHandler.java:225)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.seleniumhq.jetty7.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.seleniumhq.jetty7.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.seleniumhq.jetty7.server.Server.handle(Server.java:349)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
        at org.seleniumhq.jetty7.http.HttpParser.parseNext(HttpParser.java:857)
        at org.seleniumhq.jetty7.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
        at org.seleniumhq.jetty7.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Broken pipe
        at java.net.SocketOutputStream.socketWrite0(Native Method)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:113)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:159)
        at org.seleniumhq.jetty7.io.ByteArrayBuffer.writeTo(ByteArrayBuffer.java:354)
        at org.seleniumhq.jetty7.io.bio.StreamEndPoint.flush(StreamEndPoint.java:159)
        at org.seleniumhq.jetty7.io.bio.StreamEndPoint.flush(StreamEndPoint.java:177)
        at org.seleniumhq.jetty7.http.HttpGenerator.flushBuffer(HttpGenerator.java:863)
        ... 33 more

On my side of script i get

RuntimeError: RuntimeError
     (UnknownError:13) An unknown server-side error occurred while processing th
e command.
     Problem: Process timed out after waiting for 45000 ms.←[1;30m

     Callstack:
     -> init()

Somethimes i get error (only spawning firefox e.g 2 requests)

RuntimeError: RuntimeError
     (UnknownError:13) An unknown server-side error occurred while processing th
e command.
     Problem: Unable to bind to locking port 7054 within 45000 ms
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch:
 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: FirefoxDriver←[1;30m

     Callstack:
     -> init()

Note that chrome is performing fine under 5 requests even instance and session are 25.

node firefox shows:

Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: FirefoxDriver
15:48:40.185 WARN - Exception thrown
java.util.concurrent.ExecutionException: org.openqa.selenium.WebDriverException: java.lang.reflect.InvocationTargetException
Build info: version: '2.44.0', revision: '76d78cf', time: '2014-10-23 20:02:37'
System info: host: '7baa98c0e9ff', ip: '172.17.0.70', os.name: 'Linux', os.arch: 'amd64', os.version: '3.13.0-37-generic', java.version: '1.7.0_65'
Driver info: driver.version: unknown
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:188)
        at org.openqa.selenium.remote.server.DefaultSession.execute(DefaultSession.java:175)
        at org.openqa.selenium.remote.server.DefaultSession.<init>(DefaultSession.java:111)
        at org.openqa.selenium.remote.server.DefaultSession.createSession(DefaultSession.java:88)
        at org.openqa.selenium.remote.server.DefaultDriverSessions.newSession(DefaultDriverSessions.java:108)
        at org.openqa.selenium.remote.server.handler.NewSession.handle(NewSession.java:57)
        at org.openqa.selenium.remote.server.handler.NewSession.handle(NewSession.java:1)
        at org.openqa.selenium.remote.server.rest.ResultConfig.handle(ResultConfig.java:112)
        at org.openqa.selenium.remote.server.JsonHttpCommandHandler.handleRequest(JsonHttpCommandHandler.java:172)
        at org.openqa.selenium.remote.server.DriverServlet.handleRequest(DriverServlet.java:201)
        at org.openqa.selenium.remote.server.DriverServlet.doPost(DriverServlet.java:163)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at org.openqa.selenium.remote.server.DriverServlet.service(DriverServlet.java:129)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.openqa.jetty.jetty.servlet.ServletHolder.handle(ServletHolder.java:428)
        at org.openqa.jetty.jetty.servlet.ServletHandler.dispatch(ServletHandler.java:680)
        at org.openqa.jetty.jetty.servlet.ServletHandler.handle(ServletHandler.java:571)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1526)
        at org.openqa.jetty.http.HttpContext.handle(HttpContext.java:1479)
        at org.openqa.jetty.http.HttpServer.service(HttpServer.java:920)
        at org.openqa.jetty.http.HttpConnection.service(HttpConnection.java:820)
        at org.openqa.jetty.http.HttpConnection.handleNext(HttpConnection.java:986)
        at org.openqa.jetty.http.HttpConnection.handle(HttpConnection.java:837)
        at org.openqa.jetty.http.SocketListener.handleConnection(SocketListener.java:243)
        at org.openqa.jetty.util.ThreadedServer.handle(ThreadedServer.java:358)
        at org.openqa.jetty.util.ThreadPool$PoolThread.run(ThreadPool.java:537)
@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Instead of overriding the config.json, have you you tried running multiple node-chrome and node-firefox containers. The docker philosophy is one process per container. So, ideally if you want to run more tests in parallel you'd want to run multiple containers. Note, this is really cheap to do with docker. Do you still run into the same problem when you do that?

Member

mtscout6 commented Dec 1, 2014

Instead of overriding the config.json, have you you tried running multiple node-chrome and node-firefox containers. The docker philosophy is one process per container. So, ideally if you want to run more tests in parallel you'd want to run multiple containers. Note, this is really cheap to do with docker. Do you still run into the same problem when you do that?

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

I will do that, although ram memory is a bit more consumed when generating nodes.
I will report my findings, i must wait a bit longer for register cycle on nodes, its not instant.

c0nstructer commented Dec 1, 2014

I will do that, although ram memory is a bit more consumed when generating nodes.
I will report my findings, i must wait a bit longer for register cycle on nodes, its not instant.

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Also, when your node does not register immediately and sometimes taking 40 minutes is this the output you see:

09:00:36.034 INFO - Java: Oracle Corporation 24.65-b04
09:00:36.035 INFO - OS: Linux 3.16.4-tinycore64 amd64
09:00:36.039 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
09:00:36.091 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: LINUX
09:00:36.118 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:5555/wd/hub
09:00:36.119 INFO - Version Jetty/5.1.x
09:00:36.121 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
09:00:36.122 INFO - Started HttpContext[/selenium-server,/selenium-server]
09:00:36.122 INFO - Started HttpContext[/,/]

If so this is something I've been seeing as well, I'm still trying to figure out the cause.

Member

mtscout6 commented Dec 1, 2014

Also, when your node does not register immediately and sometimes taking 40 minutes is this the output you see:

09:00:36.034 INFO - Java: Oracle Corporation 24.65-b04
09:00:36.035 INFO - OS: Linux 3.16.4-tinycore64 amd64
09:00:36.039 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
09:00:36.091 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: LINUX
09:00:36.118 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:5555/wd/hub
09:00:36.119 INFO - Version Jetty/5.1.x
09:00:36.121 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
09:00:36.122 INFO - Started HttpContext[/selenium-server,/selenium-server]
09:00:36.122 INFO - Started HttpContext[/,/]

If so this is something I've been seeing as well, I'm still trying to figure out the cause.

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

@c0nstructer Interesting thought about RAM, I had not considered that. I seem to recall something about running jar files and restricting RAM consumption. I wonder, since we are only running at most one test session per node would limiting the RAM consumed by the JVM be reasonable here?

Member

mtscout6 commented Dec 1, 2014

@c0nstructer Interesting thought about RAM, I had not considered that. I seem to recall something about running jar files and restricting RAM consumption. I wonder, since we are only running at most one test session per node would limiting the RAM consumed by the JVM be reasonable here?

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

Ok,

Ive pulled clean images hub, chrome and firefox

This is log of chrome

16:36:04.336 INFO - Launching a selenium grid node
16:36:05.250 INFO - Java: Oracle Corporation 24.65-b04
16:36:05.250 INFO - OS: Linux 3.13.0-37-generic amd64
16:36:05.262 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
16:36:05.351 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: LINUX
16:36:05.381 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:5555/wd/hub
16:36:05.382 INFO - Version Jetty/5.1.x
16:36:05.383 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
16:36:05.384 INFO - Started HttpContext[/selenium-server,/selenium-server]
16:36:05.384 INFO - Started HttpContext[/,/]

Also same as firefox

c0nstructer commented Dec 1, 2014

Ok,

Ive pulled clean images hub, chrome and firefox

This is log of chrome

16:36:04.336 INFO - Launching a selenium grid node
16:36:05.250 INFO - Java: Oracle Corporation 24.65-b04
16:36:05.250 INFO - OS: Linux 3.13.0-37-generic amd64
16:36:05.262 INFO - v2.44.0, with Core v2.44.0. Built from revision 76d78cf
16:36:05.351 INFO - Default driver org.openqa.selenium.ie.InternetExplorerDriver registration is skipped: registration capabilities Capabilities [{platform=WINDOWS, ensureCleanSession=true, browserName=internet explorer, version=}] does not match with current platform: LINUX
16:36:05.381 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:5555/wd/hub
16:36:05.382 INFO - Version Jetty/5.1.x
16:36:05.383 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
16:36:05.384 INFO - Started HttpContext[/selenium-server,/selenium-server]
16:36:05.384 INFO - Started HttpContext[/,/]

Also same as firefox

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

You are seeing the same thing I am. I'm talking with some core Selenium contribs on IRC and it looks like this is a known issue in a Java core lib. I has something to do with a bad random number generator. So, it's not a Selenium or Docker specific problem. I'll see if I can find a published issue about it and post the link here.

Member

mtscout6 commented Dec 1, 2014

You are seeing the same thing I am. I'm talking with some core Selenium contribs on IRC and it looks like this is a known issue in a Java core lib. I has something to do with a bad random number generator. So, it's not a Selenium or Docker specific problem. I'll see if I can find a published issue about it and post the link here.

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

Well my idea was that if one node is runned under 20 sessions it consumes ram for one node and to run chrome. If i spawn 1 node for 1 process, it will take ram for node and for browser, and that consumes alot of ram.

So basicly my idea was 1 node 25 chome, second node 25 firefox browsers, and less ram consumptions.

c0nstructer commented Dec 1, 2014

Well my idea was that if one node is runned under 20 sessions it consumes ram for one node and to run chrome. If i spawn 1 node for 1 process, it will take ram for node and for browser, and that consumes alot of ram.

So basicly my idea was 1 node 25 chome, second node 25 firefox browsers, and less ram consumptions.

@mtscout6 mtscout6 referenced this issue Dec 1, 2014

Open

Investigate RAM considerations in the Nodes #15

0 of 4 tasks complete
@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Let's move the RAM discussion to a new issue see #15

Member

mtscout6 commented Dec 1, 2014

Let's move the RAM discussion to a new issue see #15

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

When if finally does register the following log entry is that the ServletHandler was finally started. Which is right where @WhereIsMySpoon mentioned in the #Selenium IRC channel (log)

09:02:25.204 INFO - Started HttpContext[/,/]
09:05:58.688 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@1c664c4f
09:05:58.689 INFO - Started HttpContext[/wd,/wd]
09:05:58.694 INFO - Started SocketListener on 0.0.0.0:5555
09:05:58.694 INFO - Started org.openqa.jetty.jetty.Server@d0adc0c
Member

mtscout6 commented Dec 1, 2014

When if finally does register the following log entry is that the ServletHandler was finally started. Which is right where @WhereIsMySpoon mentioned in the #Selenium IRC channel (log)

09:02:25.204 INFO - Started HttpContext[/,/]
09:05:58.688 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@1c664c4f
09:05:58.689 INFO - Started HttpContext[/wd,/wd]
09:05:58.694 INFO - Started SocketListener on 0.0.0.0:5555
09:05:58.694 INFO - Started org.openqa.jetty.jetty.Server@d0adc0c

@c0nstructer c0nstructer closed this Dec 1, 2014

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

@c0nstructer were you able to figure out the registration problem?

Member

mtscout6 commented Dec 1, 2014

@c0nstructer were you able to figure out the registration problem?

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

They registered, i spawned 1 request per node working ok. I will spawn 20, 30 nodes and try 20, 30 request, will report to you whats happening

c0nstructer commented Dec 1, 2014

They registered, i spawned 1 request per node working ok. I will spawn 20, 30 nodes and try 20, 30 request, will report to you whats happening

@mtscout6 mtscout6 changed the title from Status 13 to Node registration is occasionally delayed many minutes Dec 1, 2014

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Can you re-open this issue, I renamed it. The registration delay is still a problem. The RAM discussion though is a separate issue to discuss in #15.

Member

mtscout6 commented Dec 1, 2014

Can you re-open this issue, I renamed it. The registration delay is still a problem. The RAM discussion though is a separate issue to discuss in #15.

@mtscout6 mtscout6 reopened this Dec 1, 2014

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

Nevermind I didn't have the re-open button at first... Odd.

Member

mtscout6 commented Dec 1, 2014

Nevermind I didn't have the re-open button at first... Odd.

@mtscout6 mtscout6 added the D-defect label Dec 1, 2014

@mtscout6

This comment has been minimized.

Show comment
Hide comment
Member

mtscout6 commented Dec 1, 2014

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

@mtscout6 Do you want me to register it on github?

c0nstructer commented Dec 1, 2014

@mtscout6 Do you want me to register it on github?

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 1, 2014

Member

register it on github? What are you asking? I was asking you to re-open the issue since you closed it, but it's good now.

Member

mtscout6 commented Dec 1, 2014

register it on github? What are you asking? I was asking you to re-open the issue since you closed it, but it's good now.

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 1, 2014

I closed it by mistake, sorry :)
I was thinking on open issue on github about node to hub registration.

c0nstructer commented Dec 1, 2014

I closed it by mistake, sorry :)
I was thinking on open issue on github about node to hub registration.

@jaredm4

This comment has been minimized.

Show comment
Hide comment
@jaredm4

jaredm4 Dec 3, 2014

Happening to me as well. Over 6 minutes! Hoping Selenium team can help with a fix for this soon. Really wanted to Fig the Selenium instances. :)

firefox_1 | 00:22:44.580 INFO - Started HttpContext[/,/]
firefox_1 | 00:28:57.556 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@5b3dbee4

jaredm4 commented Dec 3, 2014

Happening to me as well. Over 6 minutes! Hoping Selenium team can help with a fix for this soon. Really wanted to Fig the Selenium instances. :)

firefox_1 | 00:22:44.580 INFO - Started HttpContext[/,/]
firefox_1 | 00:28:57.556 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler@5b3dbee4
@eugene-dounar

This comment has been minimized.

Show comment
Hide comment
@eugene-dounar

eugene-dounar Dec 11, 2014

Also affected.
node-chrome connects to the hub instantly on my laptop with 8Gb RAM, however when launched on DigitalOcean 1Gb droplet it takes 4 minutes.

Logs https://gist.github.com/eugene-dounar/52a302f52dcdf61e5e1a

eugene-dounar commented Dec 11, 2014

Also affected.
node-chrome connects to the hub instantly on my laptop with 8Gb RAM, however when launched on DigitalOcean 1Gb droplet it takes 4 minutes.

Logs https://gist.github.com/eugene-dounar/52a302f52dcdf61e5e1a

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 11, 2014

Member

This is not a RAM issue, this is an issue with selenium core when generating a random session seed. When debug logging is turned up this is the line that causes the hangup:

15:13:00.159 DEBUG - New random session seed
Member

mtscout6 commented Dec 11, 2014

This is not a RAM issue, this is an issue with selenium core when generating a random session seed. When debug logging is turned up this is the line that causes the hangup:

15:13:00.159 DEBUG - New random session seed
@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 11, 2014

I am also again reporting, 30 minutes, and nothing, even builded images from this git repo.

c0nstructer commented Dec 11, 2014

I am also again reporting, 30 minutes, and nothing, even builded images from this git repo.

@eugene-dounar

This comment has been minimized.

Show comment
Hide comment
@eugene-dounar

eugene-dounar Dec 12, 2014

Ok, it seems that I've been able to fix that. @mtscout6 hint should have been obvious that selenium just blocks until kernel have enough entropy to generate this random seed.
So just try installing haveged. For Ubuntu:

apt-get install haveged
service haveged start
update-rc.d haveged defaults

eugene-dounar commented Dec 12, 2014

Ok, it seems that I've been able to fix that. @mtscout6 hint should have been obvious that selenium just blocks until kernel have enough entropy to generate this random seed.
So just try installing haveged. For Ubuntu:

apt-get install haveged
service haveged start
update-rc.d haveged defaults
@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 12, 2014

Member

@eugene-dounar Ya, that sounds really obvious 😜

Did you install this into your docker host or into the selenium/node-* image itself? If so, how did you do it? I'm not too keen on the idea of running another service to manage this within the container.

Doing some further reading I stumbled on a blog post which really explains this problem in better detail than I can. Also, there is a Stack Overflow question that suggesting running with docker run -v /dev/urandom:/dev/random ... which I have found to work for me. Unfortunately there are trade-offs to this solution as stated the the blog post I linked earlier.

Member

mtscout6 commented Dec 12, 2014

@eugene-dounar Ya, that sounds really obvious 😜

Did you install this into your docker host or into the selenium/node-* image itself? If so, how did you do it? I'm not too keen on the idea of running another service to manage this within the container.

Doing some further reading I stumbled on a blog post which really explains this problem in better detail than I can. Also, there is a Stack Overflow question that suggesting running with docker run -v /dev/urandom:/dev/random ... which I have found to work for me. Unfortunately there are trade-offs to this solution as stated the the blog post I linked earlier.

@eugene-dounar

This comment has been minimized.

Show comment
Hide comment
@eugene-dounar

eugene-dounar Dec 12, 2014

I just installed haveged on the docker host itself. It works since all docker containers use host's kernel and same /dev/random. So adding entropy on the host will resolve the problem inside containers.

I use Linux on both laptop and hosting so not sure about how to fix it in boot2docker on OSX :\

eugene-dounar commented Dec 12, 2014

I just installed haveged on the docker host itself. It works since all docker containers use host's kernel and same /dev/random. So adding entropy on the host will resolve the problem inside containers.

I use Linux on both laptop and hosting so not sure about how to fix it in boot2docker on OSX :\

@mtscout6

This comment has been minimized.

Show comment
Hide comment
@mtscout6

mtscout6 Dec 12, 2014

Member

The question I have is whether or not that is an acceptable solution for the broader community.

cc @lukeis @elgalu @psftw @shs96c @barancev @jimevans
Please CC anyone else that could weigh in on this better than I can.

Member

mtscout6 commented Dec 12, 2014

The question I have is whether or not that is an acceptable solution for the broader community.

cc @lukeis @elgalu @psftw @shs96c @barancev @jimevans
Please CC anyone else that could weigh in on this better than I can.

@jaredm4

This comment has been minimized.

Show comment
Hide comment
@jaredm4

jaredm4 Dec 12, 2014

I don't think it's acceptable personally, very kludgy. But it may give Docker some hints on how to solve this with the daemon itself?

jaredm4 commented Dec 12, 2014

I don't think it's acceptable personally, very kludgy. But it may give Docker some hints on how to solve this with the daemon itself?

@eugene-dounar

This comment has been minimized.

Show comment
Hide comment
@eugene-dounar

eugene-dounar Dec 12, 2014

I think it's a Selenium issue. Using /dev/random makes no sense here.

eugene-dounar commented Dec 12, 2014

I think it's a Selenium issue. Using /dev/random makes no sense here.

@WillAbides

This comment has been minimized.

Show comment
Hide comment
@WillAbides

WillAbides Dec 12, 2014

Contributor

Thanks, @eugene-dounar for finding the random issue. I was pulling my hair out over this last night and was dreading more troubleshooting. Your comment changed the trajectory of my day.

Contributor

WillAbides commented Dec 12, 2014

Thanks, @eugene-dounar for finding the random issue. I was pulling my hair out over this last night and was dreading more troubleshooting. Your comment changed the trajectory of my day.

@krosenvold

This comment has been minimized.

Show comment
Hide comment
@krosenvold

krosenvold Dec 12, 2014

Member

This is really a core java issue. Java uses /dev/random for lots of stuff and is not something that can be fixed within selenium. File a bug report with oracle, find enough entropy or live with the waits.

Member

krosenvold commented Dec 12, 2014

This is really a core java issue. Java uses /dev/random for lots of stuff and is not something that can be fixed within selenium. File a bug report with oracle, find enough entropy or live with the waits.

mtscout6 added a commit that referenced this issue Dec 12, 2014

Add volume mount from /dev/urandom:/dev/random
The test script will not run repeatedly due to an entropy starvation
problem with `/dev/random`. This will redirect that to the non-blocking
`/dev/urandom`.

Part of #14

@mtscout6 mtscout6 closed this in ef53bd4 Dec 12, 2014

@c0nstructer

This comment has been minimized.

Show comment
Hide comment
@c0nstructer

c0nstructer Dec 13, 2014

Hello,

I found the real problem, and workaround solution. As @krosenvold mentioned its not selenium's fault its java's fault.

Problem is that selenium cannot even open sockets, and connect to hub, looking thats something wrong.

I've used following commands (without docker, regular selenium)

$apt-get install rng-tools
$rngd -r /dev/./random

Started selenium hub and node instantly connected, even started standalone selenium, worked like a charm!!!

I hope that this solution will help you guys.

c0nstructer commented Dec 13, 2014

Hello,

I found the real problem, and workaround solution. As @krosenvold mentioned its not selenium's fault its java's fault.

Problem is that selenium cannot even open sockets, and connect to hub, looking thats something wrong.

I've used following commands (without docker, regular selenium)

$apt-get install rng-tools
$rngd -r /dev/./random

Started selenium hub and node instantly connected, even started standalone selenium, worked like a charm!!!

I hope that this solution will help you guys.

@nirvdrum

This comment has been minimized.

Show comment
Hide comment
@nirvdrum

nirvdrum Dec 15, 2014

We ran into this with JRuby, too. You can tell the JVM to seed from /dev/urandom instead, which is non-blocking, by way of a system property. You could also use something other than Linux, since no other OS seems to exhibit this behavior. Caveat emptor:

jruby/jruby#1405 (comment)

nirvdrum commented Dec 15, 2014

We ran into this with JRuby, too. You can tell the JVM to seed from /dev/urandom instead, which is non-blocking, by way of a system property. You could also use something other than Linux, since no other OS seems to exhibit this behavior. Caveat emptor:

jruby/jruby#1405 (comment)

@zxqfox

This comment has been minimized.

Show comment
Hide comment
@zxqfox

zxqfox Dec 17, 2014

In case you have enterprise standard app servers without privileges to change your binaries you can add a JVM flag to overwrite wich random generator should be used. E.g.:

-Djava.security.egd=file:/dev/./urandom

That way you don't have to change any system settings.

zxqfox commented Dec 17, 2014

In case you have enterprise standard app servers without privileges to change your binaries you can add a JVM flag to overwrite wich random generator should be used. E.g.:

-Djava.security.egd=file:/dev/./urandom

That way you don't have to change any system settings.

mtscout6 added a commit that referenced this issue Dec 31, 2014

Consolidate urandom corrective RUN command
To keep the number of layers down this logically belongs to the Java
install RUN command.

- #14
- #28
@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 25, 2015

I had this issue when spinning up my grid earlier. This fix solved it, but I was wondering if I need to open an issue elsewhere to see about getting this resolved, because in selenium-standalone-server-2.45 it's not.

ghost commented Apr 25, 2015

I had this issue when spinning up my grid earlier. This fix solved it, but I was wondering if I need to open an issue elsewhere to see about getting this resolved, because in selenium-standalone-server-2.45 it's not.

@dhruvbhatia

This comment has been minimized.

Show comment
Hide comment
@dhruvbhatia

dhruvbhatia Jul 8, 2015

I'm experiencing the same issue running docker-selenium on AWS EC2 instances (tested both Amazon Linux AMI 2015.03 (HVM) using Amazon's Docker release and Ubuntu Server 14.04 LTS (HVM) using Docker 1.7).

The docker run -v /dev/urandom:/dev/random ... solution helps a little (selenium progresses further in my queue), but things still eventually freeze up given enough time. What's weird is the entropy of the host system appears fine whenever the freeze occurs.

Has anybody experienced the same thing? For reference my grid is running 20 Chrome instances and my queue freezes after about ~1500 sessions.

dhruvbhatia commented Jul 8, 2015

I'm experiencing the same issue running docker-selenium on AWS EC2 instances (tested both Amazon Linux AMI 2015.03 (HVM) using Amazon's Docker release and Ubuntu Server 14.04 LTS (HVM) using Docker 1.7).

The docker run -v /dev/urandom:/dev/random ... solution helps a little (selenium progresses further in my queue), but things still eventually freeze up given enough time. What's weird is the entropy of the host system appears fine whenever the freeze occurs.

Has anybody experienced the same thing? For reference my grid is running 20 Chrome instances and my queue freezes after about ~1500 sessions.

@lucacri

This comment has been minimized.

Show comment
Hide comment
@lucacri

lucacri Jul 28, 2015

I spent the last 48 hours trying to fix this problem too!
@eugene-dounar 's suggestion to install haveged worked wonderfully. THANK YOU!

lucacri commented Jul 28, 2015

I spent the last 48 hours trying to fix this problem too!
@eugene-dounar 's suggestion to install haveged worked wonderfully. THANK YOU!

@nirvdrum

This comment has been minimized.

Show comment
Hide comment
@nirvdrum

nirvdrum Jul 28, 2015

@mekhami The issue is induced by the number of times random data is read (e.g., SecureRandom) and the rate at which /dev/random is being drained on a given machine. It may very well be that your selenium-server just isn't as active or isn't contending with as much. Unfortunately, it's a thorny issue.

nirvdrum commented Jul 28, 2015

@mekhami The issue is induced by the number of times random data is read (e.g., SecureRandom) and the rate at which /dev/random is being drained on a given machine. It may very well be that your selenium-server just isn't as active or isn't contending with as much. Unfortunately, it's a thorny issue.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Sep 11, 2015

FYI, I just fixed an issue with a selenium docker container hanging during startup by setting the container environment variable JAVA_OPTS to "-Djava.security.egd=file:/dev/./urandom", which the container entrypoint script respects.

ghost commented Sep 11, 2015

FYI, I just fixed an issue with a selenium docker container hanging during startup by setting the container environment variable JAVA_OPTS to "-Djava.security.egd=file:/dev/./urandom", which the container entrypoint script respects.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment