Skip to content
This repository has been archived by the owner. It is now read-only.

Tests, runned using Grid fail with error FORWARDING_TO_NODE_FAILED with Selenium 2.42.0 #7396

Closed
lukeis opened this issue Mar 4, 2016 · 36 comments

Comments

@lukeis
Copy link
Member

@lukeis lukeis commented Mar 4, 2016

Originally reported on Google Code with ID 7396

What steps will reproduce the problem?
1. Run tests with selenium grid in 2.42.0 version

What is the expected output? What do you see instead?

There are many problems like this:

Session [0f2d2a37-1892-4f1f-bc9d-7814cc74ea5a] was terminated due to FORWARDING_TO_NODE_FAILED
Command duration or timeout: 3 milliseconds Build info: version: '2.42.0', revision:
'5e824302019c86eae9c8c3ca9155e7307b410cf8', time: '2014-05-24 09:48:41' System info:
host: 'vLnxT1', ip: '127.0.1.1', os.name: 'Linux', os.arch: 'amd64', os.version: '3.0.0-16-generic',
java.version: '1.7.0_21' Session ID: 0f2d2a37-1892-4f1f-bc9d-7814cc74ea5a Driver info:
org.openqa.selenium.remote.RemoteWebDriver Capabilities [{platform=XP, acceptSslCerts=true,
javascriptEnabled=true, browserName=chrome, chrome={userDataDir=C:\xxx\scoped_dir2516_537},
rotatable=false, locationContextEnabled=true, webdriver.remote.sessionid=0f2d2a37-1892-4f1f-bc9d-7814cc74ea5a,
version=35.0.1916.114, takesHeapSnapshot=true, cssSelectorsEnabled=true, databaseEnabled=false,
handlesAlerts=true, browserConnectionEnabled=false, nativeEvents=true, webStorageEnabled=true,
applicationCacheEnabled=false, takesScreenshot=true}]


hub: linux, nodes:windows
runner:linux


Selenium version: 2.42.0
OS: Linux+Windows
Browser: Chrome
Browser version: 35.0.1916.114 m
Driver: ChromeDriver 2.10 

Additional:

There are no problems with nodes and it works fine in cases:

1) selenium-api 2.42.0 hub 2.41.0 nodes: 2.42.0
2) selenium-api 2.41.0 hub 2.41.0 nodes: 2.41.0

Theese problems appear just in the case: selenium-api 2.42.0 hub 2.42.0 nodes: 2.42.0

Reported by rekruit19881 on 2014-05-26 13:26:27

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Reported by barancev on 2014-05-26 19:53:37

  • Labels added: Component-Grid
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I'm experiencing the same bug.

Both my hub and nodes are at 2.42.0

This wasn't an issue before I upgraded from 2.41.0

Reported by matt.polyvisual on 2014-05-29 15:34:54

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I've rolled my environment back to 2.41.0 and all runs OK again.

Reported by matt.polyvisual on 2014-05-30 11:45:45

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Issue 7414 has been merged into this issue.

Reported by barancev on 2014-06-01 11:58:27

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Same issue. What are the repercussions of running the hub as 2.41 and the nodes as 2.42?

Reported by AdamJasonDev on 2014-06-04 02:24:08

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

we're also facing this issue since we've updated to 2.42.x

Reported by nico.schifferdecker on 2014-06-05 08:23:15

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Please run the hub and a node with -debug option and attach the debug log here. Or provide
a scenario to stable reproduce the issue.

Reported by barancev on 2014-06-07 22:09:23

  • Status changed: NeedsClarification
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Hub debug log attached

Reported by rekruit19881 on 2014-06-09 14:48:15


- _Attachment: [hub_log.zip](https://storage.googleapis.com/google-code-attachments/selenium/issue-7396/comment-8/hub_log.zip)_
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

The log has been attached for a while.
Can someone recheck this ticket?
We are having the same issue.

Reported by gangeld on 2014-06-24 10:00:32

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

experiencing the same prob.

for me it typically comes together with the "cannot forward the request ... has not
been leased from this pool" error and the hub (2.42) throws this when the problem occurs:

Jun 24, 2014 8:45:36 AM org.openqa.grid.web.servlet.handler.RequestHandler process
SEVERE: cannot forward the request Entry [id:1096827][route:{}->http://x.x.x.x:5555][state:null]
has not been leased from this pool
java.lang.IllegalStateException: Entry [id:1096827][route:{}->http://x.x.x.x:5555][state:null]
has not been leased from this pool
        at org.apache.http.util.Asserts.check(Asserts.java:40)
        at org.apache.http.pool.RouteSpecificPool.free(RouteSpecificPool.java:123)
        at org.apache.http.pool.AbstractConnPool.release(AbstractConnPool.java:308)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.releaseConnection(PoolingHttpClientConnectionManager.java:280)
        at org.apache.http.impl.execchain.ConnectionHolder.abortConnection(ConnectionHolder.java:132)
        at org.apache.http.impl.execchain.ResponseEntityWrapper.cleanup(ResponseEntityWrapper.java:60)
        at org.apache.http.impl.execchain.ResponseEntityWrapper.releaseConnection(ResponseEntityWrapper.java:71)
        at org.apache.http.impl.execchain.ResponseEntityWrapper.eofDetected(ResponseEntityWrapper.java:107)
        at org.apache.http.conn.EofSensorInputStream.checkEOF(EofSensorInputStream.java:199)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:138)
        at org.apache.http.conn.EofSensorInputStream.read(EofSensorInputStream.java:150)
        at com.google.common.io.ByteStreams.copy(ByteStreams.java:169)
        at com.google.common.io.ByteStreams.toByteArray(ByteStreams.java:214)
        at org.openqa.grid.internal.TestSession.drainInputStream(TestSession.java:445)
        at org.openqa.grid.internal.TestSession.forward(TestSession.java:265)
        at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:96)
        at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:129)
        at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:83)
        at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:67)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.seleniumhq.jetty7.servlet.ServletHolder.handle(ServletHolder.java:565)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doHandle(ServletHandler.java:479)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doHandle(SessionHandler.java:225)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doHandle(ContextHandler.java:1031)
        at org.seleniumhq.jetty7.servlet.ServletHandler.doScope(ServletHandler.java:406)
        at org.seleniumhq.jetty7.server.session.SessionHandler.doScope(SessionHandler.java:186)
        at org.seleniumhq.jetty7.server.handler.ContextHandler.doScope(ContextHandler.java:965)
        at org.seleniumhq.jetty7.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.seleniumhq.jetty7.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.seleniumhq.jetty7.server.Server.handle(Server.java:349)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:452)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:47)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection.content(AbstractHttpConnection.java:894)
        at org.seleniumhq.jetty7.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:948)
        at org.seleniumhq.jetty7.http.HttpParser.parseNext(HttpParser.java:857)
        at org.seleniumhq.jetty7.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.seleniumhq.jetty7.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:66)
        at org.seleniumhq.jetty7.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:254)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:599)
        at org.seleniumhq.jetty7.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:534)
        at java.lang.Thread.run(Thread.java:745)


At some point it started to happen consistently, then restarting the 2.42 hub also
helped. Not sure however what triggered this in the first place.


Reported by trieszmxd on 2014-06-24 10:21:29

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

The issue (or at least a part of it) is caused by a bug in httpclient [1], the fixed
version of httpclient will be shipped with selenium 2.43.

[1] https://issues.apache.org/jira/browse/HTTPCLIENT-1453

Reported by barancev on 2014-06-24 11:15:59

  • Status changed: Accepted
  • Labels removed: Status-Untriaged
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

We just upgraded to 2.42.2 and have experienced this issue as well.

Reported by chelseahoover on 2014-06-24 13:53:41

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

barancev, 
Could you please provide approximate extimates when v. 2.43 will be rolled out?
Thanks in advance.

Reported by grustniy.by on 2014-06-24 13:54:55

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

No estimates, sorry :)

If you're in hurry you can build selenium from the source code.
Just get the source code from the repository and run 'go release' command in the project
root.

See [1] for more info on the build system.

[1] https://code.google.com/p/selenium/wiki/BuildingWebDriver

Reported by barancev on 2014-06-24 14:03:54

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

we've been hit with this too, any updates on when we could expect to see 2.43 released
with a fix for this?

Thanks 

Reported by tomkessler3 on 2014-08-13 21:37:05

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Just to chime in that my environment is affected by this too.  

Nodes run v2.42.2 while the gridhub is running v2.41.0.  This seems to work for the
folks I support for now.

Reported by lmw94002 on 2014-08-20 12:44:35

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Was this resolved with the release of 2.43? I didn't see it in the Changlog

Reported by tomkessler3 on 2014-09-10 18:37:04

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Yes, I expect the updated version of httpclient included in 2.43 will fix the issue.
And I want someone having this issue to check if it fixed or not.

Reported by barancev on 2014-09-11 05:50:13

  • Status changed: NeedsClarification
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

We've just updated to 2.43.0 hub and nodes. There are no problems. Looks like fixed

Reported by rekruit19881 on 2014-09-11 08:22:37

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Some feedback on this - I had been running a custom build of 2.42.2 where i upgraded
the httpclient to the latest which fixed this issue. 

Also updated to the latest 2.43.1 two days ago, and this issue appears to have gone
away, as well as many other issues (specifically, i'm not seeing any more hanging browsers
pegging the CPU that never got killed by the grid). So yes i do believe as well that
it was the httpclient causing this.

Thanks for the great release guys!

Reported by mark@kabanasoft.com on 2014-09-12 17:38:34

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Thank you for getting us know!
Closing as fixed.

Reported by barancev on 2014-09-14 05:25:26

  • Status changed: Fixed
@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

unfortunately I still experience this with 2.43. I also had this with 2.42.2. I never
found a solution for this, so had to downgrade to 2.41 because of this problem.

Reported by trieszmxd on 2014-09-16 07:59:47

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Still an issue. 3 days of work are wasted trying to narrow down the issue in selenium-server-standalone.jar
version 2.42.2, 2.43.1. Downgraded to 2.41 for now.

Reported by vitaliy.grigoruk on 2014-09-22 20:23:10

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

same issue I am also facing, wholeday work wasted.

Reported by husainsoft on 2014-09-24 13:03:28

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I also still experience this issue using grid hub 2.43.1.  2.41.0 is rock solid stable
for me, even when using 2.43.1 nodes.

Reported by dprantl on 2014-09-24 14:49:02

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Anything new on this subject?

Reported by shiran.dadon@mail.huji.ac.il on 2014-10-26 13:51:57

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I somehow get this problem on version 2.41 as well. Could you send me a link maybe to
the working version? maybe I have a problematic Jar file...Thanks!

Reported by shiran.dadon@mail.huji.ac.il on 2014-10-26 14:25:55

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I am getting this issue application written with 2.43.1 and 2.41.0 Grid/Nodes.

Downgrading to 2.41 now.

Reported by jason.smiley@1stdibs.com on 2014-11-04 15:53:18

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

any updates to this ?  I am also stuck on 2.41, later versions do not work...

Reported by cfrjlr on 2014-12-03 22:56:03

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I am using 2.44.0 and everything seems to be great.

Reported by jason.smiley@1stdibs.com on 2014-12-03 22:59:23

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

2.44.0 works for some people but not for everybody ...  I am one of many stuck on 2.41

Reported by cfrjlr on 2014-12-04 22:25:58

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Workaround that helps me to get rid of this issue: play with different values and combinations
of having and not having timeouts for a hub and nodes. 
On some time it should work.

Reported by tatyana.tvardovskaya@muranosoft.com on 2014-12-08 15:05:33

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

This defect has been marked as fixed for 7 months now, even though the issue has not
been resolved and many people are still stuck at hub version 2.41.  Can someone please
reopen this defect?

Reported by dprantl on 2015-04-20 19:25:28

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Seeing it also, not resolved..

Reported by tylerbthrailkill on 2015-06-08 19:14:36

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

I would also consider issues with communication within the CI framework itself.

For instance, maybe this works with Jenkins when everything is run on the "master"
box. There could be issues with "slave" communications however since there are known
CI issues with Master and Slave relationship.

Reported by jason.smiley@1stdibs.com on 2015-06-08 19:17:52

@lukeis

This comment has been minimized.

Copy link
Member Author

@lukeis lukeis commented Mar 4, 2016

Reported by luke.semerau on 2015-09-17 18:22:29

  • Labels added: Restrict-AddIssueComment-Commit
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
1 participant
You can’t perform that action at this time.