Skip to content
This repository has been archived by the owner on Sep 21, 2021. It is now read-only.

dosel/zalenium:3.0.11g - unexpected end of stream on Connection #560

Closed
tedel88 opened this issue May 8, 2018 · 21 comments
Closed

dosel/zalenium:3.0.11g - unexpected end of stream on Connection #560

tedel88 opened this issue May 8, 2018 · 21 comments

Comments

@tedel88
Copy link

tedel88 commented May 8, 2018

Zalenium Image Version(s):
3.0.11g
Docker Version:
18.03.0-ce
If using docker-compose, version:
2.1
OS:
Windows 10
Docker Command to start Zalenium:
docker-compose up

compose file:

version: '2.1'

services:
zalenium:
image: "dosel/zalenium:3.11.0g"
container_name: zalenium
hostname: zalenium
tty: true
volumes:
- ${AUTOTEST_VIDEOSPATH}:/home/seluser/videos
- /var/run/docker.sock:/var/run/docker.sock
ports:
- 4444:4444
command: >
start --desiredContainers 1
--maxDockerSeleniumContainers 5
--maxTestSessions 1
--screenWidth 1920 --screenHeight 1000
--timeZone "Europe/Berlin"
--videoRecordingEnabled true
--sauceLabsEnabled false
--browserStackEnabled false
--testingBotEnabled false
--keepOnlyFailedTests false
--startTunnel false
environment:
- HOST_UID
- HOST_GID

Expected Behavior

With Verison dosel/zalenium:3.0.11f Tests running without the excepion, should be the same for 3.0.11g

Actual Behavior

Exception:

Console Log:
[chrome #1] Failed: cannot forward the request unexpected end of stream on Connection{172.18.0.4:40000, proxy=DIRECT hostAddress=/172.18.0.4:40000 cipherSuite=none protocol=http/1.1}
[chrome #1] Stack:
[chrome #1] WebDriverError: cannot forward the request unexpected end of stream on Connection{172.18.0.4:40000, proxy=DIRECT hostAddress=/172.18.0.4:40000 cipherSuite=none protocol=http/1.1}
[....]

Container Log:
09:40:23.010 [qtp164332069-17] ERROR o.o.g.w.s.handler.RequestHandler - cannot forward the request unexpected end of stream on Connection{172.18.0.4:40000, proxy=DIRECT hostAddress=/172.18.0.4:40000 cipherSuite=none protocol=http/1.1}
java.io.IOException: unexpected end of stream on Connection{172.18.0.4:40000, proxy=DIRECT hostAddress=/172.18.0.4:40000 cipherSuite=none protocol=http/1.1}
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:205)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at org.openqa.selenium.remote.internal.OkHttpClient.execute(OkHttpClient.java:101)
at org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:430)
at org.openqa.grid.internal.TestSession.forward(TestSession.java:237)
at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:99)
at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:133)
at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:86)
at org.openqa.grid.web.servlet.DriverServlet.doPost(DriverServlet.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:860)
at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:170)
at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.seleniumhq.jetty9.server.Server.handle(Server.java:530)
at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:347)
at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:102)
at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 54 common frames omitted

Looks like "Caused by: java.io.EOFException: \n not found: limit=0 content=…" is the problem, isn't it?

@pearj
Copy link
Collaborator

pearj commented May 8, 2018

Can you try the test again?

The changes between 3.0.11f and 3.0.11g were quite minimal, I don’t think there is anything in the release that would break like you’re seeing.

@tedel88
Copy link
Author

tedel88 commented May 8, 2018

Hi,

tried it again with 3.0.11f and there was the same Error as with 3.0.11g.
Seems that the exception does not occur every time we start the tests with 3.0.11f..sorry for the misunderstanding.
Our productive Tests are running on 3.0.10a without the error.
Do you have any idea how to find out the specific verison without trying them all?

@pearj
Copy link
Collaborator

pearj commented May 8, 2018

What happens if you make —maxTestSessions 30 ? That will reuse docker containers

@pearj
Copy link
Collaborator

pearj commented May 8, 2018

I think in selenium 3.11 they changed from Apache http to okhttp. It might be related.

@tedel88
Copy link
Author

tedel88 commented May 8, 2018

I'm running 2 Tests on my local machine..so for reusing i need to configure:
--desiredContainers 1
--maxDockerSeleniumContainers 1
--maxTestSessions 2

?

@pearj
Copy link
Collaborator

pearj commented May 8, 2018

We’re running 20 concurrent containers quite regularly on Kubernetes and it’s rock solid on 3.11. So, maybe you’re running into resource problems? Maybe try dropping max containers to 2 and see how that goes. Or try disabling video recording as that uses a significant amount of cpu

@pearj
Copy link
Collaborator

pearj commented May 8, 2018

Yeah, that would mean it will never start more than 1 container, and after running 2 tests it will delete the container and create a new one to replace it.

@tedel88
Copy link
Author

tedel88 commented May 8, 2018

hmm when i try it with that parameters i get stuck at the second test :
[12:10:20] I/launcher - 1 instance(s) of WebDriver still running

@tedel88
Copy link
Author

tedel88 commented May 8, 2018

exception occurs with and without video recording and also when i drop max containers to 2.
I will try it severall times with 3.10c and 3.11a to see if it has something to do with okhttp

@tedel88
Copy link
Author

tedel88 commented May 8, 2018

Think i tracked it down to : dosel/zalenium:3.11.0d, maybe coincidence.

Any thoughts about :
Caused by: java.io.EOFException: \n not found: limit=0 content=…
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 54 common frames omitted
?

@pearj
Copy link
Collaborator

pearj commented May 9, 2018

Can you share you test class? As I can’t reproduce your problem.

@tedel88
Copy link
Author

tedel88 commented May 9, 2018

hmm i don't think that this is possible because of a quit complex testsetup with several non public docker containers. Tell you what : I will try to break it down to a minimal setup and publish it here.
Is that ok?
I think i could publish it on friday.

@pearj
Copy link
Collaborator

pearj commented May 10, 2018

I don't think it's related to 3.11.0d, that had very minimal code changes in the release:

3.11.0c...3.11.0d

Bonus points, if you can make it reproducible via a Vagrant file, so I can run it exactly the same.

@tedel88
Copy link
Author

tedel88 commented May 11, 2018

Unfortunately we are not using any of that cool Hashicoorp Stuff right now ;)

I've tried quiet a while now to reproduce it with a minimal setup but zalenium works fine.

@diemol
Copy link
Contributor

diemol commented May 23, 2018

After reading the thread, I understand that the issue is present when the complex setup is used, but with a simplified setup the issue is gone. Could this be related to how tests are started and executed?

For me, it sounds like we cannot do a lot to help. Do you have any more findings @tedel88?

@tedel88
Copy link
Author

tedel88 commented May 24, 2018

Hi @diemol,

yes maybe it is related to that. I will take a closer look if i have time.
I will open a new issue or reopen this one when i have a reproducible setup.

@tedel88 tedel88 closed this as completed May 24, 2018
@vkris6
Copy link

vkris6 commented May 29, 2018

Hi @tedel88 ,

I was seeing the exact same issue and i switch to zalenium 3.8.1c, but still saw the same issue. So does not look like zalenium issue as i had never encountered this before while using 3.8.1c. As @diemol pointed out Selenium switched to OkHttp for all HTTP communication. The version used can be changed back to the Apache HttpClient by setting the webdriver.http.factory system property to apache. Please refer to https://github.com/SeleniumHQ/selenium/blob/master/java/CHANGELOG

we can either use System.setProperty("webdriver.http.factory", "apache"); or Add property -Dwebdriver.http.factory=apache and test this. With the tests performed, I believe this is the issue. Am going test this too :-)

@tedel88
Copy link
Author

tedel88 commented May 29, 2018

Thanks @vkris6,

will try that as soon as possible :)

@ximbal
Copy link

ximbal commented Jun 6, 2018

I am having this issue.

14:22:00.997 [qtp68857875-20] ERROR o.o.g.w.s.handler.RequestHandler - cannot forward the request unexpected end of stream on Connection{172.17.0.4:40001, proxy=DIRECT hostAddress=/172.17.0.4:40001 cipherSuite=none protocol=http/1.1}
java.io.IOException: unexpected end of stream on Connection{172.17.0.4:40001, proxy=DIRECT hostAddress=/172.17.0.4:40001 cipherSuite=none protocol=http/1.1}
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:205)
at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:88)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:125)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200)
at okhttp3.RealCall.execute(RealCall.java:77)
at org.openqa.selenium.remote.internal.OkHttpClient.execute(OkHttpClient.java:105)
at org.openqa.grid.internal.TestSession.sendRequestToNode(TestSession.java:430)
at org.openqa.grid.internal.TestSession.forward(TestSession.java:237)
at org.openqa.grid.web.servlet.handler.RequestHandler.forwardRequest(RequestHandler.java:99)
at org.openqa.grid.web.servlet.handler.RequestHandler.process(RequestHandler.java:133)
at org.openqa.grid.web.servlet.DriverServlet.process(DriverServlet.java:86)
at org.openqa.grid.web.servlet.DriverServlet.doGet(DriverServlet.java:64)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.seleniumhq.jetty9.servlet.ServletHolder.handle(ServletHolder.java:860)
at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1650)
at io.prometheus.client.filter.MetricsFilter.doFilter(MetricsFilter.java:170)
at org.seleniumhq.jetty9.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.seleniumhq.jetty9.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.seleniumhq.jetty9.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.seleniumhq.jetty9.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.seleniumhq.jetty9.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.seleniumhq.jetty9.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.seleniumhq.jetty9.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.seleniumhq.jetty9.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.seleniumhq.jetty9.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.seleniumhq.jetty9.server.handler.StatisticsHandler.handle(StatisticsHandler.java:169)
at org.seleniumhq.jetty9.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.seleniumhq.jetty9.server.Server.handle(Server.java:530)
at org.seleniumhq.jetty9.server.HttpChannel.handle(HttpChannel.java:347)
at org.seleniumhq.jetty9.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.seleniumhq.jetty9.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.seleniumhq.jetty9.io.FillInterest.fillable(FillInterest.java:102)
at org.seleniumhq.jetty9.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.seleniumhq.jetty9.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.seleniumhq.jetty9.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.seleniumhq.jetty9.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.io.EOFException: \n not found: limit=0 content=…
at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:227)
at okhttp3.internal.http1.Http1Codec.readHeaderLine(Http1Codec.java:212)
at okhttp3.internal.http1.Http1Codec.readResponseHeaders(Http1Codec.java:189)
... 57 common frames omitted

Running latest Zalenium version (3.12.0c) on a Ubuntu machine with 32 Gb in Ram and 8 prc.

with the following params:

docker run --rm -ti --name elastic-selenium-grid -p 4444:4444 -e SAUCE_USERNAME -e SAUCE_ACCESS_KEY -v /var/run/docker.sock:/var/run/docker.sock -v /tmp/videos:/home/seluser/videos --privileged dosel/zalenium start --screenWidth 1280 --screenHeight 1024 --timeZone "America/Toronto" --desiredContainers 20 --videoRecordingEnabled false --maxDockerSeleniumContainers 25 --maxTestSessions 10

Any ideas on what might be happening?

@yuandanxu
Copy link

@diemol , I encountered this issue too after update to latest Zalenium. I have 5 zalenium nodes and in front of zalenium, I set up a ggr. I have 300 test cases, and 3 of them are failed due to this issue. Any suggestions?

@diemol
Copy link
Contributor

diemol commented Jun 7, 2018

Hi @ximbal and @yuandanxu

Can you please do what is mentioned in this comment? Please report your findings on #593

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

6 participants