Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connection problem and TIME_WAIT #878

Closed
chandon opened this issue Aug 4, 2015 · 7 comments
Closed

Connection problem and TIME_WAIT #878

chandon opened this issue Aug 4, 2015 · 7 comments
Labels

Comments

@chandon
Copy link

@chandon chandon commented Aug 4, 2015

I'm using Selenium Java 2.47.1 and Firefox 39.0 on Windows.

While crawling my website, i do a lot of "findElement" and other commands with Selenium.
It seems that Selenium isn't using persistent connections (perhaps it's not possible).

After plenty commands, i have this kind of error :

août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:08 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:10 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: I/O exception (java.net.BindException) caught when connecting to {}->http://127.0.0.1:7055: Address already in use: connect
août 04, 2015 11:23:12 AM org.apache.http.impl.client.DefaultHttpClient tryConnect
INFOS: Retrying connect to {}->http://127.0.0.1:7055
août 04, 2015 11:23:12 AM org.openqa.selenium.support.ui.ExpectedConditions findElement
AVERTISSEMENT: WebDriverException thrown by findElement(By.xpath: //ul[contains(@class,"pagination")]//li[not(a) and (contains(text(),"56"))])
org.openqa.selenium.remote.UnreachableBrowserException: Error communicating with the remote browser. It may have died.
Build info: version: '2.41.0', revision: '3192d8a6c4449dc285928ba024779344f5423c58', time: '2014-03-27 11:29:39'
System info: host: 'Cedric-PC', ip: '169.254.185.162', os.name: 'Windows 7', os.arch: 'amd64', os.version: '6.1', java.version: '1.8.0_20'
Driver info: driver.version: RemoteWebDriver
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:589)
    at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:348)
    at org.openqa.selenium.remote.RemoteWebDriver.findElementByXPath(RemoteWebDriver.java:445)
    at org.openqa.selenium.By$ByXPath.findElement(By.java:357)
    at org.openqa.selenium.remote.RemoteWebDriver.findElement(RemoteWebDriver.java:340)
    at org.openqa.selenium.support.ui.ExpectedConditions.findElement(ExpectedConditions.java:861)
    at org.openqa.selenium.support.ui.ExpectedConditions.access$000(ExpectedConditions.java:41)
    at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:181)
    at org.openqa.selenium.support.ui.ExpectedConditions$6.apply(ExpectedConditions.java:178)
    at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:209)
    ...
Caused by: java.net.BindException: Address already in use: connect
    at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
    at java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:85)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.apache.http.conn.scheme.PlainSocketFactory.connectSocket(PlainSocketFactory.java:117)
    at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
    at org.apache.http.impl.conn.AbstractPoolEntry.open(AbstractPoolEntry.java:144)
    at org.apache.http.impl.conn.AbstractPooledConnAdapter.open(AbstractPooledConnAdapter.java:131)
    at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
    at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
    at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:71)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
    at org.openqa.selenium.remote.HttpCommandExecutor.fallBackExecute(HttpCommandExecutor.java:340)
    at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:301)
    at org.openqa.selenium.firefox.internal.NewProfileExtensionConnection.execute(NewProfileExtensionConnection.java:170)
    at org.openqa.selenium.firefox.FirefoxDriver$LazyCommandExecutor.execute(FirefoxDriver.java:393)
    at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:568)
    ... 17 more

According to my netstat, i got around 16350 connections in TIME_WAIT state :

TCP    127.0.0.1:49218        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49219        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49220        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49221        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49222        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49223        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49224        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49225        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49226        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49227        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49228        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49229        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49230        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49231        127.0.0.1:7056         TIME_WAIT       0
TCP    127.0.0.1:49232        127.0.0.1:7056         TIME_WAIT       0
...

Running the windows command to get dynamics port range :

netsh int ipv4 show dynamicport tcp
TPC Dynamics ports
-------------------------------------------
Starting port   : 49152
Ports count     : 16384

A simple way to reproduce it (with a fast enough computer, it stops around 16384 on windows...) :

public void testTimeWait() {
    FirefoxDriver webDriver = new FirefoxDriver();  

    webDriver.get("http://www.google.fr");

    for (int i=0;i<1000000;++i) {
        String a=webDriver.getTitle();
        if (i%10==0) {System.out.println(i);}       
    }
}

It seems that i don't have any Dynamic port available to establish a new connection... and i think the problem will be the same with Linux (TIME_WAIT is usualy 60s).

Of course, i can increase the dynamic ports count, but it'll not solve my problem (because i can't get more than 65535 ports).

So my question : is there a way or a parameter to reduce the connections count (persistent connections ?) or the TIME_WAIT connection count ?

@chandon chandon changed the title Connection problem to Firefox Connection problem and TIME_WAIT Aug 4, 2015
@chandon

This comment has been minimized.

Copy link
Author

@chandon chandon commented Aug 4, 2015

After some investigation, Selenium uses Apache HttpClient wich support persistent connections.

But for firefox, httpd.js (the Firefox Httpd server for unit tests) doesn't support it...
An issue (and some fixes) open for a long time now :

https://bugzilla.mozilla.org/show_bug.cgi?id=469228

@barancev barancev added the C-java label Sep 20, 2015
@ygmarchi

This comment has been minimized.

Copy link

@ygmarchi ygmarchi commented Oct 13, 2015

I think it's weird that a separate http connection is used for each command instead of a single plain socket for all commands issued by the same RemoteWebDriver instance.

@chandon

This comment has been minimized.

Copy link
Author

@chandon chandon commented Oct 13, 2015

In my case, it's not a problem with RemoteWebDriver (which seems to be able to use a single socket) but a problem with Firefox. A patch was submited to Mozilla 4 years ago, but the Mozilla community doesn't seem to care

@krosenvold

This comment has been minimized.

Copy link
Member

@krosenvold krosenvold commented Oct 13, 2015

As you will see, I tried patching httpd.js for firefox some time ago, but we had problems because we were supporting a too old version of firefox at the time. If anyone wants to have a go again, I'll happily review it.

@lukeis

This comment has been minimized.

Copy link
Member

@lukeis lukeis commented Oct 13, 2015

@krosenvold we do have our own internal copy of httpd.js that you could patch :)

Although not sure if it'll matter much in the long run. Firefox is soon going to require extension signing by AMO and Mozilla has now said they have no intention of signing our extension. This leaves us with only being able to use marionette for future versions.

@krosenvold

This comment has been minimized.

Copy link
Member

@krosenvold krosenvold commented Oct 14, 2015

I probably have the old version I patched somewhere anyway, but given the current situation I dont think this is worth the effort :)

@barancev

This comment has been minimized.

Copy link
Member

@barancev barancev commented Feb 1, 2016

This issue will definetely remain not fixed in FirefoxDriver. Let's hope Marionette will utilise sockets more carefully.

@barancev barancev closed this Feb 1, 2016
@lock lock bot locked and limited conversation to collaborators Aug 20, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
5 participants
You can’t perform that action at this time.