3.0.x selenium SessionCleaner insideBrowserTimeout and clientGoneTimeout not working properly

[mzhang444]

Meta -

OS: OSX 10.12 Sierre
Selenium Version: 3.0.1 java standalone server, 3.0.0.beta2 javascript client binding
Browser: Safari 10.0 (12602.1.50.0.10)

Expected Behavior -

I thought insideBrowserTimeout matches browserTimeout, and clientGoneTimeout matches timeout in the hub and node configuration parameters.

1. If browserTimeout and timeout is configured in the selenium hubConfig.json file, Selenium node should pick up those values, and so are insideBrowserTimeout and clientGoneTimeout.
2. In selenium nodeConfig.json file, I set browserTimeout: 60000, timeout: 120000, the values are in milliseconds according to documentation. SessionCleaner should pick up these values
3. If the testing running inside browser finishes but not calling webdriver.quit(), the clientGoneTimeout should be triggered, session is destroyed and the browser instance is free to be used again

Actual Behavior -

1. From hub grid console, I can see the selenium node configuration has browserTimeout and timeout value set correctly from the hub configuration, but according to node.log, SessionCleaner still uses default insideBrowserTimeout (0) and clientGoneTimeout (1800000)
2. According to node.log, SessionCleaner uses insideBrowserTimeout 60000000 and clientGoneTimeout 120000000. Somehow times 1000 on the milliseconds again.
3. Session is destroyed after clientGoneTimeout expires. However, the hub doesn't seem to be notified of this event, so from the hub console, that browser instance is grey out and cannot be used.

Steps to reproduce -

1. Start hub with following command
   java -jar selenium-server-standalone-3.0.1.jar -role hub -log hub.log -port 4444 -host <ip> -hubConfig hubConfig.json, and hubConfig.json:
   { "browserTimeout": 60000, "timeout": 120000 }
2. Start node with following command
   java -jar selenium-server-standalone-3.0.1.jar -role node -log node.log -hub $HUB_URL -nodeConfig nodeConfig.json, and nodeConfig.json:
   { "capabilities": [ { "browserName":"safari", "maxInstances":1, "platform":"MAC", "version":"10", "seleniumProtocol": "WebDriver" } ], "debug": false, "proxy": "org.openqa.grid.selenium.proxy.DefaultRemoteProxy", "register": true, "registerCycle": 5000, "maxSession":6, "port":"7777" }
3. check node.log and you will see the following lines:
   11:40:18.427 INFO - SessionCleaner initialized with insideBrowserTimeout 0 and clientGoneTimeout 1800000 polling every 180000
4. add the following properties to nodeConfig.json and restart selenium node
   "browserTimeout": 60000, "timeout": 120000
5. check node.log and you will see the following lines:
   11:45:57.387 INFO - SessionCleaner initialized with insideBrowserTimeout 60000000 and clientGoneTimeout 60000000 polling every 6000000"
6. change browserTimeout and timeout in nodeConfig.json to the following and restart selenium node
   "browserTimeout": 60, "timeout": 120"
7. check node.log and you will see the following lines:
   11:52:21.122 INFO - SessionCleaner initialized with insideBrowserTimeout 60000 and clientGoneTimeout 120000 polling every 6000
8. Run a QUnit test but don't call webdriver.quit() when test finishes
9. After 120 seconds, browser is shutdown as expected due to client timeout and the following log can be seen in node.log:
   11:56:15.237 INFO - Session 4e364367-9f6e-43fc-8591-ad27c82506c9 deleted due to client timeout.

However, the hub doesn't seem to be aware of the session being deleted. From the hub console page, that instance remains grey out and cannot be used.

[lukeis]

you need to specify the cleanUpCycle on the hub too

[lukeis]

you need to specify the cleanUpCycle on the hub too

[mzhang444]

@lukeis The hub console shows cleanupCycle for the hub and the node is 5000, the default value. Do I need to specify it explicitly in the hubConfig and nodeConfig json?

Here is the hub configuration output from hub console web page:
updated with params :
browserTimeout : 60000
debug : false
help : false
jettyMaxThreads : -1
role : hub
timeout : 120000
cleanUpCycle : 5000
maxSession : 1
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : 30000
throwOnCapabilityNotPresent : true

[mzhang444]

@lukeis The hub console shows cleanupCycle for the hub and the node is 5000, the default value. Do I need to specify it explicitly in the hubConfig and nodeConfig json?

Here is the hub configuration output from hub console web page:
updated with params :
browserTimeout : 60000
debug : false
help : false
jettyMaxThreads : -1
role : hub
timeout : 120000
cleanUpCycle : 5000
maxSession : 1
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : 30000
throwOnCapabilityNotPresent : true

[derinbay]

@mzhang444 Found any solution to this? Same problem here. Instance remains grey out on selenium grid console and cannot be used.
@lukeis cleanUpCycle is set by default to 5000 and I can see it on grid console config.

[derinbay]

@mzhang444 Found any solution to this? Same problem here. Instance remains grey out on selenium grid console and cannot be used.
@lukeis cleanUpCycle is set by default to 5000 and I can see it on grid console config.

[Eyjafjallajokull]

I'm having the same issue using latest docker images (3.0.1).
I am manually setting -timeout and -browserTimeout when starting nodes (SE_OPTS), but still when node destroys session due to client timeout it has unavailable status in hub.

[Eyjafjallajokull]

I'm having the same issue using latest docker images (3.0.1).
I am manually setting -timeout and -browserTimeout when starting nodes (SE_OPTS), but still when node destroys session due to client timeout it has unavailable status in hub.

[Eyjafjallajokull]

Ok, I have figured this out! I've previously set timout for hub to 120000 but it seems that this value should be in seconds, not milliseconds. After changing timeout to 120, browser sessions were released in hub.

My final config:

Config for the hub :
browserTimeout : 120000
timeout : 120
cleanUpCycle : 5000
maxSession : 5
newSessionWaitTimeout : -1

[Eyjafjallajokull]

Ok, I have figured this out! I've previously set timout for hub to 120000 but it seems that this value should be in seconds, not milliseconds. After changing timeout to 120, browser sessions were released in hub.

My final config:

Config for the hub :
browserTimeout : 120000
timeout : 120
cleanUpCycle : 5000
maxSession : 5
newSessionWaitTimeout : -1

[derinbay]

@Eyjafjallajokull Are you sure it's not milliseconds? Is it going down after 2 minutes, or 1.2 seconss? I'll try this one also. If that's the case documentation needs to be update.

[derinbay]

@Eyjafjallajokull Are you sure it's not milliseconds? Is it going down after 2 minutes, or 1.2 seconss? I'll try this one also. If that's the case documentation needs to be update.

[lukeis]

It's seconds:

https://github.com/SeleniumHQ/selenium/blob/master/java/server/src/org/openqa/grid/internal/utils/configuration/StandaloneConfiguration.java#L118-L119

[lukeis]

It's seconds:

https://github.com/SeleniumHQ/selenium/blob/master/java/server/src/org/openqa/grid/internal/utils/configuration/StandaloneConfiguration.java#L118-L119

[derinbay]

All right, thanks @lukeis

[derinbay]

All right, thanks @lukeis

[derinbay]

Problem is still exist on me.
My node configuration is:

{
  "capabilities": [
    {
      "browserName": "firefox",
      "firefox_profile": "selenium",
      "platform": "WIN10",
      "maxInstances": 12
    },
    {
      "browserName": "chrome",
      "platform": "WIN10",
      "webdriver.chrome.driver": "path\\to\\chromedriver.exe",
      "maxInstances": 12
    }
  ],
  "configuration": {
    "port": 5555,
    "hub": "http://hubaddress/grid/register",
    "maxSession": 12
  }
}

and my hub is:
{
"browserTimeout": 300,
"timeout": 600
}

When a test has failed, it's remain as greyed out in console:

node configurtion seems like this on console:

and hub config is seems like this on console:

[derinbay]

Problem is still exist on me.
My node configuration is:

{
  "capabilities": [
    {
      "browserName": "firefox",
      "firefox_profile": "selenium",
      "platform": "WIN10",
      "maxInstances": 12
    },
    {
      "browserName": "chrome",
      "platform": "WIN10",
      "webdriver.chrome.driver": "path\\to\\chromedriver.exe",
      "maxInstances": 12
    }
  ],
  "configuration": {
    "port": 5555,
    "hub": "http://hubaddress/grid/register",
    "maxSession": 12
  }
}

and my hub is:
{
"browserTimeout": 300,
"timeout": 600
}

When a test has failed, it's remain as greyed out in console:

node configurtion seems like this on console:

and hub config is seems like this on console:

[mach6]

@derinbay
You can't use that -nodeConfig on a 3.0.0+ selenium node without getting this error;

Exception in thread "main" org.openqa.grid.common.exception.GridConfigurationException: Error with the JSON of the config : Deprecated -nodeConfig file encountered. Please update the file to work with Selenium 3. See https://github.com/SeleniumHQ/selenium/wiki/Grid2#configuring-the-nodes-by-json for more details.

you sure that is your node config?

[mach6]

@derinbay
You can't use that -nodeConfig on a 3.0.0+ selenium node without getting this error;

Exception in thread "main" org.openqa.grid.common.exception.GridConfigurationException: Error with the JSON of the config : Deprecated -nodeConfig file encountered. Please update the file to work with Selenium 3. See https://github.com/SeleniumHQ/selenium/wiki/Grid2#configuring-the-nodes-by-json for more details.

you sure that is your node config?

[derinbay]

@mach6 Yes, you can use -nodeConfig on selenium 3 also. Only json file is changed a bit. You can see the new file template from here: https://github.com/SeleniumHQ/selenium/blob/master/java/server/src/org/openqa/grid/common/defaults/DefaultNodeWebDriver.json

Simply remove the "configuration" tag from the file and just write the configurations at the end. Then it will be ok.

[derinbay]

@mach6 Yes, you can use -nodeConfig on selenium 3 also. Only json file is changed a bit. You can see the new file template from here: https://github.com/SeleniumHQ/selenium/blob/master/java/server/src/org/openqa/grid/common/defaults/DefaultNodeWebDriver.json

Simply remove the "configuration" tag from the file and just write the configurations at the end. Then it will be ok.

[mach6]

@derinbay yes, I know. I wrote that change ;)

my point is -- the config file you mentioned above for reproducing the issue is not a Se3 compatible one

[mach6]

@derinbay yes, I know. I wrote that change ;)

my point is -- the config file you mentioned above for reproducing the issue is not a Se3 compatible one

[derinbay]

@mach6 oh ok :)
Well, I don't get any exception with these configurations. Both hub and nodes are using selenium3. My only problem is browser is staying hanged when a problem occured on the test and session is deleted.

[derinbay]

@mach6 oh ok :)
Well, I don't get any exception with these configurations. Both hub and nodes are using selenium3. My only problem is browser is staying hanged when a problem occured on the test and session is deleted.

[mach6]

@derinbay back to my question.. are you sure the -nodeConfig you supplied above is the one you are using? it can't be... perhaps you using one that is equal and missing the configuration { } block? I'm trying to get enough details to attempt reproducing this. :)

[mach6]

@derinbay back to my question.. are you sure the -nodeConfig you supplied above is the one you are using? it can't be... perhaps you using one that is equal and missing the configuration { } block? I'm trying to get enough details to attempt reproducing this. :)

[derinbay]

@mach6 Yes, sorry I'm using the file without the configuration, selenium 3 one. I wrote the wrong old json file above (I'm switching files so much while trying). But insides are same except the configuration. and there is 1 extra "register": true line

{
  "capabilities": [
    {
      "browserName": "chrome",
      "platform": "WIN10",
      "webdriver.chrome.driver": "C:\\path\\to\\chromedriver.exe",
      "maxInstances": 12
    }
  ],
  "port": 5555,
  "hub": "http://hubaddress:4444/grid/register",
  "maxSession": 12,
  "register": true
}

[derinbay]

@mach6 Yes, sorry I'm using the file without the configuration, selenium 3 one. I wrote the wrong old json file above (I'm switching files so much while trying). But insides are same except the configuration. and there is 1 extra "register": true line

{
  "capabilities": [
    {
      "browserName": "chrome",
      "platform": "WIN10",
      "webdriver.chrome.driver": "C:\\path\\to\\chromedriver.exe",
      "maxInstances": 12
    }
  ],
  "port": 5555,
  "hub": "http://hubaddress:4444/grid/register",
  "maxSession": 12,
  "register": true
}

[Eyjafjallajokull]

@derinbay this json file is not valid there is one extra "}" in the end. Remove it and try again.

[Eyjafjallajokull]

@derinbay this json file is not valid there is one extra "}" in the end. Remove it and try again.

[derinbay]

@Eyjafjallajokull Thanks, I'm working with valid json file. I wrote typos here because I can't copy paste at the moment. The problem is not about json file validation, It couldn't even work if there is a validation error on json file : ) It's about the same problem as @mzhang444 mentioned.

[derinbay]

@Eyjafjallajokull Thanks, I'm working with valid json file. I wrote typos here because I can't copy paste at the moment. The problem is not about json file validation, It couldn't even work if there is a validation error on json file : ) It's about the same problem as @mzhang444 mentioned.

[mzhang444]

@derinbay I resolved problem number 3 by setting timeout: 120 and browserTimeout: 60 in the hub. These two properties are in seconds, while others such as registerCycle, cleanupCycle are in milliseconds. The unit used among properties are not consistent and very confusing.

Problem number 1 still exists but if we configure the hub properly, the hub will instruct the node to do the right thing.

[mzhang444]

@derinbay I resolved problem number 3 by setting timeout: 120 and browserTimeout: 60 in the hub. These two properties are in seconds, while others such as registerCycle, cleanupCycle are in milliseconds. The unit used among properties are not consistent and very confusing.

Problem number 1 still exists but if we configure the hub properly, the hub will instruct the node to do the right thing.

[mach6]

Confirmations and observations (assuming 3.x hub and nodes, only):

1. Yes, the configuration for timeout and browserTimeoutshould be specified in seconds.
2. The server code internally converts these to milliseconds and in some cases displays them as such. For Example: SessionCleaner INFO messages includes values which were internally converted using TimeUnit.SECONDS.toMillis(value)
3. There is a bug in all 3.0.0-beta1 <= x <= 3.0.1 versions where hub timeout and browserTimeout values are read but NEVER used to start a new node. Furthermore, the bug exists in such a fashion where the node registers to the hub with the hub values -- even though the node is not using them.
4. There is a bug and/or documentation issue preventing nodes from registering with timeout and browserTimeout values that override the hub values. This document suggests that this path is supported (though not recommended) -- https://github.com/SeleniumHQ/selenium/wiki/Grid2#configuring-timeouts-version-221-required.

As a work-around / because of 3 & 4, you need to sync your timeout and browserTimeout values across the hub and all nodes, specifying them in BOTH places during startup as config parameters.

I will provide fixes for 3 & 4 soon.

---

Observation update 1:

For SessionCleaner

• "clientGoneTimeout" = -timeout
• "insideBrowserTimeout" = -browserTimeout
• "polling every" is automatically computed and is equal to the lower non-zero value of timeout or browserTimeout converted to milliseconds and divided by 10.. E.g. timeout = 120 (sec) , browserTimeout=100 (sec) --> SessionCleaner polling interval will be (100 * 1000) / 10 = 10000 milliseconds (10 seconds)

[mach6]

Confirmations and observations (assuming 3.x hub and nodes, only):

1. Yes, the configuration for timeout and browserTimeoutshould be specified in seconds.
2. The server code internally converts these to milliseconds and in some cases displays them as such. For Example: SessionCleaner INFO messages includes values which were internally converted using TimeUnit.SECONDS.toMillis(value)
3. There is a bug in all 3.0.0-beta1 <= x <= 3.0.1 versions where hub timeout and browserTimeout values are read but NEVER used to start a new node. Furthermore, the bug exists in such a fashion where the node registers to the hub with the hub values -- even though the node is not using them.
4. There is a bug and/or documentation issue preventing nodes from registering with timeout and browserTimeout values that override the hub values. This document suggests that this path is supported (though not recommended) -- https://github.com/SeleniumHQ/selenium/wiki/Grid2#configuring-timeouts-version-221-required.

As a work-around / because of 3 & 4, you need to sync your timeout and browserTimeout values across the hub and all nodes, specifying them in BOTH places during startup as config parameters.

I will provide fixes for 3 & 4 soon.

---

Observation update 1:

For SessionCleaner

• "clientGoneTimeout" = -timeout
• "insideBrowserTimeout" = -browserTimeout
• "polling every" is automatically computed and is equal to the lower non-zero value of timeout or browserTimeout converted to milliseconds and divided by 10.. E.g. timeout = 120 (sec) , browserTimeout=100 (sec) --> SessionCleaner polling interval will be (100 * 1000) / 10 = 10000 milliseconds (10 seconds)

[ChrisAuto]

I have set both timeout 120 and browserTimeout 60 for the hub and nodes to be the same and double checked this.

When tests fail the grid console is showing that it is greyed out and the console (command window) always displays "Error - Unable to kill process with PID"

Failed tests are not releasing the browser instance in the Grid Console

[ChrisAuto]

I have set both timeout 120 and browserTimeout 60 for the hub and nodes to be the same and double checked this.

When tests fail the grid console is showing that it is greyed out and the console (command window) always displays "Error - Unable to kill process with PID"

Failed tests are not releasing the browser instance in the Grid Console

[ChrisAuto]

How do i get this fix, do i need to wait for the next build for selenium-server-standalone.jar ?

[ChrisAuto]

How do i get this fix, do i need to wait for the next build for selenium-server-standalone.jar ?

[mach6]

Re-opening.. Only point 3 in my comment above is fixed..

[mach6]

Re-opening.. Only point 3 in my comment above is fixed..

[shirandedan]

i wanna ask how you solve your problem at last..i meet the same problem as yours..

[shirandedan]

i wanna ask how you solve your problem at last..i meet the same problem as yours..

[derinbay]

@shirandedan I solved by updating selenium from 3.0.1 to newer one. There were some commits to fixing this one after 3.0.1.

[derinbay]

@shirandedan I solved by updating selenium from 3.0.1 to newer one. There were some commits to fixing this one after 3.0.1.

[shirandedan]

@derinbay but what i used is selenium 3.2.0 and selenium-server 3.2.0,the latest version.

[shirandedan]

@derinbay but what i used is selenium 3.2.0 and selenium-server 3.2.0,the latest version.

[shirandedan]

@derinbay i've also tried to set up the browserTimeout and tiemout,but just it didn't work!

[shirandedan]

@derinbay i've also tried to set up the browserTimeout and tiemout,but just it didn't work!

[shirandedan]

what should i do to solve it
Config for the hub :
browserTimeout : 0
debug : false
help : false
port : 4444
role : hub
timeout : 1800
cleanUpCycle : 5000
host : 192.168.199.146
maxSession : 40
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : -1
throwOnCapabilityNotPresent : true
Config for the hub :
browserTimeout : 0
debug : false
help : false
port : 4444
role : hub
timeout : 1800
cleanUpCycle : 5000
host : 192.168.199.146
maxSession : 40
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : -1
throwOnCapabilityNotPresent : true

[shirandedan]

what should i do to solve it
Config for the hub :
browserTimeout : 0
debug : false
help : false
port : 4444
role : hub
timeout : 1800
cleanUpCycle : 5000
host : 192.168.199.146
maxSession : 40
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : -1
throwOnCapabilityNotPresent : true
Config for the hub :
browserTimeout : 0
debug : false
help : false
port : 4444
role : hub
timeout : 1800
cleanUpCycle : 5000
host : 192.168.199.146
maxSession : 40
capabilityMatcher : org.openqa.grid.internal.utils.DefaultCapabilityMatcher
newSessionWaitTimeout : -1
throwOnCapabilityNotPresent : true

[ChrisAuto]

This is still not fixed even with update to selenium 3.3.1.
Upon a failed test, the node displays in console "unable to kill process with PID XXXX".
Grid Console still shows greyed out firefox icons for every failed test. When all slots are greyed out, no more connections are allowed and the entire suite fails.

Has anyone managed to get this working. Its been a problem since last year for me (upgrades to selenium 3.x). I never had this issue previously with lower versions of selenium with lower versions of firefox. I am using remotewebdriver (grid) with firefox but only with 1 concurrent thread for the moment and i still get this issue.

If somebody has managed to get grid/remotewbdriver on 3.3.1 working with failed tests then perhaps it would be useful to share the hub config and node config. My config is the same as shirandedan above.

I just need to kill the process when a failed test occurs which should also close the related firefox browser instance that remains idle.

[ChrisAuto]

This is still not fixed even with update to selenium 3.3.1.
Upon a failed test, the node displays in console "unable to kill process with PID XXXX".
Grid Console still shows greyed out firefox icons for every failed test. When all slots are greyed out, no more connections are allowed and the entire suite fails.

Has anyone managed to get this working. Its been a problem since last year for me (upgrades to selenium 3.x). I never had this issue previously with lower versions of selenium with lower versions of firefox. I am using remotewebdriver (grid) with firefox but only with 1 concurrent thread for the moment and i still get this issue.

If somebody has managed to get grid/remotewbdriver on 3.3.1 working with failed tests then perhaps it would be useful to share the hub config and node config. My config is the same as shirandedan above.

I just need to kill the process when a failed test occurs which should also close the related firefox browser instance that remains idle.