[馃悰 Bug]: Selenium Hub unusable after idling for long period of time #1701
Labels
Comments
|
Seems you are bumping into #1605, can you please check and confirm? |
|
Yes, seems like the same steps to force it into this state. I will try with 4.6.0 once it is released. Thanks. |
|
How did it go? |
|
Closing this as we did not get more feedback. |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
What happened?
Issue is very similar to this closed issue #1439 and I was able to recreate it by using the same steps provided there.
Typically we run into this issue after leaving the hub idle for a few days. The same behavior occurs. The hub UI is visible but does not show any available sessions. Only a loading bar or an error message as shown in the images below.
Also another major problem with this is the the /status endpoint shows as ready and has all of the nodes showing as available. So the hub can not be easily restarted automatically if it is in this state.
The hub is still able to register new nodes but they also do not show on the /ui page.
We are running the hub as a Fargate service.
Command used to start Selenium Grid with Docker
{ "ipcMode": null, "executionRoleArn": "arn:aws:iam::acctid:role/selenium-chrome-Chrome1HubServiceselenium4albhubse-UMATAVK5A9WH", "containerDefinitions": [ { "dnsSearchDomains": [], "environmentFiles": [], "logConfiguration": { "logDriver": "awslogs", "secretOptions": [], "options": { "awslogs-group": "selenium-chrome-Chrome1HubServiceselenium4albhubservicehubtaskdefhubcontainerLogGroup4C3B4649-XuJrKy2IAYnK", "awslogs-region": "us-east-1", "awslogs-stream-prefix": "SeleniumStructuredAlbhub-logs" } }, "entryPoint": [], "portMappings": [ { "hostPort": 4444, "protocol": "tcp", "containerPort": 4444 }, { "hostPort": 5555, "protocol": "tcp", "containerPort": 5555 }, { "hostPort": 4443, "protocol": "tcp", "containerPort": 4443 }, { "hostPort": 4442, "protocol": "tcp", "containerPort": 4442 } ], "command": [], "linuxParameters": null, "cpu": 0, "environment": [ { "name": "GRID_TIMEOUT", "value": "180" }, { "name": "NETWORK_RELAX_CHECKS", "value": "true" }, { "name": "SE_START_XVFB", "value": "false" } ], "resourceRequirements": null, "ulimits": [], "dnsServers": [], "mountPoints": [], "workingDirectory": null, "secrets": [], "dockerSecurityOptions": [], "memory": null, "memoryReservation": null, "volumesFrom": [], "stopTimeout": null, "image": "rehosted-hub-image/selenium-hub-docker:0.1.12", "startTimeout": null, "firelensConfiguration": null, "dependsOn": null, "disableNetworking": null, "interactive": null, "healthCheck": null, "essential": true, "links": [], "hostname": null, "extraHosts": [], "pseudoTerminal": null, "user": null, "readonlyRootFilesystem": null, "dockerLabels": {}, "systemControls": [], "privileged": null, "name": "selenium-hub" } ], "placementConstraints": [], "memory": "16384", "taskRoleArn": "arn:aws:iam::acctid:role/selenium-chrome-Chrome1HubServiceselenium4albhubse-1CKCOBDW89E81", "compatibilities": [ "EC2", "FARGATE" ], "taskDefinitionArn": "arn:aws:ecs:us-east-1:504055967630:task-definition/seleniumchromeChrome1HubServiceselenium4albhubservicehubtaskdef5DCC1D45:5", "family": "seleniumchromeChrome1HubServiceselenium4albhubservicehubtaskdef5DCC1D45", "requiresAttributes": [ { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.logging-driver.awslogs" }, { "targetId": null, "targetType": null, "value": null, "name": "ecs.capability.execution-role-awslogs" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.ecr-auth" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.task-iam-role" }, { "targetId": null, "targetType": null, "value": null, "name": "ecs.capability.execution-role-ecr-pull" }, { "targetId": null, "targetType": null, "value": null, "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18" }, { "targetId": null, "targetType": null, "value": null, "name": "ecs.capability.task-eni" } ], "pidMode": null, "requiresCompatibilities": [ "FARGATE" ], "networkMode": "awsvpc", "runtimePlatform": null, "cpu": "4096", "revision": 5, "status": "ACTIVE", "inferenceAccelerators": null, "proxyConfiguration": null, "volumes": [] }Relevant log output
| 1666299647715 | 21:00:47.715 WARN [SeleniumSpanExporter$1.lambda$export$1] - Unable to execute request for an existing session: Unable to find session with ID: 964f51fa5e7874eb8397d817be23afe3 | | 1666299647715 | Build info: version: '4.4.0', revision: 'e5c75ed026a' | | 1666299647715 | System info: host: 'ip-172-24-158-248.ec2.internal', ip: '172.24.158.248', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.291-218.527.amzn2.x86_64', java.version: '11.0.16' | | 1666299647715 | Driver info: driver.version: unknown | | 1666299647715 | 21:00:47.715 WARN [SeleniumSpanExporter$1.lambda$export$1] - org.openqa.selenium.NoSuchSessionException: Unable to find session with ID: 964f51fa5e7874eb8397d817be23afe3 | | 1666299647715 | Build info: version: '4.4.0', revision: 'e5c75ed026a' | ... | 1666299942759 | 21:05:42.759 WARN [SeleniumSpanExporter$1.lambda$export$3] - {"traceId": "a994da1521cd079775b7323db08410a6","eventTime": 1666299942758013514,"eventName": "exception","attributes": {"exception.message": "Unable to execute request: Expected to read a START_MAP but instead have: END. Last 0 characters read: \nBuild info: version: '4.4.0', revision: 'e5c75ed026a'\nSystem info: host: 'ip-172-24-158-248.ec2.internal', ip: '172.24.158.248', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.291-218.527.amzn2.x86_64', java.version: '11.0.16'\nDriver info: driver.version: unknown","exception.stacktrace": "org.openqa.selenium.json.JsonException: Expected to read a START_MAP but instead have: END. Last 0 characters read: \nBuild info: version: '4.4.0', revision: 'e5c75ed026a'\nSystem info: host: 'ip-172-24-158-248.ec2.internal', ip: '172.24.158.248', os.name: 'Linux', os.arch: 'amd64', os.version: '4.14.291-218.527.amzn2.x86_64', java.version: '11.0.16'\nDriver info: driver.version: unknown\n\tat org.openqa.selenium.json.JsonInput.expect(JsonInput.java:307)\n\tat org.openqa.selenium.json.JsonInput.beginObject(JsonInput.java:229)\n\tat org.openqa.selenium.remote.NewSessionPayload.getOss(NewSessionPayload.java:306)\n\tat org.openqa.selenium.remote.NewSessionPayload.\u003cinit>(NewSessionPayload.java:84)\n\tat org.openqa.selenium.remote.NewSessionPayload.create(NewSessionPayload.java:118)\n\tat org.openqa.selenium.grid.data.SessionRequest.\u003cinit>(SessionRequest.java:65)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.lambda$new$0(NewSessionQueue.java:66)\n\tat org.openqa.selenium.remote.http.Route$TemplatizedRoute.handle(Route.java:192)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.sessionqueue.NewSessionQueue.execute(NewSessionQueue.java:120)\n\tat org.openqa.selenium.remote.tracing.SpanWrappedHttpHandler.execute(SpanWrappedHttpHandler.java:86)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.grid.router.Router.execute(Router.java:91)\n\tat org.openqa.selenium.grid.web.EnsureSpecCompliantResponseHeaders.lambda$apply$0(EnsureSpecCompliantResponseHeaders.java:34)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$NestedRoute.handle(Route.java:270)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.http.Route$CombinedRoute.handle(Route.java:336)\n\tat org.openqa.selenium.remote.http.Route.execute(Route.java:68)\n\tat org.openqa.selenium.remote.AddWebDriverSpecHeaders.lambda$apply$0(AddWebDriverSpecHeaders.java:35)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.remote.ErrorFilter.lambda$apply$0(ErrorFilter.java:44)\n\tat org.openqa.selenium.remote.http.Filter$1.execute(Filter.java:64)\n\tat org.openqa.selenium.netty.server.SeleniumHandler.lambda$channelRead0$0(SeleniumHandler.java:44)\n\tat java.base\u002fjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)\n\tat java.base\u002fjava.util.concurrent.FutureTask.run(FutureTask.java:264)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)\n\tat java.base\u002fjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)\n\tat java.base\u002fjava.lang.Thread.run(Thread.java:829)\n","exception.type": "org.openqa.selenium.json.JsonException","http.client_ip": "10.19.49.147","http.flavor": 1,"http.handler_class": "org.openqa.selenium.grid.sessionqueue.local.LocalNewSessionQueue","http.host": "selenium-hub-chrome-1.us-east-1.private.cvent-development.cvent.cloud:4444","http.method": "POST","http.request_content_length": "122","http.scheme": "HTTP","http.target": "\u002fsession","http.user_agent": "webdriver\u002f7.19.7"}} | -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------Operating System
Docker Hub image in Fargate
Docker Selenium version (tag)
4.4.0
The text was updated successfully, but these errors were encountered: