-
-
Notifications
You must be signed in to change notification settings - Fork 2.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.
Already on GitHub? Sign in to your account
[馃悰 Bug]: Grid 4 - Selenium hub crashes after a while #1439
Comments
Can you share how you start the whole Grid? You mention you have 10 Chrome nodes but I only see one in the script above. Also, what resources do you have available in the VM you are running all the containers? What tests could I use to reproduce the issue? |
I use the following commands to start the grid: I have 16GB memory on the VM. The same as with grid 3 which is not running out of memory. The graphic is showing the VM's resources. |
I've got similar condition with 50 chrome nodes and used a workaround. But have lower efficiency rather than had with 3rd version around 60%. |
Getting the same error with Grid4 having 30 chrome nodes.
|
Setup is in docker swarm mode. Below is the docker-compose.yaml. Running UI tests to generate load. Below are the VM configurations.
docker-compose.yaml
|
Hello, Images:selenium/hub:latest Error Message: There is insufficient memory for the Java Runtime Environment to continue. --------------- S Y S T E M --------------- OS:DISTRIB_ID=Ubuntu /proc/meminfo: >> docker statsCONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS Do let me know if any further info if required to troubleshoot the issue. TIA. |
Hi all, My docker-compose looks pretty much like the below (just change the {i} with the node number):
The problem After some time I find that:
While the nodes are running good I see the following from
I'll post back the logs for when the memory usage reaches 99% and nodes become unusable. Thank you, |
After 1/2 days in my case using Kubernetes in EKS (Amazon WS) my tests start to fail because they can't connect to the grid. |
I have retested this issue using the latest versions selenium-hub [4.1.3] and node-chrome [100.0]. Selenium grid still crashes although I did not see any out of memory exceptions. The grid UI will show loading and no new sessions are created. Already running tests will be hanging. The logs are full with this warning:
|
I have managed to understand the steps how to reproduce this issue mentioned in the last comment with the loading screen in the grid. Steps to repro:
I have added a project which you can use to reproduce the issue since it's not that straightforward. |
@lszasz thank you for the GitHub repository. Sadly, I am not able to reproduce the issue with it. However, the UI looks similar to what happens in #10485. Are you running a security scanning tool or something similar? |
Spoke too fast, seems I was able to reproduce it, please disregard my previous comment and I will let you know if more details are needed. |
The fix for this will be released in the next couple of days. |
Please @diemol let us know when you release this fix. |
It was already released. |
Thanks @diemol |
This is needed when a client sends a request, and we reply right away (e.g. session not found), but we do not read the whole request content. If we do not read the whole request content, Netty will wait until it does, and then it blocks. Which is not ideal since no more requests can be processed. Now, we close the pipeline when the client disconnects. Fixes SeleniumHQ#10485 Fixes SeleniumHQ/docker-selenium#1439
What happened?
We've been using Selenium Grid 3 for years and we are running small tests frequently. We have a hub and 10 nodes on the same machine which we start and stop once a day. After upgrading to Grid 4 the tests started failing after about 40 minutes most probably due to memory problems. We have reproduced this constantly.
Could you please look into it?
Command used to start Selenium Grid with Docker
Relevant log output
Operating System
Centos
Docker Selenium version (tag)
selenium/hub:4.0.0-20211025 and node-chrome:95.0-chromedriver-95.0-20211025
The text was updated successfully, but these errors were encountered: