Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New chrome:92 image flakiness with selenoid 1.10.4 #1124

Closed
kornel-filep opened this issue Jul 26, 2021 · 22 comments
Closed

New chrome:92 image flakiness with selenoid 1.10.4 #1124

kornel-filep opened this issue Jul 26, 2021 · 22 comments
Labels
Milestone

Comments

@kornel-filep
Copy link

Hi!
Since the friday release of selenoid 1.10.4 and chrome 92.0 browser image we experience some flakyness when executing tests.

The chrome 92.0 image does not work with selenoid 1.10.3 at all, resulting in a WebDriverError: Host header or origin header is specified and is not whitelisted or localhost. error.

When running selenoid 1.10.4 with the chrome 92.0 image, we see tests fail randomly with WebDriverError: New session attempts retry count exceeded. This seems like the new image might not work as expected.
We have added a retry count to affected tests, but that does not fix fully the issue, sometimes it still can happen.

Any ideas of how to correct this issue? Maybe the image was not built properly and a rebuild is required?

@kornel-filep
Copy link
Author

Selenoid logs:

2021/07/26 09:52:12 [-] [NEW_REQUEST] [unknown] [172.18.0.1]

2021/07/26 09:52:12 [-] [NEW_REQUEST_ACCEPTED] [unknown] [172.18.0.1]

2021/07/26 09:52:12 [13] [LOCATING_SERVICE] [chrome] [92.0]

2021/07/26 09:52:12 [13] [USING_DOCKER] [chrome] [92.0]

2021/07/26 09:52:12 [13] [CREATING_CONTAINER] [selenoid/chrome:92.0]


2021/07/26 09:52:12 [13] [STARTING_CONTAINER] [selenoid/chrome:92.0] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf]


2021/07/26 09:52:12 [13] [CONTAINER_STARTED] [selenoid/chrome:92.0] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf] [0.58s]


2021/07/26 09:52:13 [13] [SERVICE_STARTED] [selenoid/chrome:92.0] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf] [0.52s]

2021/07/26 09:52:13 [13] [PROXY_TO] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf] [http://172.18.0.4:4444/]


2021/07/26 09:52:13 [13] [SESSION_ATTEMPTED] [http://172.18.0.4:4444/] [1]


2021/07/26 09:52:43 [13] [SESSION_ATTEMPT_TIMED_OUT] [30s]


2021/07/26 09:52:43 [13] [SESSION_FAILED] [http://172.18.0.4:4444/] [New session attempts retry count exceeded]

2021/07/26 09:52:43 [13] [REMOVING_CONTAINER] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf]


2021/07/26 09:52:43 [13] [CONTAINER_REMOVED] [9b64a3113b5bb0cb3ea3a768ce557a418fb882ad87525c22115b3e8849a0c5bf]

I have done this using selenoid 1.10.4, chrome:92 image provided by aerokube and requested the container using selenoid-ui.

@kornel-filep
Copy link
Author

kornel-filep commented Jul 26, 2021

When having a successful container start after several crashes/timeouts with retry count 10, in the browser container logs I see this:



--- x11vnc loop: 1 ---


--- x11vnc loop: waiting for: 188



20

Logging to: /dev/null

Waiting X server...

Waiting X server...

Starting ChromeDriver 92.0.4515.43 (8c61b7e2989f2990d42f859cac71319137787cce-refs/branch-heads/4515@{#306}) on port 4444

All remote connections are allowed. Use an allowlist instead!

Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.

ChromeDriver was started successfully.

[89:134:0726/120104.552793:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:144:0726/120104.577713:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[89:144:0726/120104.577927:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[89:118:0726/120104.608962:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.609042:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 1 time(s)

[89:144:0726/120104.623251:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[89:144:0726/120104.623475:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[89:118:0726/120104.649457:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.649519:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 2 time(s)

[89:118:0726/120104.706669:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.706727:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 3 time(s)

[89:118:0726/120104.741710:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.741778:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 4 time(s)

[89:189:0726/120104.752783:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:189:0726/120104.753006:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:189:0726/120104.753093:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:189:0726/120104.754468:WARNING:property.cc(144)] DaemonVersion: GetAndBlock: failed.

[89:189:0726/120104.755148:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:189:0726/120104.755418:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[89:118:0726/120104.779421:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.779494:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 5 time(s)

[89:118:0726/120104.812711:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[89:118:0726/120104.812815:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 6 time(s)

PORT=5900

[192:192:0726/120104.834923:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled

[89:118:0726/120104.843904:WARNING:gpu_process_host.cc(1023)] Reinitialized the GPU process after a crash. The reported initialization time was 0 ms

[203:235:0726/120124.474355:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:244:0726/120124.499148:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[203:244:0726/120124.499212:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[203:244:0726/120124.529313:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[203:244:0726/120124.529405:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[203:232:0726/120124.547482:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.547517:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 1 time(s)

[203:290:0726/120124.607602:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:290:0726/120124.607700:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:290:0726/120124.607737:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:290:0726/120124.607741:WARNING:property.cc(144)] DaemonVersion: GetAndBlock: failed.

[203:290:0726/120124.607755:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:290:0726/120124.607769:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[203:232:0726/120124.636434:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.636488:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 2 time(s)

[203:232:0726/120124.663584:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.663680:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 3 time(s)

[203:232:0726/120124.691195:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.691268:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 4 time(s)

[203:232:0726/120124.718748:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.718821:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 5 time(s)

[203:232:0726/120124.745892:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[203:232:0726/120124.745992:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 6 time(s)

[300:300:0726/120124.757238:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled

[203:232:0726/120124.768615:WARNING:gpu_process_host.cc(1023)] Reinitialized the GPU process after a crash. The reported initialization time was 0 ms

[309:339:0726/120144.812225:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[309:337:0726/120144.883823:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[309:337:0726/120144.883867:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 1 time(s)

[309:351:0726/120144.900988:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[309:351:0726/120144.901469:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[309:337:0726/120144.967360:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[309:337:0726/120144.967422:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 2 time(s)

[309:351:0726/120144.974721:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[309:351:0726/120144.974908:ERROR:bus.cc(393)] Failed to connect to the bus: Address does not contain a colon

[309:337:0726/120144.992832:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[309:337:0726/120144.992893:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 3 time(s)

[309:337:0726/120145.013862:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[309:337:0726/120145.013911:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 4 time(s)

[309:337:0726/120145.044290:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4

[309:337:0726/120145.044312:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 5 time(s)

[309:337:0726/120145.082080:ERROR:gpu_process_host.cc(995)] GPU process exited unexpectedly: exit_code=4


[309:337:0726/120145.082129:WARNING:gpu_process_host.cc(1308)] The GPU process has crashed 6 time(s)

[384:384:0726/120145.104155:ERROR:gpu_init.cc(441)] Passthrough is not supported, GL is disabled

[384:384:0726/120145.106357:ERROR:sandbox_linux.cc(374)] InitializeSandbox() called with multiple threads in process gpu-process.

[309:337:0726/120145.114255:WARNING:gpu_process_host.cc(1023)] Reinitialized the GPU process after a crash. The reported initialization time was 0 ms

[309:399:0726/120145.222455:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[309:399:0726/120145.222546:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[309:399:0726/120145.222575:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[309:399:0726/120145.222579:WARNING:property.cc(144)] DaemonVersion: GetAndBlock: failed.

[309:399:0726/120145.222593:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory

[309:399:0726/120145.222606:ERROR:bus.cc(393)] Failed to connect to the bus: Failed to connect to socket /var/run/dbus/system_bus_socket: No such file or directory 

I hope it is somewhat useful for debugging the issue.

@kornel-filep
Copy link
Author

More info:
It seems that this container flakiness is not present if we request a headless chrome from selenoid.

@vania-pooh vania-pooh added the bug label Jul 26, 2021
@vania-pooh vania-pooh added this to the 1.10.5 milestone Jul 26, 2021
@vania-pooh
Copy link
Member

Let's try to add Host: localhost to all requests.

@kornel-filep
Copy link
Author

So from yesterday I can confirm that the container manages to start if we start a headless chrome.
Somehow the issue is with headful chrome 92.0.

@rodrigoodhin
Copy link

Let's try to add Host: localhost to all requests.

I have the same issue. Where I have to put "Host: localhost" ?
I only get this error on chrome.
I've tested on Firefox and Opera and works fine.

@vania-pooh
Copy link
Member

Please check whether it works with aerokube/selenoid:latest image.

@kornel-filep
Copy link
Author

Sorry, still seems to be unstable, tried with the new aerokube/selenoid:latest (double checked, i am 100% I used the new one). Using the chrome 92.0 image, I ran 2 times 20 tests which run sequentially and request an instance on test start, and destroy the container on test end. Sadly both times there were cases where the instance creation timed out just like before.

@vania-pooh
Copy link
Member

Then this is not related to Host header. Should be related to GPU then.

@OHaimanov
Copy link

Is it any updates? Have the same issue

@vania-pooh
Copy link
Member

@OHaimanov the only solution right now is to set disable-gpu flag.

@OHaimanov
Copy link

@OHaimanov the only solution right now is to set disable-gpu flag.

For me helped to use chrome 91.0 instead of 92.0.

@kornel-filep
Copy link
Author

@OHaimanov the only solution right now is to set disable-gpu flag.

For me helped to use chrome 91.0 instead of 92.0.

Of course that helps, but that ruins the point of using the latest and greatest chrome and testing against it.

@edurb
Copy link

edurb commented Aug 6, 2021

Hi is this issue resolved, i am still seeing issues when i use chrome 92. I keep getting
java.lang.UnsupportedOperationException: Connecting to remote URL 'http://localhost:4444/wd/hub/' failed: New session attempts retry count exceeded

@vania-pooh
Copy link
Member

@edurb did you try to add disable-gpu flag in capabilities?

@symonk
Copy link

symonk commented Aug 9, 2021

same here, unable to upgrade to 92.0 chrome nodes, it's currently unusable, stemming from:

selenium.common.exceptions.WebDriverException: Message: New session attempts retry count exceeded

@edurb
Copy link

edurb commented Aug 9, 2021

i had to go back to older version of browser because of this 92 issues. but again i am seeing seeing now with #1129

@vania-pooh
Copy link
Member

@edurb we are going to rebuild Chrome 92.0 image with disable-gpu flag soon.

@vania-pooh
Copy link
Member

Rebuilt Chrome 92.0 image with --disable-gpu flag. Please try now.

@daroshchanka
Copy link

Rebuilt Chrome 92.0 image with --disable-gpu flag. Please try now.

Works for me with the new selenoid/vnc_chrome:92.0 and latest aerokube/selenoid.

Also works for edge 92, 93.

@symonk
Copy link

symonk commented Aug 16, 2021

seems better @vania-pooh

@vania-pooh
Copy link
Member

Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants