Cannot use Firefox + Remote Selenium + Geckodriver Server in a docker container - Cannot recover from Timeout loading page

[nddipiazza]

System

• Version: 0.19.1, also tried the latest nightly geckodriver from https://queue.taskcluster.net/v1/task/B08EaV_xTv-mruOj4X8IiA/runs/0/artifacts/public/build/target.common.tests.zip
• Platform: Debian stretch docker container, Docker version 17.05.0-ce, build 89658be
  also tried a centos docker container, same issue
• Firefox: tried 58.0.2 (64-bit), firefox-59.0b13 and firefox-60.0a1.en-US.linux-x86_64
• Selenium: tried both 3.5.3, 3.9.1

Testcase

Full reproducer available https://github.com/nddipiazza/geckodriver-issue1193-reproducer

I also repeated this test using non-headless firefox with XVFB. Same issue.

Steps are below how to use it:

Pull docker instance, run it, open a bash

docker pull debian:stretch
docker run -t -d --name reproduce1193 debian:stretch
docker exec -i -t reproduce1193 /bin/bash

Install prereqs and run the reproducer

apt-get -y --fix-missing update && \
	apt-get -y --fix-missing upgrade && \
	apt-get -y --no-install-recommends install apt-utils && \
	apt-get install -y wget lbzip2 git openjdk-8-jdk libdbus-glib-1-2 xvfb libgtk-3.0 procps
git clone https://github.com/nddipiazza/geckodriver-issue1193-reproducer 
cd geckodriver-issue1193-reproducer
wget https://ftp.mozilla.org/pub/firefox/releases/58.0.2/linux-x86_64/en-US/firefox-58.0.2.tar.bz2
tar xvf firefox-58.0.2.tar.bz2
./gradlew build
java -jar ./build/libs/reproduce-marionette-timeouts-1.0-SNAPSHOT.jar firefox/firefox >>reproducer.log 2>&1

Stacktrace and trace level logging

Log when program is run from docker: https://gist.github.com/nddipiazza/088b874b575f85b847a4468424cde925

Log when program is run from non-docker (ubuntumate 16, 64-bit):
https://gist.github.com/nddipiazza/08484d65f1e9b2e814d2aca575b518ea

Main stack of interest seems to be:

###!!! [Parent][MessageChannel] Error: (msgtype=0x490017,name=PHttpChannel::Msg_DeleteSelf) Channel error: cannot send/recv


###!!! [Parent][MessageChannel] Error: (msgtype=0x490017,name=PHttpChannel::Msg_DeleteSelf) Channel error: cannot send/recv


###!!! [Parent][MessageChannel] Error: (msgtype=0x150084,name=PBrowser::Msg_Destroy) Channel error: cannot send/recv

1519915620166	Marionette	DEBUG	Register listener.js for window 17
A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
1519915620222	Marionette	DEBUG	Check readyState "loading for "about:tabcrashed?e=tabcrashed&u=https%3A//www.cnn.com/2012/08/14/tech/gallery/mars-curiosity-rover/index.html&c=UTF-8&f=regular&d=NASA%27s%20Mars%20Curiosity%20Rover"
1519915620244	Marionette	DEBUG	Received DOM event unload for [object XULDocument]
1519915620249	Marionette	TRACE	0 <- [1,5,null,{}]
1519915620548	geckodriver::marionette	TRACE	<- [1,5,null,{}]
1519915620548	webdriver::server	DEBUG	<- 200 OK {"value": {}}
1519915620550	webdriver::server	DEBUG	-> POST /session/0b2bbf88-c75d-4e08-bbf5-06622d7b3c8d/execute/sync {"script":"return document.readyState","args":[]}
1519915620550	geckodriver::marionette	TRACE	-> 133:[0,6,"executeScript",{"args":[],"newSandbox":false,"script":"return document.readyState","scriptTimeout":null,"specialPowers":false}]
1519915620554	webdriver::server	DEBUG	Deleting session
1519915620554	geckodriver::marionette	DEBUG	Stopping browser process
1519915620572	webdriver::server	DEBUG	<- 500 Internal Server Error {"value":{"error":"unknown error","message":"Failed to decode response from marionette","stacktrace":"stack backtrace:\n   0:           0x4edb3c - backtrace::backtrace::trace::hc4bd56a2f176de7e\n   1:           0x4edb72 - backtrace::capture::Backtrace::new::he3b2a15d39027c46\n   2:           0x4409a1 - webdriver::error::WebDriverError::new::h81babdd86c977032\n   3:           0x4295be - <webdriver::server::Dispatcher<T, U>>::run::h2119c674d7b88193\n   4:           0x4029b9 - std::sys_common::backtrace::__rust_begin_short_backtrace::h21d98a9ff86d4c25\n   5:           0x40be65 - std::panicking::try::do_call::h5cff0c9b18cfdbba\n   6:           0x5e6a6c - panic_unwind::__rust_maybe_catch_panic\n                        at /checkout/src/libpanic_unwind/lib.rs:99\n   7:           0x41eb22 - <F as alloc::boxed::FnBox<A>>::call_box::h413eb1d9d9f1c473\n   8:           0x5df13b - alloc::boxed::{{impl}}::call_once<(),()>\n                        at /checkout/src/liballoc/boxed.rs:692\n                         - std::sys_common::thread::start_thread\n                        at /checkout/src/libstd/sys_common/thread.rs:21\n                         - std::sys::imp::thread::{{impl}}::new::thread_start\n                        at /checkout/src/libstd/sys/unix/thread.rs:84"}}
Exception in thread "Thread-0" org.openqa.selenium.WebDriverException: Failed to decode response from marionette
Build info: version: 'unknown', revision: 'unknown', time: 'unknown'
System info: host: '077f9b6571c8', ip: '172.17.0.2', os.name: 'Linux', os.arch: 'amd64', os.version: '4.13.0-36-generic', java.version: '1.8.0_151'
Driver info: org.openqa.selenium.remote.RemoteWebDriver
Capabilities [{moz:profile=/tmp/rust_mozprofile.dTeNqQxqkjKr, rotatable=false, timeouts={implicit=0, pageLoad=300000, script=30000}, pageLoadStrategy=normal, moz:headless=true, platform=LINUX, moz:accessibilityChecks=false, acceptInsecureCerts=true, browserVersion=58.0.2, platformVersion=4.13.0-36-generic, moz:processID=10531, browserName=firefox, javascriptEnabled=true, platformName=LINUX, moz:webdriverClick=true}]
Session ID: 0b2bbf88-c75d-4e08-bbf5-06622d7b3c8d
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.createException(W3CHttpResponseCodec.java:185)
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:120)
	at org.openqa.selenium.remote.http.W3CHttpResponseCodec.decode(W3CHttpResponseCodec.java:49)
	at org.openqa.selenium.remote.HttpCommandExecutor.execute(HttpCommandExecutor.java:164)
	at org.openqa.selenium.remote.RemoteWebDriver.execute(RemoteWebDriver.java:646)
	at org.openqa.selenium.remote.RemoteWebDriver.executeScript(RemoteWebDriver.java:582)
	at Reproducer.lambda$waitForPageToBeReady$2(Reproducer.java:63)
	at org.openqa.selenium.support.ui.FluentWait.until(FluentWait.java:209)
	at Reproducer.waitForPageToBeReady(Reproducer.java:63)
	at Reproducer.loadUrls(Reproducer.java:158)
	at Reproducer.lambda$main$0(Reproducer.java:31)
	at java.lang.Thread.run(Thread.java:748)

You can clearly see the non-docker version is able to recover from this timeout.
But when on docker, it will not recover.

I also tried this same test with Selenium + Chromium + Chrome Driver. seems not to have the issue. so it's not selenium itself as far as I can tell.

[andreastt]

Thanks for the logs, but neither of them include the trace-level from geckodriver. Can you please attach some with the trace-level?

[nddipiazza]

oops! trace is on now @andreastt links are updated

[nddipiazza]

the memory setting as indicated here: #285 (comment) fixes the issue.

docker run -t -d --name reproduce1193 debian:stretch -> docker run -t -d --shm-size 2g --name reproduce1193 debian:stretch fixes the issue for me.

This ticket was opened to address the need for documentation here: https://bugzilla.mozilla.org/show_bug.cgi?id=1442356

[nddipiazza]

@andreastt kubernetes users don't get access to this setting. so feel free to re-open this ticket if you feel like this can be avoided in the first place.

[nddipiazza]

kubernetes/kubernetes#28272
is unfortunate. making this a tricky issue for us. but it does have a workaround.

[whimboo]

There is no way to get around that unless the above mentioned bug has been fixed.

So this issue is a dupe of #285.

[lock]

This issue has been automatically locked since there has not been any recent activity after it was closed. If you have run into an issue you think is related, please open a new issue.