firefox headless - Error: no DISPLAY environment variable specified

[jbnjohnathan]

When running the command pytest app/gui_tests/selenium/ --browser=firefox --headless -v -s in Docker I got the error from selenium: Exception: Message: invalid argument: can't kill an exited process

I tested to run firefox manually with firefox -headless and it seemed to work fine as it output *** You are running in headless mode.
So I checked the geckodriver.log for clues

I found this log entry

1600603147457   mozrunner::runner       INFO    Running command: "/usr/bin/firefox" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileLUeNt1"
Error: no DISPLAY environment variable specified

What struck me was that it wasn't using -headless as an argument to Firefox, which would explain why it would complain about the DISPLAY env not being set.
Is this a bug since I used the parameter --headless for selenium (and it seem to be default set as well). Or am I misunderstanding something?

Also as a side note. It would have been nice if the error from seleniumbase would have been nicer and included the Error: no DISPLAY environment variable specified message directly, instead of having to check the geckodriver.log

The annoying part is that it works perfectly fine in a Github Action container, but not when I create my own Dockerfile. Not sure what magic Github is doing when they create their own.
FYI When I run it in Github Actions the geckodriver.log file has this content instead

1600627799085	mozrunner::runner	INFO	Running command: "/usr/bin/firefox" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofile7Vv40c"
console.error: SearchCache: "_readCacheFile: Error reading cache file:" (new Error("", "(unknown module)"))
1600627814960	Marionette	INFO	Listening on port 35239
1600627815019	Marionette	WARN	TLS certificate errors will be ignored for this session
JavaScript error: resource://gre/modules/URIFixup.jsm, line 271: NS_ERROR_FAILURE: Should pass a non-null uri
1600627819117	Marionette	INFO	Stopped listening on port 35239

In docker:
seleniumbase==1.42.5 (tried upgrading to 1.49.16 but same issue)
pytest==5.4.3
python 3.8.2
firefox 80.0.1
Ubuntu 20.04
geckodriver = 2.44

[mdmintz]

Not sure why Docker is working differently from GitHub Actions. I did find https://stackoverflow.com/a/1095668 to take care of the display issue. See if that helps. I also noticed that your version of pytest isn’t consistent with the version that’s pinned to SeleniumBase. pytest should be on v6.0.2 when using Python 3. I’m currently away from my computer and am trying to answer as well as I can from my phone.

[mdmintz]

@jbnjohnathan , I figured out your issue. You weren't using the included Dockerfile with SeleniumBase. See https://github.com/seleniumbase/SeleniumBase/blob/master/Dockerfile (It sets the display parameter, which prevents your error.)

[jbnjohnathan]

Ah I see, didn't know they had one. I will try that one which will probably work : )
But I still think it is a pretty strange approach. So we want to trick Firefox into thinking we have a display, even though Firefox has a -headless setting which makes it not care of you have a display or not.
For me the most logical way would be for Selenium to set the -headless option on Firefox in case --headless is set in Selenium.
What do you think?

[jbnjohnathan]

A quick FYI in case someone else with a similar problem reads this.
Adding the below to my Dockerfile solved it

RUN apt-get -qy --no-install-recommends install xvfb

RUN set -e
RUN echo "Starting X virtual framebuffer (Xvfb) in background..."
RUN Xvfb -ac :99 -screen 0 1280x1024x16 > /dev/null 2>&1 &
RUN export DISPLAY=:99
RUN exec "$@"

My opinion that Selenium should set -headless option ton Firefox remains however, as then the above code and packages wouldn't be needed which would keep the Docker image smaller, and less complicated.

[mdmintz]

@jbnjohnathan SeleniumBase already sets the -headless option for Firefox headless mode.

The following is from https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/core/browser_launcher.py

options = webdriver.FirefoxOptions()
if headless:
    options.add_argument('-headless')
    firefox_capabilities['moz:firefoxOptions'] = (
        {'args': ['-headless']})

That wasn't enough for Docker builds for some reason. Hence the reason for the customized Dockerfile that's included: https://github.com/seleniumbase/SeleniumBase/blob/master/Dockerfile, which adds the DISPLAY parameter that is necessary.

[jbnjohnathan]

Ah okay, but shouldn't that be a command line parameter? From the log file I see this
1600603147457 mozrunner::runner INFO Running command: "/usr/bin/firefox" "-marionette" "-foreground" "-no-remote" "-profile" "/tmp/rust_mozprofileLUeNt1"
Nothing about -headless here that I can see.

[mdmintz]

@jbnjohnathan Are you able to debug to find out why the lines in https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/core/browser_launcher.py aren't setting it properly? This wouldn't happen with SeleniumBase's Dockerfile due to the display being set, but it's happening specifically with yours.

[jbnjohnathan]

Even if the DISPLAY is set I would assume Selenium to set the correct parameter for Firefox (-headless).
Have you seen it being set in the logfile when you run SeleniumBase?
Not sure right now how to debug it

[mdmintz]

@jbnjohnathan It's working for me. Here's output from the geckodriver.log file when running an example test using headless firefox:

1600711311971 mozrunner::runner INFO Running command: "/Applications/Firefox.app/Contents/MacOS/firefox-bin" "--marionette" "-headless" "-foreground" "-no-remote" "-profile" "/var/folders/r4/5fwh92896td3lp6cp_y0pplc0000gn/T/rust_mozprofileGOIJ19" *** You are running in headless mode.

There's something unusual about your Docker environment. Here's how you can debug it: Clone SeleniumBase, put a breakpoint in https://github.com/seleniumbase/SeleniumBase/blob/master/seleniumbase/core/browser_launcher.py before running a test in headless Firefox, and see what happens when stepping through the lines that start Firefox in headless mode. Place the breakpoint before the following lines:

options.add_argument('-headless')
firefox_capabilities['moz:firefoxOptions'] = (
    {'args': ['-headless']})

While in pdb debugging mode, you'll be able to detect the exact cause of what went wrong with your environment. Maybe it's a bug with Docker, or maybe it's something that can be improved with SeleniumBase. Right now, there are no issues with those using SeleniumBase's Dockerfile. For those using their own Dockerfile, there are no guarantees.