Certain exceptions cause WebDriver to become unresponsive

[Thenakedgun]

🐛 Bug Report

Under certain circumstances when an Exception is thrown within Python in seemingly unrelated code, the Firefox/WebDriver instance becomes unresponsive & it is not possible to close open Firefox instances.

When called, driver.quit() returns None, but does not close the active Firefox instance. Additionally, driver.close() also doesn't close the browser instance & instead produces the following error:

urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='127.0.0.1', port=61067): Max retries exceeded with url: /session/5385f593-7782-4073-996e-b226f0e9dc4e/window (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x00000114A0CE05F8>: Failed to establish a new connection: [WinError 10061] No connection could be made because the target machine actively refused it',))

If in the "Test Script" below, I enter either a valid or invalid URI, then the browser closes as expected. In the event of an invalid URI, selenium throws a selenium.common.exceptions.InvalidArgumentException, but this is caught & terminates normally. Similarly, if I replace the input statement with, raise KeyboardInterrupt() or raise EOFError() then the browser is also closed as expected.

Using pdb, the expected path of execution seems to be followed. I posted this on SO yesterday to see if I was just expecting / doing something incorrectly, but the post didn't get any bites: https://stackoverflow.com/questions/54069341/exception-prevents-webdriver-from-responding-properly

To Reproduce

Run the "Test Script" below; this example seems to be reflective of the issue I'm running into.

When prompted for input, rather the entering a string, press Ctrl+C to send the SIGINT. During the input statement on my machine, this raises an EOFError().

Expected behavior

Execution should be halted when the EOFError is raised & flow resumes at the except statement. It will then pass, and continues in the finally statement & the browser instance is closed by either driver.close() or driver.quit().

Test script or set of commands reproducing this issue

from selenium import webdriver
import pdb

try:
	#pdb.set_trace()
	driver = webdriver.Firefox(executable_path='./bin/geckodriver.exe')
	variable = input("Enter something or Ctrl+C: ")
	driver.get(variable)
except Exception as e:
	pass
finally:
	driver.close()
	driver.quit()

Within my real code environment, I've wrapped my script in a similar try except finally statement, and similarly, after certain exceptions are thrown in seemingly unrelated statements the driver object stops responding as expected. This seemed to be the minimum amount of code which produced the same error.

Environment

OS: Windows 10
Browser: Firefox
Browser version: 64.0
Browser Driver version: GeckoDriver v0.23.0
Language Bindings version: Python 3.6.4
Selenium Version: 3.141.0

[diemol]

@Thenakedgun I am not working with the Python bindings, but after reading the issue, I can see that there is a big difference between the version of the language bindings and the Selenium version, these should be in synch.
Perhaps try that first and report back your results.

[Thenakedgun]

@diemol K, just updated to Python 3.7.2 which seems to be the most recent version for Windows. I reinstalled selenium via pip, double-checked the Python version it was going to run under, and re-ran the Test Script.

Unfortunately I arrived at the exact same result-- the browser instance can't be terminated in the finally section. Tried swapping the order of driver.close and driver.quit & it made no difference. Again, driver.close() throws the error, where driver.quit seemingly does nothing.

Here's the geckodriver.log file for the sake of it:
geckodriver.log

[diemol]

@Thenakedgun I meant the Python language bindings, you mentioned:

Language Bindings version: Python 3.6.4
and the latest one should be around 3.141.59

[Thenakedgun]

@diemol Sorry, could you point me to the resource? Or are you saying try an older Geckodriver / Selenium install for working in Python at the moment?

I installed selenium via pip and 3.141.0 seems to be the most up-to-date available there. I went into the github repo & it seems to list this version as the most recent changeset as well: selenium/py/CHANGES ... Selenium 3.141.0

Pseudo-relatedly, on the SeleniumHQ.org website, the language bindings listed only showed Python up to a slightly older version.

Language	Client Version	Release Date
Java	3.141.59	2018-11-14	Download	Change log	Javadoc
C#	3.14.0	2018-08-02	Download	Change log	API docs
Ruby	3.14.0	2018-08-03	Download	Change log	API docs
Python	3.14.0	2018-08-02	Download	Change log	API docs

[diemol]

You are right, the latest Python bindings version is 3.141.0

I see that you are getting this in the log:

1547132933531	mozrunner::runner	INFO	Running command: "C:\\Program Files\\Mozilla Firefox\\firefox.exe" "-marionette" "-foreground" "-no-remote" "-profile" "C:\\Users\\ETOOLL~1\\AppData\\Local\\Temp\\rust_mozprofile.fA9HxLJFTsgi"
1547132933888	addons.webextension.screenshots@mozilla.org	WARN	Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: resource://pdf.js/
1547132933888	addons.webextension.screenshots@mozilla.org	WARN	Loading extension 'screenshots@mozilla.org': Reading manifest: Invalid host permission: about:reader*

Looks like you are taking a screenshot, It seems something related to GeckoDriver, but I cannot confirm. Best would be to provide first a concise reproducible test case, that includes the page required to execute the scenario (a sample page or a public site), then @AutomatedTester or @whimboo could confirm if it is an issue in GeckoDriver or not.

[AutomatedTester]

It looks like Firefox is crashing from that log but there isnt much there. Could you enable the tracelog and add a link to that please

https://firefox-source-docs.mozilla.org/testing/geckodriver/geckodriver/TraceLogs.html

[Thenakedgun]

@diemol I'm not sure what the purpose of the "screenshot" extension comments were in the logfile-- what was being run was the ~10 line "Test Script" specified in the initial post; no screenshots were attempted.

@AutomatedTester Yep, enabled it, and here's the log: geckodriver.log

Ctrl+C was pressed in response to the input statement after ensuring Firefox had completely finished launching. This was the complete program, updated with the specified firefox_options (with the extension changed as the file was blocked): test.txt

Finally, this was the full console output: ConsoleOutput.txt

Incidentally, I had a friend test the same code. He had the same Firefox, Selenium & Geckodriver versions, but is running Windows 7 & Python 3.5. He received a different error, which was raised when the code reached driver.quit() but Firefox was similarly left open.

[whimboo]

The screenshot log entries are totally unrelated here. Those come from the screenshot extension in Firefox. Further note that your example code is wrong given that KeyboardInterrupt is derived from BaseException, and your except block will not be used. The exception bubbles up.

driver.quit()
driver.close()

There is no need for the second line given that at this point Firefox has already been exited.

Also the attached log is NOT a trace level log. Please make sure that you take it from geckodriver.log.

[Thenakedgun]

@whimboo

There is no need for the second line given that at this point Firefox has already been exited.

Sorry if the order wouldn't make sense under normal circumstances. In my case, it doesn't matter whether driver.close() or driver.quit() is called, either together or independently, or what order I attempt them in, Firefox will not close from either statement. I'll reverse the order of these statements in future however.

Additionally, I revised the catch statement slightly, this should now properly catch every exception, preventing it from bubbling up, correct? After these changes, the same behaviour from the bug report is still exhibited:

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

try:
	opts = Options()
	opts.log.level = "trace"
	driver = webdriver.Firefox(firefox_options=opts, executable_path='./bin/geckodriver.exe')
	variable = input("Enter something or Ctrl+C: ")
	driver.get(variable)
except:
	pass
finally:
	driver.close()
	driver.quit()	

Also the attached log is NOT a trace level log. Please make sure that you take it from geckodriver.log

The file I attached previously was the geckodriver.log file that was generated upon running the above script & pressing Ctrl+C at the input statement. I followed the link that @AutomatedTester provided, and included the code specified in the Python section of the Enabling Trace Logs post. If the resulting logfile is insufficient, could you please advise as to how I should modify the above code to provide the results you require? Incidentally, the provided resource may be slightly outdated, as I receive a deprecation warning when running it: "DeprecationWarning: use options instead of firefox_options"

Is no one able to actually reproduce this error, given the provided code?

[whimboo]

The problem actually is that you are using firefox_options as kwarg name when instantiating the Firefox driver. Instead it has to be just options.

[barancev]

@whimboo That's exactly what @Thenakedgun said, the document
https://firefox-source-docs.mozilla.org/testing/geckodriver/geckodriver/TraceLogs.html
recommends to use firefox_options argument that is deprecated, please update the instruction.

[Thenakedgun]

@whimboo K. replaced the options line with:

	opts = Options()
	opts.log.level = "trace"
	driver = webdriver.Firefox(options=opts, executable_path='./bin/geckodriver.exe')

And this was the generated geckodriver.log: geckodriver.log

[whimboo]

@whimboo That's exactly what @Thenakedgun said, the document
https://firefox-source-docs.mozilla.org/testing/geckodriver/geckodriver/TraceLogs.html
recommends to use firefox_options argument that is deprecated, please update the instruction.

Now I see. I filed https://bugzilla.mozilla.org/show_bug.cgi?id=1519925 to get this fixed. Thanks.

[whimboo]

@Thenakedgun thank you for the log. Now it is really a trace log. So I see that Marionette closes its connection about 5s after the new session has been created. I assume that is the time it has been taken for you to enter something for the input() command? It would be good if you could add some print lines to the Python code, so it will be visible when exactly which steps are getting executed.

Note that I cannot replicate the issue on MacOS. And I don't have a chance to test under Windows right now. Maybe some more logging on the Selenium side might help.