Unexpected FileNotFoundError when current directory is removed

[vadmium]

BPO	22834
Nosy	@brettcannon, @terryjreedy, @bitdancer, @ericsnowcurrently, @vadmium, @serhiy-storchaka

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/brettcannon'
closed_at = <Date 2015-02-27.17:14:11.000>
created_at = <Date 2014-11-10.03:56:59.304>
labels = ['interpreter-core', 'type-bug']
title = 'Unexpected FileNotFoundError when current directory is removed'
updated_at = <Date 2015-09-23.04:42:08.243>
user = 'https://github.com/vadmium'

bugs.python.org fields:

activity = <Date 2015-09-23.04:42:08.243>
actor = 'Christopher Meng'
assignee = 'brett.cannon'
closed = True
closed_date = <Date 2015-02-27.17:14:11.000>
closer = 'brett.cannon'
components = ['Interpreter Core']
creation = <Date 2014-11-10.03:56:59.304>
creator = 'martin.panter'
dependencies = []
files = []
hgrepos = []
issue_num = 22834
keywords = []
message_count = 18.0
messages = ['230933', '230963', '231161', '231170', '231173', '231192', '231477', '231479', '231480', '231521', '236092', '236122', '236289', '236292', '236294', '236336', '236795', '236796']
nosy_count = 9.0
nosy_names = ['brett.cannon', 'terry.reedy', 'Arfrever', 'r.david.murray', 'python-dev', 'eric.snow', 'martin.panter', 'serhiy.storchaka', 'Christopher Meng']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22834'
versions = ['Python 3.5']

[vadmium]

I encountered this when I added a unit test case that invoked os.chdir() with a temporary directory on Linux. After the directory was removed, some of the subsequent test cases failed, although I don’t think they should depend on a particular CWD.

I suspect the main problem might be code that is trying to dynamically import modules, and the interpreter is trying to search for modules in the current directory. I would expect it to happily go on to the other standard module directories or raise ImportError, just like if the current directory is valid but empty, or an nonexistent directory is in the module search path list.

Code to set up missing CWD:

import os
from tempfile import TemporaryDirectory
with TemporaryDirectory() as dir:
    os.chdir(dir)

Quick recovery:

os.chdir("/")

Examples of failures:

>>> "\N{COPYRIGHT SIGN}"
  File "<stdin>", line 1
SyntaxError: (unicode error) \N escapes not supported (can't load unicodedata module)
>>> datetime.strptime("", "")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2222, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 2164, in _find_spec
  File "<frozen importlib._bootstrap>", line 1940, in find_spec
  File "<frozen importlib._bootstrap>", line 1911, in _get_spec
  File "<frozen importlib._bootstrap>", line 1879, in _path_importer_cache
FileNotFoundError: [Errno 2] No such file or directory
>>> HTTPConnection("localhost").request("GET", "/")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.4/http/client.py", line 1090, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.4/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.4/http/client.py", line 1086, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.4/http/client.py", line 924, in _send_output
    self.send(msg)
  File "/usr/lib/python3.4/http/client.py", line 859, in send
    self.connect()
  File "/usr/lib/python3.4/http/client.py", line 836, in connect
    self.timeout, self.source_address)
  File "/usr/lib/python3.4/socket.py", line 491, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.4/socket.py", line 530, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
  File "/usr/lib/python3.4/encodings/__init__.py", line 98, in search_function
    level=0)
  File "/usr/lib/python3.4/encodings/idna.py", line 3, in <module>
    import stringprep, re, codecs
  File "<frozen importlib._bootstrap>", line 2237, in _find_and_load
  File "<frozen importlib._bootstrap>", line 2222, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 2164, in _find_spec
  File "<frozen importlib._bootstrap>", line 1940, in find_spec
  File "<frozen importlib._bootstrap>", line 1911, in _get_spec
  File "<frozen importlib._bootstrap>", line 1879, in _path_importer_cache
FileNotFoundError: [Errno 2] No such file or directory

>>> from datetime import datetime
>>> from http.client import HTTPConnection
These two also generate the FileNotFoundError

My workaround is to add this to my test case:
self.addCleanup(os.chdir, os.getcwd())

[bitdancer]

Looks like importlib doesn't handle the case of a directory on the path being deleted? If so, I'm surprised this hasn't been reported before.

[brettcannon]

So it isn't about importlib not handling missing directories on sys.path directly, it has to do with the fact that os.getcwd() raises FileNotFoundError when CWD is no longer valid and that is in a fundamental part of importlib that isn't worrying about non-existent directories.

We could be robust and simply have instances of os.getcwd() failing just move on, e.g. when '' is hit in sys.path and os.getcwd() raises an exception just give up. The other option is leaving this to raise FileNotFoundError but throwing a new copy with a better error message mentioning this is because the current working directory no longer exists. That would lead to easier debugging since paths on sys.path are typically obvious -- since site.py makes them absolute -- but I'm willing to bet people don't bother checking CWD is still good. So it's robustness vs. debugging when you make this mistake.

Any opinions on the matter?

[bitdancer]

Well, once I've launched a program, regardless of whether or not I expected some stuff to come from the CWD, I generally don't think about whether or not the CWD might go away, and in a complex setup it would most likely be getting deleted by some other process. So I think I would prefer an import that is of something that wasn't ever in the CWD to succeed if the CWD goes away. I'm not even sure I'd want a warning.

What happens if a directory on the path disappears? If that is treated as a non-error, then I think a missing CWD on the path should also be treated as a non-error.

[terryjreedy]

How about issue a verbose warning for possible debugging and continue for robustness?

[vadmium]

The only time I see a warning would be useful is if you intended to override a standard module with a module of the same name in the current directory. In all other cases I think it would be better to either generate an ImportError if the module is not found, or import it from wherever it is found. So I think a warning would not be useful in most cases.

Having any other non-existant directory in the search path is not an error and there is no warning either:
$ python3 -btWall
Python 3.4.2 (default, Oct  8 2014, 14:33:30) 
[GCC 4.9.1 20140903 (prerelease)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path.insert(0, "/blaua")
>>> import sadface
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named 'sadface'
>>> import urllib
>>> # Interpreter = happy
...

[brettcannon]

I have a patch to silence the exception and I'm running the test suite now.

I'm planning to keep this to a 3.5 fix and not changing the semantics in Python 3.4 as the fix is a little different from the standard "directory in sys.path is invalid" since '' is dynamic and I can see someone relying on the current exception bubbling out somehow and not caching the dead directory in sys.path_importer_cache.

[python-dev]

New changeset 8558fff73032 by Brett Cannon in branch 'default':
Issue bpo-22834: Have import suppress FileNotFoundError when the current
https://hg.python.org/cpython/rev/8558fff73032

[brettcannon]

Along with fixing this I also updated the import reference to mention how the current working directory is handled.

Thanks to Martin for the report and everyone for their input!

[python-dev]

New changeset d065e6474b67 by Zachary Ware in branch 'default':
Issue bpo-22834: cwd can't not exist on Windows, skip the test
https://hg.python.org/cpython/rev/d065e6474b67

[serhiy-storchaka]

Tests failed on Solaris:
http://buildbot.python.org/all/builders/AMD64%20Solaris%2011%20%5BSB%5D%203.x/builds/3895/steps/test/logs/stdio

======================================================================
ERROR: test_deleted_cwd (test.test_importlib.import_.test_path.Source_FinderTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/test/test_importlib/import_/test_path.py", line 167, in test_deleted_cwd
    os.chdir(path)
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/tempfile.py", line 711, in __exit__
    self.cleanup()
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/tempfile.py", line 715, in cleanup
    _shutil.rmtree(self.name)
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/shutil.py", line 474, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/cpython/buildslave/cc-32/3.x.snakebite-solaris11-amd64/build/Lib/shutil.py", line 472, in rmtree
    os.rmdir(path)
OSError: [Errno 22] Invalid argument: '/tmp/tmpsx3fm0t4'

---