-
-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
os.fwalk() unhandled exception when error occurs accessing symbolic link target #72539
Comments
The bug is os.fwalk() crashes with unhandled exception when there is an error To reproduce the bug, create a symbolic link that targets a file that you do not $ touch handsoff
$ sudo chown root:root handsoff
$ sudo chmod 700 handsoff
$ ln -s handsoff blah Now, os.fwalk() fails: >>> for root, dirs, files, fd in os.fwalk('.'):
... print(root, dirs, files)
...
Traceback (most recent call last):
File "test_fwalk_permission_error.py", line 3, in <module>
for root, dirs, files, fd in os.fwalk('.'):
File "/usr/lib64/python3.5/os.py", line 520, in fwalk
yield from _fwalk(topfd, top, topdown, onerror, follow_symlinks)
File "/usr/lib64/python3.5/os.py", line 537, in _fwalk
if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
PermissionError: [Errno 13] Permission denied: 'blah' The cause of the problem is in this part of os.py: for name in names:
try:
# Here, we don't use AT_SYMLINK_NOFOLLOW to be consistent with
# walk() which reports symlinks to directories as directories.
# We do however check for symlinks before recursing into
# a subdirectory.
if st.S_ISDIR(stat(name, dir_fd=topfd).st_mode):
dirs.append(name)
else:
nondirs.append(name)
except FileNotFoundError:
try:
# Add dangling symlinks, ignore disappeared files
if st.S_ISLNK(stat(name, dir_fd=topfd, follow_symlinks=False)
.st_mode):
nondirs.append(name)
except FileNotFoundError:
continue To fix it, simply replace FileNotFoundError with more general OSError. Cheers |
Similar to http://bugs.python.org/issue25860 |
I can't reproduce the issue. $ mkdir testdir
$ touch testdir/handsoff
$ sudo chown root:root testdir/handsoff
$ sudo chmod 700 testdir/handsoff
$ ln -s handsoff testdir/blah
$ python3.5
>>> import os
>>> list(os.walk('testdir'))
[('testdir', [], ['handsoff', 'blah'])]
>>> list(os.fwalk('testdir'))
[('testdir', [], ['handsoff', 'blah'], 3)] Ubuntu 16.04, Linux 4.4, tested on ext4 and tmpfs filesystems. |
Sorry, the target file needs to be in a directory you don't have permission to access: $ mkdir handsoff
$ sudo chown root:root handsoff
$ sudo chmod 700 handsoff
$ ln -s handsoff/anything blah |
This is definitely a bug because there is no way an unhandled exception like this is acceptable behaviour. The behaviour is that the fwalk iteration will stop and there is no way to recover / continue. Also fwalk takes an onerror callback function that won't be called with this error. |
Here is a patch with tests. Needed to test it on Windows. |
New changeset fe1fea4ded04 by Serhiy Storchaka in branch '3.5': New changeset e99ec3c77a63 by Serhiy Storchaka in branch '3.6': New changeset 33a1a3dd0051 by Serhiy Storchaka in branch 'default': |
New changeset ec12e16ea6a1 by Serhiy Storchaka in branch '3.6': New changeset a0913dbadea6 by Serhiy Storchaka in branch 'default': |
New changeset 470224ec16b6 by Serhiy Storchaka in branch '3.5': |
Serhiy, after your commits, test_os requires root privileges or it'll fail. This is not the case before. [cpython]$ ./python -m test test_os
Run tests sequentially
0:00:00 [1/1] test_os
test test_os crashed -- Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 164, in runtest_inner
test_runner()
File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 163, in test_runner
support.run_unittest(tests)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1849, in run_unittest
_run_suite(suite)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 1824, in _run_suite
raise TestFailed(err)
test.support.TestFailed: multiple errors occurred; run in verbose mode for details
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/libregrtest/runtest.py", line 167, in runtest_inner
test_time = time.time() - start_time
File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 278, in __exit__
restore(original)
File "/home/angwer/cpython/Lib/test/libregrtest/save_env.py", line 235, in restore_files
support.rmtree(fn)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
_rmtree(path)
File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21' 'test_os' left behind directory '@test_23400_tmp' and it couldn't be removed: [Errno 13] Permission denied: 'SUB21' 1 test failed: Total duration: 292 ms
Tests result: FAILURE
Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 914, in temp_dir
yield path
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd
yield cwd_dir
File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
self._main(tests, kwargs)
File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 497, in _main
sys.exit(len(self.bad) > 0 or self.interrupted)
SystemExit: True
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/angwer/cpython/Lib/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/home/angwer/cpython/Lib/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/angwer/cpython/Lib/test/__main__.py", line 2, in <module>
main()
File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 532, in main
Regrtest().main(tests=tests, **kwargs)
File "/home/angwer/cpython/Lib/test/libregrtest/main.py", line 468, in main
self._main(tests, kwargs)
File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 963, in temp_cwd
yield cwd_dir
File "/home/angwer/cpython/Lib/contextlib.py", line 100, in __exit__
self.gen.throw(type, value, traceback)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 917, in temp_dir
rmtree(path)
File "/home/angwer/cpython/Lib/test/support/__init__.py", line 377, in rmtree
_rmtree(path)
File "/home/angwer/cpython/Lib/shutil.py", line 474, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 412, in _rmtree_safe_fd
_rmtree_safe_fd(dirfd, fullname, onerror)
File "/home/angwer/cpython/Lib/shutil.py", line 408, in _rmtree_safe_fd
onerror(os.open, fullname, sys.exc_info())
File "/home/angwer/cpython/Lib/shutil.py", line 406, in _rmtree_safe_fd
dirfd = os.open(name, os.O_RDONLY, dir_fd=topfd)
PermissionError: [Errno 13] Permission denied: 'SUB21' |
Hmm, tests were passed when I ran them before committing. Seems this is heisenbug. |
Seems like the dir you add breaks other cases. I change addCleanUp to os.chmod and get: ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
self.assertEqual(len(all), 4)
AssertionError: 5 != 4 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
self.assertEqual(len(all), 4)
AssertionError: 5 != 4 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 950, in test_file_like_path
self.test_walk_prune(_PathLike(self.walk_path))
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 956, in test_walk_bottom_up
self.assertEqual(len(all), 4, all)
AssertionError: 5 != 4 : [('@test_24702_tmp/TEST1/SUB2/SUB21', [], ['tmp3']), ('@test_24702_tmp/TEST1/SUB2', ['link', 'SUB21'], ['tmp3', 'broken_link3', 'broken_link', 'broken_link2']), ('@test_24702_tmp/TEST1/SUB1/SUB11', [], []), ('@test_24702_tmp/TEST1/SUB1', ['SUB11'], ['tmp2']), ('@test_24702_tmp/TEST1', ['SUB2', 'SUB1'], ['tmp1'])] ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 941, in test_walk_prune
self.assertEqual(len(all), 2)
AssertionError: 3 != 2 ====================================================================== Traceback (most recent call last):
File "/home/angwer/cpython/Lib/test/test_os.py", line 916, in test_walk_topdown
self.assertEqual(len(all), 4)
AssertionError: 5 != 4 |
New changeset 655510dd46fd by Serhiy Storchaka in branch '3.5': New changeset df28e536f19d by Serhiy Storchaka in branch '3.6': New changeset aa891d13e1cf by Serhiy Storchaka in branch 'default': |
test_walk_bad_dir renamed one of subdirs and tearDown failed to clean up changed tree. An error happened depending on the order of listed subdirectories: ['SUB1', 'SUB2'] -- passed, ['SUB2', 'SUB1'] -- failed. |
It works. Close. :-) |
Misc/NEWS
so that it is managed by towncrier #552Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: