OSError when using pathlib.Path.rglob() to list device files

[VictorDomingos]

BPO	33635
Nosy	@ronaldoussoren, @ned-deily, @zooba, @miss-islington, @Windsooon, @tirkarthi
PRs	bpo-33635: Handling Bad file descriptor in Path.is_file and related. #8542 [3.7] bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542) #8960

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-03-20.00:52:19.850>
created_at = <Date 2018-05-24.12:15:27.517>
labels = ['OS-mac', 'type-bug', 'library', 'expert-IO', '3.7']
title = 'OSError when using pathlib.Path.rglob() to list device files'
updated_at = <Date 2019-03-20.00:52:19.849>
user = 'https://bugs.python.org/VictorDomingos'

bugs.python.org fields:

activity = <Date 2019-03-20.00:52:19.849>
actor = 'cheryl.sabella'
assignee = 'none'
closed = True
closed_date = <Date 2019-03-20.00:52:19.850>
closer = 'cheryl.sabella'
components = ['Library (Lib)', 'macOS', 'IO']
creation = <Date 2018-05-24.12:15:27.517>
creator = 'Victor Domingos'
dependencies = []
files = []
hgrepos = []
issue_num = 33635
keywords = ['patch']
message_count = 6.0
messages = ['317566', '317791', '321993', '322308', '324199', '324203']
nosy_count = 7.0
nosy_names = ['ronaldoussoren', 'ned.deily', 'steve.dower', 'miss-islington', 'Victor Domingos', 'Windson Yang', 'xtreak']
pr_nums = ['8542', '8960']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue33635'
versions = ['Python 3.7']

[VictorDomingos]

This method fails with an error when it finds a character or block device (like those found in /dev on macOS). However, in the same machine, with the same Python version, the alternative os.walk() performs basically the same job with no errors. I am not sure if that error is the expected behaviour, but I wasn't able to find a clear explanation in the docs. Anyway, it seems to me, as a user, that the os.walk() error-less behaviour is more desirable.

$ python3
Python 3.7.0b4 (v3.7.0b4:eb96c37699, May  2 2018, 04:13:13) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> from pathlib import Path
>>> [f for f in Path(os.path.expanduser('/dev')).rglob("*") if f.is_file()]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 1, in <listcomp>
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1344, in is_file
    return S_ISREG(self.stat().st_mode)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/pathlib.py", line 1140, in stat
    return self._accessor.stat(self)
OSError: [Errno 9] Bad file descriptor: '/dev/fd/3'
>>> 

[ronaldoussoren]

This is fairly odd behaviour of macOS, the same error can be seen from a bash session:

$ stat /dev/fd/3
stat: /dev/fd/3: stat: Bad file descriptor

The reason os.walk() works while the Path().is_file doesn't is that os.walk() explicitly guards against OSError exceptions raised by os.stat().

Looking at the documentation this could be seen as a bug in macOS, as the manual page for stat(2) doesn't mention EBADF as a valid error for this system call.

I'm not sure at this point if we should add a workaround for this. An actual patch would be easy enough, "just" add EBADF to the list of ignored errno values in the implementation of is_file (and related method) in pathlib.py

[Windsooon]

I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?

[ronaldoussoren]

Os.stat is defined in C code, in particular in Modules/posixmodule.c.

Op 20 jul. 2018 om 10:44 heeft Windson Yang <report@bugs.python.org> het volgende geschreven:

Windson Yang <wiwindson@gmail.com> added the comment:

I tried to fix this issue and I found this should be related to os.stat() in os.py. But I can't find the os.stat() API in the file, any ideas?

----------
nosy: +Windson Yang

---

Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue33635\>

---

[zooba]

New changeset 216b745 by Steve Dower (Przemysław Spodymek) in branch 'master':
bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
216b745

[miss-islington]

New changeset 2cf3316 by Miss Islington (bot) in branch '3.7':
bpo-33635: Handling Bad file descriptor in Path.is_file and related. (GH-8542)
2cf3316