[Windows] OSError when testing whether pathlib.Path('*') exists

[jimbo1qaz]

BPO	35306
Nosy	@pfmoore, @pitrou, @tjguk, @zware, @serhiy-storchaka, @eryksun, @zooba, @miss-islington
PRs	bpo-35306: Handle '*' in pathlib.Path functions on Windows #11133 bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename #25529 [3.9] bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename (GH-25529) #25534 [3.8] bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename (GH-25529) #25535

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zooba'
closed_at = <Date 2021-04-22.20:12:38.885>
created_at = <Date 2018-11-24.05:55:59.185>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'OS-windows']
title = "[Windows] OSError when testing whether pathlib.Path('*') exists"
updated_at = <Date 2021-04-22.20:53:11.398>
user = 'https://bugs.python.org/jimbo1qaz'

bugs.python.org fields:

activity = <Date 2021-04-22.20:53:11.398>
actor = 'steve.dower'
assignee = 'steve.dower'
closed = True
closed_date = <Date 2021-04-22.20:12:38.885>
closer = 'steve.dower'
components = ['Library (Lib)', 'Windows']
creation = <Date 2018-11-24.05:55:59.185>
creator = 'jimbo1qaz_'
dependencies = []
files = []
hgrepos = []
issue_num = 35306
keywords = ['patch']
message_count = 8.0
messages = ['330371', '330394', '333092', '333728', '333730', '391627', '391630', '391631']
nosy_count = 9.0
nosy_names = ['paul.moore', 'pitrou', 'tim.golden', 'zach.ware', 'serhiy.storchaka', 'eryksun', 'steve.dower', 'miss-islington', 'jimbo1qaz_']
pr_nums = ['11133', '25529', '25534', '25535']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35306'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[jimbo1qaz]

I'm writing a program taking paths from user input through CLI.

path is a pathlib.Path().

Since Windows doesn't expand asterisks, I check if the path doesn't exist. If so I expand using Path().glob(path).

Unfortunately on Windows, if path (type: Path) contains asterisks, checking path.exists() or path.is_dir() raises WinError 123.

Python 3.7.0 (default, Jun 28 2018, 08:04:48) [MSC v.1912 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from pathlib import Path
>>> Path('*').exists()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\jimbo1qaz\Miniconda3\envs\python37\lib\pathlib.py", line 1318, in exists
    self.stat()
  File "C:\Users\jimbo1qaz\Miniconda3\envs\python37\lib\pathlib.py", line 1140, in stat
    return self._accessor.stat(self)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '*'
>>> Path('*').is_dir()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\jimbo1qaz\Miniconda3\envs\python37\lib\pathlib.py", line 1330, in is_dir
    return S_ISDIR(self.stat().st_mode)
  File "C:\Users\jimbo1qaz\Miniconda3\envs\python37\lib\pathlib.py", line 1140, in stat
    return self._accessor.stat(self)
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '*'

I also reproduced on Miniconda 3.6.6, 3.7.0, and official Python 3.7.1.

According to https://bugs.python.org/issue29827 , os.path.exists() (not Path.exists() ) returns False on any OSError.

-----------------

On Linux, checking paths with null bytes (a less common occurrence) raises a different error:

>>> import pathlib
>>> pathlib.Path("\x00").exists()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/pathlib.py", line 1336, in exists
    self.stat()
  File "/usr/lib/python3.6/pathlib.py", line 1158, in stat
    return self._accessor.stat(self)
  File "/usr/lib/python3.6/pathlib.py", line 387, in wrapped
    return strfunc(str(pathobj), *args)
ValueError: embedded null byte

[eryksun]

Path.exists should ignore all OSError exceptions, as os.path.exists does. Barring that, I suppose for Windows we could add EINVAL to IGNORED_ERROS.

The problem with ValueError was already addressed in bpo-33721.

[jimbo1qaz]

Should Path.resolve() also avoid raising OSError?

Path('*').resolve()

Traceback (most recent call last):
...truncated
  File "<ipython-input-5-4fa2fec5c8b3>", line 1, in <module>
    Path('*').resolve()
  File "C:\Users\jimbo1qaz\AppData\Local\Programs\Python\Python37\lib\pathlib.py", line 1134, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Users\jimbo1qaz\AppData\Local\Programs\Python\Python37\lib\pathlib.py", line 192, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 123] The filename, directory name, or volume label syntax is incorrect: '*'

os.path.realpath('"*')
Out[8]: 'C:\\Users\\jimbo1qaz\\Dropbox\\encrypted\\code\\corrscope\\"*'
os.path.abspath('*"')
Out[13]: 'C:\\Users\\jimbo1qaz\\Dropbox\\encrypted\\code\\corrscope\\*"'

(sidenote: what os.path operation does Path.resolve() match? Path('nonexistent').resolve() returns a relative path on Python 3.7.1, whereas Path().resolve() returns an absolute path.)

[zooba]

Pathlib doesn't necessarily directly follow os on its error handling - adding Antoine for comment.

Passing strict=False to resolve() should be able to handle an invalid name like that. If not, I propose that we change it so that it does.

[pitrou]

I'm fine with swallowing the error in both exists() and resolve(). We should be careful not to swallow errors too broadly, though. The code paths should be audited to check that EINVAL can't mean something else.

[zooba]

New changeset 4696f12 by Steve Dower in branch 'master':
bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename (GH-25529)
4696f12

[miss-islington]

New changeset 1575ea0 by Miss Islington (bot) in branch '3.8':
bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename (GH-25529)
1575ea0

[miss-islington]

New changeset ab5d78c by Miss Islington (bot) in branch '3.9':
bpo-35306: Avoid raising OSError from pathlib.Path.exists when passed an invalid filename (GH-25529)
ab5d78c