pathlib match parameters error

[hadim]

Consider this code:

import pathlib
import shutil


data_dir = pathlib.Path("/tmp")
archive_path = data_dir / "hello"
shutil.rmtree(archive_path)

PyRight raises this error:

No overloads for 'shutil.rmtree(archive_path)' match parameters
  Argument types: (Path)

Originally posted at microsoft/pyright#397

[JelleZijlstra]

Actually, I think pyright is wrong here. The overloads allow os.PathLike, and a Path is an instance of os.PathLike.

[hadim]

Ping @erictraut

[erictraut]

The overload allows an _AnyPath, which is constrained to str or os.PathLike[str]. TypeVar constraints must match exactly. Derived classes are not allowed. At least that's how I interpret PEP 484, which says:

type constraints cause the inferred type to be exactly one of the constraint types, while an upper bound just requires that the actual type is a subtype of the boundary type.

[srittau]

Closing this per discussion in microsoft/pyright#397.

[JelleZijlstra]

Looking at the rmtree stubs again, I do think there is a small issue. The Python 3 stub is currently like this:

    @overload
    def rmtree(path: bytes, ignore_errors: bool = ...,
               onerror: Optional[Callable[[Any, str, Any], Any]] = ...) -> None: ...
    @overload
    def rmtree(path: _AnyPath, ignore_errors: bool = ...,
               onerror: Optional[Callable[[Any, _AnyPath, Any], Any]] = ...) -> None: ...

_AnyPath is a typevar over str and PathLike[str].

But looking at the implementation, the onerror callback can be invoked either on the path directly passed to rmtree, or on a directory entry inside the path, as produced by scandir, which are str or bytes depending on what we gave scandir. Bytes paths are decoded only if _use_fd_functions is True (which I think basically means modern Linux). In any case, the argument to onerror can be a str even if a PathLike was passed to the function. Therefore, we can't really give the second argument to onerror a more precise type than Any. This also makes the use of the TypeVar unnecessary. I'll submit a PR to make this change.

[AnyOldName3]

I'm unsure what this discussion concluded. Do the maintainers of this project believe mypy outputting

error: Argument 1 to "move" has incompatible type "Path"; expected "str"

is a bug in the standard library stubs, or that shutil is genuinely not intended to be used with pathlib directly?

I see an rstrip call in _basename, so it might well not be intended to work with anything except str, but that would surprise me given that PEP 519 was supposed to make the standard library work with pathlib without jumping through hoops.

[hauntsaninja]

Most of the discussion here and in microsoft/pyright#397 centred around pyright's handling of constrained type vars. Additionally, there was a small improvement suggested and made to rmtree in #3536, so going ahead and closing this issue.

Your issue with shutil stubs is unrelated, so generally speaking would best be served by opening a new issue. However, since we're already here, my quick reply is I've encountered cases where shutil does not work with pathlib, i.e. there's a bug in CPython, e.g. #4041 (comment) . I'd need to look further to know what's happening with move.

[hauntsaninja]

Oh actually, see #3559 and https://bugs.python.org/issue32689