Skip to content
Permalink
Browse files

bpo-37834: Normalise handling of reparse points on Windows (GH-15231)

bpo-37834: Normalise handling of reparse points on Windows
* ntpath.realpath() and nt.stat() will traverse all supported reparse points (previously was mixed)
* nt.lstat() will let the OS traverse reparse points that are not name surrogates (previously would not traverse any reparse point)
* nt.[l]stat() will only set S_IFLNK for symlinks (previous behaviour)
* nt.readlink() will read destinations for symlinks and junction points only

bpo-1311: os.path.exists('nul') now returns True on Windows
* nt.stat('nul').st_mode is now S_IFCHR (previously was an error)
  • Loading branch information
zooba committed Aug 21, 2019
1 parent bcc446f commit df2d4a6f3d5da2839c4fc11d31511c8e028daf2c
@@ -1858,6 +1858,12 @@ features:
.. versionchanged:: 3.6
Accepts a :term:`path-like object` for *src* and *dst*.

.. versionchanged:: 3.8
On Windows, now opens reparse points that represent another path
(name surrogates), including symbolic links and directory junctions.
Other kinds of reparse points are resolved by the operating system as
for :func:`~os.stat`.


.. function:: mkdir(path, mode=0o777, *, dir_fd=None)

@@ -2039,6 +2045,10 @@ features:
This function can also support :ref:`paths relative to directory descriptors
<dir_fd>`.

When trying to resolve a path that may contain links, use
:func:`~os.path.realpath` to properly handle recursion and platform
differences.

.. availability:: Unix, Windows.

.. versionchanged:: 3.2
@@ -2053,6 +2063,11 @@ features:
.. versionchanged:: 3.8
Accepts a :term:`path-like object` and a bytes object on Windows.

.. versionchanged:: 3.8
Added support for directory junctions, and changed to return the
substitution path (which typically includes ``\\?\`` prefix) rather
than the optional "print name" field that was previously returned.

.. function:: remove(path, *, dir_fd=None)

Remove (delete) the file *path*. If *path* is a directory, an
@@ -2366,7 +2381,8 @@ features:

On Unix, this method always requires a system call. On Windows, it
only requires a system call if *follow_symlinks* is ``True`` and the
entry is a symbolic link.
entry is a reparse point (for example, a symbolic link or directory
junction).

On Windows, the ``st_ino``, ``st_dev`` and ``st_nlink`` attributes of the
:class:`stat_result` are always set to zero. Call :func:`os.stat` to
@@ -2403,6 +2419,17 @@ features:
This function can support :ref:`specifying a file descriptor <path_fd>` and
:ref:`not following symlinks <follow_symlinks>`.

On Windows, passing ``follow_symlinks=False`` will disable following all
name-surrogate reparse points, which includes symlinks and directory
junctions. Other types of reparse points that do not resemble links or that
the operating system is unable to follow will be opened directly. When
following a chain of multiple links, this may result in the original link
being returned instead of the non-link that prevented full traversal. To
obtain stat results for the final path in this case, use the
:func:`os.path.realpath` function to resolve the path name as far as
possible and call :func:`lstat` on the result. This does not apply to
dangling symlinks or junction points, which will raise the usual exceptions.

.. index:: module: stat

Example::
@@ -2427,6 +2454,14 @@ features:
.. versionchanged:: 3.6
Accepts a :term:`path-like object`.

.. versionchanged:: 3.8
On Windows, all reparse points that can be resolved by the operating
system are now followed, and passing ``follow_symlinks=False``
disables following all name surrogate reparse points. If the operating
system reaches a reparse point that it is not able to follow, *stat* now
returns the information for the original path as if
``follow_symlinks=False`` had been specified instead of raising an error.


.. class:: stat_result

@@ -2578,7 +2613,7 @@ features:

File type.

On Windows systems, the following attribute is also available:
On Windows systems, the following attributes are also available:

.. attribute:: st_file_attributes

@@ -2587,6 +2622,12 @@ features:
:c:func:`GetFileInformationByHandle`. See the ``FILE_ATTRIBUTE_*``
constants in the :mod:`stat` module.

.. attribute:: st_reparse_tag

When :attr:`st_file_attributes` has the ``FILE_ATTRIBUTE_REPARSE_POINT``
set, this field contains the tag identifying the type of reparse point.
See the ``IO_REPARSE_TAG_*`` constants in the :mod:`stat` module.

The standard module :mod:`stat` defines functions and constants that are
useful for extracting information from a :c:type:`stat` structure. (On
Windows, some items are filled with dummy values.)
@@ -2614,6 +2655,14 @@ features:
.. versionadded:: 3.7
Added the :attr:`st_fstype` member to Solaris/derivatives.

.. versionadded:: 3.8
Added the :attr:`st_reparse_tag` member on Windows.

.. versionchanged:: 3.8
On Windows, the :attr:`st_mode` member now identifies special
files as :const:`S_IFCHR`, :const:`S_IFIFO` or :const:`S_IFBLK`
as appropriate.

.. function:: statvfs(path)

Perform a :c:func:`statvfs` system call on the given path. The return value is
@@ -304,6 +304,10 @@ Directory and files operations
Added a symlink attack resistant version that is used automatically
if platform supports fd-based functions.

.. versionchanged:: 3.8
On Windows, will no longer delete the contents of a directory junction
before removing the junction.

.. attribute:: rmtree.avoids_symlink_attacks

Indicates whether the current platform and implementation provides a
@@ -425,3 +425,13 @@ for more detail on the meaning of these constants.
FILE_ATTRIBUTE_VIRTUAL

.. versionadded:: 3.5

On Windows, the following constants are available for comparing against the
``st_reparse_tag`` member returned by :func:`os.lstat`. These are well-known
constants, but are not an exhaustive list.

.. data:: IO_REPARSE_TAG_SYMLINK
IO_REPARSE_TAG_MOUNT_POINT
IO_REPARSE_TAG_APPEXECLINK

.. versionadded:: 3.8
@@ -808,6 +808,21 @@ A new :func:`os.memfd_create` function was added to wrap the
``memfd_create()`` syscall.
(Contributed by Zackery Spytz and Christian Heimes in :issue:`26836`.)

On Windows, much of the manual logic for handling reparse points (including
symlinks and directory junctions) has been delegated to the operating system.
Specifically, :func:`os.stat` will now traverse anything supported by the
operating system, while :func:`os.lstat` will only open reparse points that
identify as "name surrogates" while others are opened as for :func:`os.stat`.
In all cases, :attr:`stat_result.st_mode` will only have ``S_IFLNK`` set for
symbolic links and not other kinds of reparse points. To identify other kinds
of reparse point, check the new :attr:`stat_result.st_reparse_tag` attribute.

On Windows, :func:`os.readlink` is now able to read directory junctions. Note
that :func:`~os.path.islink` will return ``False`` for directory junctions,
and so code that checks ``islink`` first will continue to treat junctions as
directories, while code that handles errors from :func:`os.readlink` may now
treat junctions as links.


os.path
-------
@@ -824,6 +839,9 @@ characters or bytes unrepresentable at the OS level.
environment variable and does not use :envvar:`HOME`, which is not normally set
for regular user accounts.

:func:`~os.path.isdir` on Windows no longer returns true for a link to a
non-existent directory.

:func:`~os.path.realpath` on Windows now resolves reparse points, including
symlinks and directory junctions.

@@ -912,6 +930,9 @@ format for new archives to improve portability and standards conformance,
inherited from the corresponding change to the :mod:`tarfile` module.
(Contributed by C.A.M. Gerlach in :issue:`30661`.)

:func:`shutil.rmtree` on Windows now removes directory junctions without
recursively removing their contents first.


ssl
---
@@ -84,6 +84,7 @@ struct _Py_stat_struct {
time_t st_ctime;
int st_ctime_nsec;
unsigned long st_file_attributes;
unsigned long st_reparse_tag;
};
#else
# define _Py_stat_struct stat
@@ -452,7 +452,14 @@ def _copytree(entries, src, dst, symlinks, ignore, copy_function,
dstname = os.path.join(dst, srcentry.name)
srcobj = srcentry if use_srcentry else srcname
try:
if srcentry.is_symlink():
is_symlink = srcentry.is_symlink()
if is_symlink and os.name == 'nt':
# Special check for directory junctions, which appear as
# symlinks but we want to recurse.
lstat = srcentry.stat(follow_symlinks=False)
if lstat.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT:
is_symlink = False
if is_symlink:
linkto = os.readlink(srcname)
if symlinks:
# We can't just leave it to `copy_function` because legacy
@@ -537,6 +544,37 @@ def copytree(src, dst, symlinks=False, ignore=None, copy_function=copy2,
ignore_dangling_symlinks=ignore_dangling_symlinks,
dirs_exist_ok=dirs_exist_ok)

if hasattr(stat, 'FILE_ATTRIBUTE_REPARSE_POINT'):
# Special handling for directory junctions to make them behave like
# symlinks for shutil.rmtree, since in general they do not appear as
# regular links.
def _rmtree_isdir(entry):
try:
st = entry.stat(follow_symlinks=False)
return (stat.S_ISDIR(st.st_mode) and not
(st.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT
and st.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT))
except OSError:
return False

def _rmtree_islink(path):
try:
st = os.lstat(path)
return (stat.S_ISLNK(st.st_mode) or
(st.st_file_attributes & stat.FILE_ATTRIBUTE_REPARSE_POINT
and st.st_reparse_tag == stat.IO_REPARSE_TAG_MOUNT_POINT))
except OSError:
return False
else:
def _rmtree_isdir(entry):
try:
return entry.is_dir(follow_symlinks=False)
except OSError:
return False

def _rmtree_islink(path):
return os.path.islink(path)

# version vulnerable to race conditions
def _rmtree_unsafe(path, onerror):
try:
@@ -547,11 +585,7 @@ def _rmtree_unsafe(path, onerror):
entries = []
for entry in entries:
fullname = entry.path
try:
is_dir = entry.is_dir(follow_symlinks=False)
except OSError:
is_dir = False
if is_dir:
if _rmtree_isdir(entry):
try:
if entry.is_symlink():
# This can only happen if someone replaces
@@ -681,7 +715,7 @@ def onerror(*args):
os.close(fd)
else:
try:
if os.path.islink(path):
if _rmtree_islink(path):
# symlinks to directories are forbidden, see bug #1669
raise OSError("Cannot call rmtree on a symbolic link")
except OSError:

0 comments on commit df2d4a6

Please sign in to comment.