nt._getfinalpathname on WindowsApps\**.exe raises WinError 1920 ERROR_CANT_ACCESS_FILE

[hhmmjjnn]

I just can't get tox to run in my Windows 10 development machine:

C:\myrepo>tox -ve py311-unit
py311-unit: find interpreter for spec PythonSpec(major=3, minor=11)
py311-unit: proposed PythonInfo(spec=CPython3.9.13.final.0-64, exe=C:\Users\myusername\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe, platform=win32, version='3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8)
py311-unit: proposed Pep514PythonInfo(spec=CPython3.11.3.final.0-64, exe=C:\Users\myusername\AppData\Local\Programs\Python\Python311\python.exe, platform=win32, version='3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8)
.pkg: find interpreter for spec PythonSpec(path=C:\Users\myusername\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe)
.pkg: proposed PythonInfo(spec=CPython3.9.13.final.0-64, exe=C:\Users\myusername\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe, platform=win32, version='3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)]', encoding_fs_io=utf-8-utf-8)
py311-unit: internal error
Traceback (most recent call last):
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\session\cmd\run\single.py", line 45, in _evaluate
    tox_env.setup()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\api.py", line 250, in setup
    self._setup_with_env()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\runner.py", line 143, in _setup_with_env
    self._setup_pkg()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\runner.py", line 166, in _setup_pkg
    self._packages = self._build_packages()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\runner.py", line 118, in _build_packages
    packages = package_env.perform_packaging(self.conf)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\virtual_env\package\pyproject.py", line 204, in perform_packaging
    self.setup()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\api.py", line 249, in setup
    self._setup_env()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\virtual_env\package\pyproject.py", line 154, in _setup_env
    super()._setup_env()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\package.py", line 57, in _setup_env
    super()._setup_env()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\api.py", line 198, in _setup_env
    self.ensure_python_env()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\api.py", line 202, in ensure_python_env
    conf = self.python_cache()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\virtual_env\api.py", line 77, in python_cache
    base = super().python_cache()
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\api.py", line 240, in python_cache
    "version_info": list(self.base_python.version_info),
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 64, in __getattribute__
    obj = object.__getattribute__(self, name)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\api.py", line 250, in base_python
    self._base_python = self._get_python(base_pythons)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\package.py", line 47, in _func
    return meth(*args, **kwargs)
  File "C:\Users\myusername\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\LocalCache\local-packages\Python39\site-packages\tox\tox_env\python\virtual_env\api.py", line 143, in _get_python
    extra={"executable": Path(interpreter.system_executable).resolve()},
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 1215, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 215, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1920] Não é possível o acesso ao arquivo pelo sistema: 'C:\\Users\\myusername\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe'
  py311-unit: FAIL code 2 (0.20 seconds)
  evaluation failed :( (0.70 seconds)

C:\myrepo>

I didn't even want to run python 3.9 (I asked for py311-unit) but tox insists in probing its path anyway... Shoot!! (ᗒᗣᗕ)

I narrowed it down to this snippet:

import nt
import os
import pathlib

p = (os.environ['LOCALAPPDATA'] + '\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe')

pathlib.WindowsPath(p).resolve() ## raises WinError 1920 ERROR_CANT_ACCESS_FILE
nt._getfinalpathname(p) ## raises WinError 1920 ERROR_CANT_ACCESS_FILE

MSDN documentation on GetFinalPathNameByHandleW doesn't seem to mention ERROR_CANT_ACCESS_FILE at all.

Full CMD output:

Microsoft Windows [versão 10.0.19044.2846]
(c) Microsoft Corporation. Todos os direitos reservados.

C:\Users\myusername>%LOCALAPPDATA%\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe
Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import nt
>>> import os
>>> import pathlib
>>> p = (os.environ['LOCALAPPDATA'] + '\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe')
>>> pathlib.WindowsPath(p).resolve()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 1215, in resolve
    s = self._flavour.resolve(self, strict=strict)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.9_3.9.3568.0_x64__qbz5n2kfra8p0\lib\pathlib.py", line 215, in resolve
    s = self._ext_to_normal(_getfinalpathname(s))
OSError: [WinError 1920] Não é possível o acesso ao arquivo pelo sistema: 'C:\\Users\\myusername\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe'
>>> nt._getfinalpathname(p)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OSError: [WinError 1920] Não é possível o acesso ao arquivo pelo sistema: 'C:\\Users\\myusername\\AppData\\Local\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe'
>>> ^Z


C:\Users\myusername>err 1920
# for decimal 1920 / hex 0x780
  ERROR_CANT_ACCESS_FILE                                         winerror.h
# The file cannot be accessed by the system.
# No results found for hex 0x1920 / decimal 6432
# as an HRESULT: Severity: SUCCESS (0), FACILITY_NULL (0x0), Code 0x780
# for decimal 1920 / hex 0x780
  ERROR_CANT_ACCESS_FILE                                         winerror.h
# The file cannot be accessed by the system.
# 2 matches found for "1920"

C:\Users\myusername>

[hhmmjjnn]

I worked around the issue by installing tox in a venv directory, then patching api.py:143 to remove the .resolve() call.

This is sub-optimal, so the issue stands.

[eryksun]

ERROR_CANT_ACCESS_FILE (1920) usually means that the I/O system failed to traverse a filesystem reparse point because neither the I/O manager nor any device driver supports it. A reparse point is a special file or directory type, such as a symlink, mount point (i.e. a filesystem junction), or a (partially) remote file (e.g. cloud storage such as OneDrive). The type of a reparse point is specified by a tag value in the header of its reparse data, such as IO_REPARSE_TAG_SYMLINK, IO_REPARSE_TAG_MOUNT_POINT, and IO_REPARSE_TAG_CLOUD.

The files under "%LocalAppData%\Microsoft\WindowsApps" are appexec links (i.e. IO_REPARSE_TAG_APPEXECLINK reparse points). WinAPI CreateProcessW() uses these 'links' indirectly in order to execute apps installed from the Microsoft Store. Unlike almost all other types of reparse points, appexec links are intentionally unsupported at the I/O level, so trying to traverse them fails with ERROR_CANT_ACCESS_FILE. The process API handles this error by reading the raw data of the reparse point, which contains the real executable path as well as other information that identifies the app. If the user has a license to execute the app, the system creates an access token that allows executing files in the app's installation directory (typically under "%ProgramFiles%\WindowsApps"). It uses this access token to spawn the real executable, basically via the internal implementation of CreateProcessAsUserW().

The implementation of pathlib.Path.resolve() in Python 3.9 only handles FileNotFoundError in its non-strict mode. It fails for all other errors. Since Python 3.9 no longer receives bug fixes, this issue would have to be addressed by tox.

Starting with Python 3.10, the resolve() method of pathlib.Path uses os.path.realpath(). Its non-strict mode is much less strict. In particular, it doesn't fail if a reparse point can't be accessed.

[hhmmjjnn]

Dear @eryksun Thank you so much for your quick and enlightening reply.

Indeed, calling resolve(strict = False) in a fresh version of Python doesn't throw ERROR_CANT_ACCESS_FILE anymore.

Microsoft Windows [versão 10.0.19044.2846]
(c) Microsoft Corporation. Todos os direitos reservados.

C:\Users\myusername>%LOCALAPPDATA%\Programs\Python\Python311\python.exe
Python 3.11.3 (tags/v3.11.3:f3909b8, Apr  4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> import pathlib
>>> p = (os.environ['LOCALAPPDATA'] + '\\Microsoft\\WindowsApps\\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\\python.exe')
>>> pathlib.WindowsPath(p).resolve(False)
WindowsPath('C:/Users/myusername/AppData/Local/Microsoft/WindowsApps/PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0/python.exe')
>>>

I should've had it checked before. My bad 🤦‍♂️

I'll be sending a PR to tox to get it worked around, there.

I would argue that pathlib.Path.readlink() should've been reading the APPEXECLINK_READ_BUFFER structure. As of Python 3.11.3, it won't. But, that's a discussion for a new issue...

Not much left to be done here, so I'm closing the current issue... Thanks, once again.

[eryksun]

I would argue that pathlib.Path.readlink() should've been reading the APPEXECLINK_READ_BUFFER structure.

I disagree. Appexec links are an implementation detail of the Windows process API to support UWP apps, which could have been handled in various completely different ways.

They're not supported by the I/O system. Trying to traverse them fails with STATUS_IO_REPARSE_TAG_NOT_HANDLED (i.e. Windows ERROR_CANT_ACCESS_FILE). Technically, they're not even categorized as filesystem links. Unlike IO_REPARSE_TAG_MOUNT_POINT (0xA0000003) and IO_REPARSE_TAG_SYMLINK (0xA000000C), the tag value IO_REPARSE_TAG_APPEXECLINK (0x8000001B) doesn't include the name-surrogate bit (0x20000000) that indicates that a reparse-point file or directory represents another named entity in the system (i.e. a symbolic link). Supporting them in os.readlink() would be far beyond the specification and intention of POSIX readlink().

Even if none of that were the case, the format of the reparse buffer of an appexec link is not documented by Microsoft, which makes it an internal implementation detail that Microsoft is free to change at any time if it wants. The standard library should be limited to public interfaces that Microsoft documents and supports.