Python 3.11.0 -> 3.11.1 : ...\Python311\DLLs not added to sys.path in embedded startup on Windows

[kxrob]

Bug report

When updating Python 3.11.0 -> 3.11.1 (or reverse to reverse the issue), the ...\Python311\DLLs folder is suddenly not added anymore to sys.path in embedded startup due to somehow impaired search path algorithm. E.g. in Pythonwin.exe. And thus things like ctypes, socket, hashlib etc. cannot be imported. But ...\Python311\Lib and all the other paths are correctly there, just the DLLs path missing.

The same was observed e.g. here:
mhammond/pywin32#1995

The issue is also in current Py 3.12.0a3 at least.
The issue seems not to be with python.exe startup.
The issue also disappears when I monkey-copy the Pythonwin.exe next to python.exe and use that copy.

Note: Pythonwin.exe locates pythonNN.dll dynamically and does the usual Python init.

And extra confusing: in the registry there is a PythonPath key like C:\Python312\Lib\;C:\Python312\DLLs\ :
I always thought that the DLLs path is taken from there. But when I edit-damage that like C:\Python312\Lib\;C:\Python312\DLLsx\ e.g., it has no effect :-)
The correct DLLs dir (only) is still in the sys.path in the above working cases, and DLLsx also does not appear on sys.path in the non-working cases.

Reproduce:

• Win10
• pip install pywin32; run Pythonwin.exe
• import ctypes and/or inspect sys.path after start.

Linked PRs

• gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location #100947
• [3.11] gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location (GH-100947) #101082
• [3.11] gh-100320: Fix path calculations on Windows when python.exe is moved outside of the normal location (GH-100947) #101080

[kxrob]

An also:

When I symlink or copy the Python3.12 or 3.11.1 python.exe + pythonNN.dll somewhere else, e.g. in a \bin folder, this moved python.exe (or renamed pythonNN.exe) doesn't work anymore. This worked before, and does so still for 3.10- at least.

A long error:

Could not find platform independent libraries <prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python312.exe'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = 'C:\bin\Lib'
  sys._base_executable = 'C:\\bin\\python312.exe'
  sys.base_prefix = 'C:\\bin'
  sys.base_exec_prefix = 'C:\\bin'
  sys.platlibdir = 'DLLs'
  sys.executable = 'C:\\bin\\python312.exe'
  sys.prefix = 'C:\\bin'
  sys.exec_prefix = 'C:\\bin'
  sys.path = [
    'C:\\bin\\python312.zip',
    'C:\\bin\\Lib',
    'C:\\bin',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00002f68 (most recent call first):
  <no Python frame>

Interesting also here is the line

sys.platlibdir = 'DLLs'

which is the only entry with non-absolute path, thus "double wrong" - which may point back to the initial error issue.

[eryksun]

When I symlink or copy the Python3.12 or 3.11.1 python.exe + pythonNN.dll somewhere else, e.g. in a \bin folder, this moved python.exe (or renamed pythonNN.exe) doesn't work anymore.

Prior to 3.11, if Python's home directory can't be determined, then the default value of the "PythonPath" key in the registry gets added to sys.path. For some reason this was dropped from the new implementation in 3.11+. It's still the documented behavior, however:

If a Python home is found, the relevant sub-directories added to sys.path (Lib, plat-win, etc) are based on that folder. Otherwise, the core Python path is constructed from the PythonPath stored in the registry.

---

sys.platlibdir = 'DLLs'

which is the only entry with non-absolute path, thus "double wrong" - which may point back to the initial error issue.

The value of sys.platlibdir is used in the platform library landmark. On Windows, this landmark is just "{platlibdir}", i.e. "DLLs". It's looked for in the exec_prefix directory, which on Windows is the directory of the 'real' executable. In fact, this is not necessarily the real path of the executable because the realpath() function that's used in "Modules/getpath.py" does nothing on Windows. (This is because neither _Py_wreadlink() nor _Py_wrealpath() is implemented on Windows.) Thus if the executable path is a symlink, then exec_prefix is initially the directory of the symlink. This is unlikely to contain the "DLLs" landmark directory (i.e. it's unlikely that "DLLs" is linked as well).

If the platform library landmark isn't found, then both exec_prefix and the platform library directory are set to prefix. This is where the standard library "Lib" directory is supposed to be. But if the interpreter DLL and executable are symlinks, then the landmark for the standard library ("Lib\os.py") is also unlikely to be found (i.e. it's unlikely that "Lib" is linked as well). In this case, the values of prefix, exec_prefix, and the platform library are set to the current working directory, and the standard library path "{prefix}\Lib" is added to sys.path even if it doesn't exist.

[kxrob]

So I guess this needs to be re-established at priority due to a bug introduced from 3.11.0 to .1 :

a)

Otherwise, the core Python path is constructed from the PythonPath stored in the registry.

This will solve both issues above (embedded.exe w dynamic loading of python DLL + linked/copied python.exe & .dll) at least as far as to regain the previous function and not break things.

---

In addition, reading the description and the linked doc regarding sys.path initialization (on Windows), the following may be reasonable to add / change in the logic for making it more robust and smart:

Regarding,

If the environment variable PYTHONHOME is set, it is assumed as “Python Home”. Otherwise, the path of the main Python executable is used to locate a “landmark file” (either Lib\os.py or pythonXY.zip) to deduce the “Python Home”. If a Python home is found, the relevant sub-directories added to sys.path (Lib, plat-win, etc) are based on that folder. Otherwise, the core Python path is constructed from the PythonPath stored in the registry.

b)
after trying the executable location as Python Home (can be a embedding.exe dynamically using the python DLL from a python installation) and before trying the fallback registry entries, also the python DLL location should be tried (when existing and different), because that is a stronger indicator for a specific python installation, which may in some cases be different from the registry entry.

c)
In Python 2 when the pythonNN.dll was in system32 for a system install, it was enough to put a link/copy of the only python.exe to e.g. C:\bin\python(NN).exe on the PATH (not for the python DLL) for handy short cut. Of course there are several good reasons to not use the system32 folder. But to regain that function, python.exe could become worth its salt and dynamically locate the python DLL (relative & symlink aware + from registry) regarding Py_Main, similar to Pythonwin.exe, instead of having a hard dependency at Windows link level on pythonNN.dll .

d)
The symlink / real realpath should be used on Windows as well (at priority before registry entry) regarding detection of the most relevant Python Home. (Of course respecting a nearby pyvenv.cfg before dereferencing the symlink)

[skochinsky]

We're also running into this issue. Has anyone figured out a good workaround? Also, how come python.exe works correctly?

[kxrob]

Also, how come python.exe works correctly?

That also works only as far as it sits in the 'home' directory besides DLLs/ and Lib/, see

When I symlink or copy the Python3.12 or 3.11.1 python.exe + pythonNN.dll somewhere else, e.g. in a \bin folder, this moved python.exe (or renamed pythonNN.exe) doesn't work anymore ...

.. and explanations from eryksun and in the linked document.
Its obviously a bug which broke "Otherwise, the core Python path is constructed from the PythonPath stored in the registry." ( a) ) besides general weaknesses of the implemented logic, which may be improved by b) c) d) .

Maybe @zooba @vstinner have an idea what caused the bug?

workaround?

Like mmomtchev (linked issue/commit) I use 3.11.0 until the bug is hopefully fixed again in 3.11.2 . More are facing this bug.
Another workaround would also be possible, but I don't wont to raise memes which trigger adaptation to a bug in the middle of the road.

[skochinsky]

I came up with the following workaround for the time being:

import os
import os.path

if os.name == 'nt' and \
    sys.version_info.major == 3 and \
    sys.version_info.minor >= 11:
    # Python 3.11 has a bug with DLLs directory missing from sys.path
    # so add it if it's not there
    base = sys.base_exec_prefix
    dllspath = os.path.join(base, sys.platlibdir)
    if os.path.exists(dllspath) and dllspath not in sys.path:
        i = sys.path.index(base) if base in sys.path else len(sys.path)
        sys.path.insert(i, dllspath)

[skochinsky]

By the way, Anaconda fork has a similar issue (ignoring PythonPath in registry) which prevents it from being used as embedded runtime:
ContinuumIO/anaconda-issues#11374

[eryksun]

We're also running into this issue. Has anyone figured out a good workaround? Also, how come python.exe works correctly?

Part of the change in 3.11.1 is that it no longer searches up from the executable directory to find exec_prefix, i.e. the directory that contains "DLLs". This change restores compatibility with 3.10 and earlier versions. Searching up would help with the Pythonwin case, but I agree that this is not a secure practice on Windows (e.g. with the default security, any authenticated user can create a directory in the root directory of a volume, including the system volume), and it was a mistake to enable this behavior in 3.11.0. The problem is better addressed in terms of the other changes in 3.11.1.

In 3.10 on Windows, exec_prefix and prefix are always the same, and "{exec_prefix}\DLLs" is added to sys.path whether or not it exists. In 3.11.1, exec_prefix is set to the executable directory and platstdlib_dir is set to "{exec_prefix}\DLLs" only if "DLLs" is found in the executable directory. Otherwise both exec_prefix and platstdlib_dir are set to the same directory as prefix.

I don't understand the cited reason for not setting platstdlib_dir to "{exec_prefix}\DLLs", as was always done in previous versions. The comment says that this "would give site-packages precedence over executable_dir". This might be related to another bug in Modules/getpath.py, in the following code:

    # Then add stdlib_dir and platstdlib_dir
    if os_name == 'nt' and venv_prefix:
        # QUIRK: Windows generates paths differently in a venv
        if platstdlib_dir:
            pythonpath.append(platstdlib_dir)
        if stdlib_dir:
            pythonpath.append(stdlib_dir)
        if executable_dir not in pythonpath:
            pythonpath.append(executable_dir)
    else:
        if stdlib_dir:
            pythonpath.append(stdlib_dir)
        if platstdlib_dir:
            pythonpath.append(platstdlib_dir)

On Windows, executable_dir should always be added, and the order should always be platstdlib_dir, stdlib_dir, executable_dir. I verified this in versions 3.6-3.10, running with -S (no site import) for both the base environment and a virtual environment.

I think the above snippet should be modified to remove the venv_prefix check and always add executable_dir, which should never be the same as stdlib_dir or platstdlib_dir (see further modifications below). For example:

    # Then add stdlib_dir and platstdlib_dir
    if os_name == 'nt':
        # QUIRK: Windows uses a different ordering, and it includes the
        # directory of the executable.
        if platstdlib_dir:
            pythonpath.append(platstdlib_dir)
        if stdlib_dir:
            pythonpath.append(stdlib_dir)
        if executable_dir:
            pythonpath.append(executable_dir)
    else:
        if stdlib_dir:
            pythonpath.append(stdlib_dir)
        if platstdlib_dir:
            pythonpath.append(platstdlib_dir)

I'd modify the code that sets exec_prefix as follows:

    # Detect exec_prefix by searching from executable for the platstdlib_dir
    if PLATSTDLIB_LANDMARK and not exec_prefix:
        if executable_dir:
            if os_name == 'nt':
                # QUIRK: For compatibility and security, do not search for the
                # DLLs directory.
                if isdir(joinpath(executable_dir, PLATSTDLIB_LANDMARK)):
                    exec_prefix = executable_dir
            else:
                exec_prefix = search_up(executable_dir, PLATSTDLIB_LANDMARK, test=isdir)
        if not exec_prefix:
            if EXEC_PREFIX:
                exec_prefix = EXEC_PREFIX
            # QUIRK: On Windows, if DLLs isn't found in the executable
            # directory, don't warn. Just use the fallback that makes
            # exec_prefix the same as prefix, and assume that DLLs exists
            # there. If DLLs doesn't exist, an embedding application can place
            # extension modules in the directory of the executable, which is
            # always in sys.path ahead of site-packages.
            elif os_name != 'nt':
                warn('Could not find platform dependent libraries <exec_prefix>')

    # Fallback: assume exec_prefix == prefix
    if not exec_prefix:
        exec_prefix = prefix

[zooba]

There's a lot of discussion here, but fundamentally it seems like the bug is the registry code not loading properly.

For the (IMHO, broken) scenario where you have a normal Python install but are trying to use it from a different executable, the registry key is how the stdlib should be found. The better layout is to not just copy the Python executable, but also the libraries that match it.¹

We have preferred "adjacent to the executable" since 3.6-ish because it is necessary to be able to run a full copy of Python independent from a "real" install. Before that, you could have had a whole copy of 3.5.5 in one directory and an installed copy of 3.5.1 would break it because the registry key "won".

But we shouldn't be breaking the registry here. I'll take a look. I've not been convinced of any of the other changes that are mentioned.

Footnotes

1. Though it does seem like this is the safest possible approach that involves loading the DLL and initialising it. Could be better if the DLL was initialised properly and provided the paths it needs, but I know that's incredibly painful because... well... it's not how we want 3rd party apps using the Python runtime. Just bring your own copy of it, and don't try and do anything complicated with someone else's install, or else ask the user for the executable path and run it that way. ↩

[zooba]

Looks like I missed the case of reading directly from the PythonPath key, but got the read for all subkeys. The read of the top one only happens when home isn't known, which is how it slipped through all other validation, because there are very few scenarios where that occurs anymore.

Setting PYTHONHOME environment variable before loading the DLL ought to be a suitable workaround for 3.11.1 if needed (and setting that against 3.10 should also skip reading the registry). But I think we'll be okay with a fix for 3.11.2 to read from that key under these circumstances.

[eryksun]

There's a lot of discussion here, but fundamentally it seems like the bug is the registry code not loading properly.

No, the problem is this change, which I discussed in my previous post and made suggestions to fix. Pythonwin works in 3.11.0 and earlier versions without using the fallback sys.path in the registry because we always either use the DLL directory or search up from the executable directory for the prefix landmark ("Lib\os.py"); exec_prefix is set to prefix; and "{exec_prefix}\DLLs" is added to sys.path.

[zooba]

There was a lot going on in that post, I didn't come away feeling like I had a fix :)

In the past, we never used the library path to find prefix except in the case where the zip file existed (or a ._pth file existed). That seems to have changed now, but in previous versions we would always find the prefix from argv0, which was always taken from GetModuleFileName(0).

[Many revisions later]

Okay, I've convinced myself that Eryk's logic for setting exec_prefix is almost good, so I'll add that in as well. We want to keep the platstdlib_dir setting though. For layouts that deliberately keep their .pyd files alongside the executable, you shouldn't be able to install packages that supersede theme. But if we only have the executable dir in sys.path for back-compat reasons, then it should remain near the end of the search path.

With that change, Eryk's change for appending the two/three paths is also good, though we can't assume that executable_dir != exec_prefix, and need to keep the check.

[eryksun]

For layouts that deliberately keep their .pyd files alongside the executable, you shouldn't be able to install packages that supersede theme.

That cannot happen, unless via the PYTHONPATH variable. The base executable directory is supposed to always precede site-packages. The current behavior handles this incorrectly compared to 3.10 and earlier. It's supposed to be:

    # Then add stdlib_dir and platstdlib_dir
    if os_name == 'nt':
        # QUIRK: Windows uses a different ordering, and it includes the
        # directory of the executable.
        if platstdlib_dir:
            pythonpath.append(platstdlib_dir)
        if stdlib_dir:
            pythonpath.append(stdlib_dir)
        if executable_dir:
            pythonpath.append(executable_dir)

[zooba]

I wonder what situation I ran into that convinced me otherwise? Hmm

[zooba]

In any case, the change I just pushed should satisfy things. The only thing we're discussing now is whether DLLs should precede Lib in the case where DLLs is assumed to be the same as prefix because we couldn't find it. I think it should precede Lib, rather than switching back and forth, as I consider "location where our standard .pyd's live" to be the main reason we are including the directory at all. executable_dir being in there is considered a mistake that we can't fix, but it can at least remain at the end of the list when that's the only reason we have to include it.

[eryksun]

The code in 3.10 is partly based on the PYTHONPATH macro, which used to be defined as L".\\DLLs;.\\lib" in PC/pyconfig.h. It proceeds as follows:

cpython/PC/getpathp.c

Lines 877 to 910 in a3b6577

	} else {
	const wchar_t *p = PYTHONPATH;
	const wchar_t *q;
	size_t n;
	for (;;) {
	q = wcschr(p, DELIM);
	if (q == NULL) {
	n = wcslen(p);
	}
	else {
	n = q-p;
	}
	if (p[0] == '.' && is_sep(p[1])) {
	if (wcscpy_s(buf, bufsz - (buf - start_buf), calculate->home)) {
	return INIT_ERR_BUFFER_OVERFLOW();
	}
	buf = wcschr(buf, L'\0');
	p++;
	n--;
	}
	wcsncpy(buf, p, n);
	buf += n;
	*buf++ = DELIM;
	if (q == NULL) {
	break;
	}
	p = q+1;
	}
	}
	if (argv0_path) {
	wcscpy(buf, argv0_path);
	buf = wcschr(buf, L'\0');
	*buf++ = DELIM;
	}

Thus it adds (f"{home}\\DLLs", f"{home}\\lib", f"{argv0_path}").

In a 3.10 virtual environment, argv0_path goes through a couple steps if the venv launcher is used. First it changes to the value of the environment variable __PYVENV_LAUNCHER__ in order to find "pyvenv.cfg". Then argv0_path is set to the "home" value in the venv configuration, which should be the base executable directory (venv gets the "home" value from splitting sys._base_executable).

In 3.10, sys.exec_prefix is always the same as sys.prefix (i.e. home). So the "DLLs" directory would always be found relative to prefix, if it's found at all. You changed it in 3.11 to allow using the executable directory for sys.exec_prefix if the "DLLs" landmark is found. I'm okay with that change. In 3.11.1, you modified it to no longer search the parent directories for the "DLLs" landmark, relative to the executable directory. I think it was the right decision. The search up strategy is not secure on Windows, both in terms of file security and layout.

[skochinsky]

Are there tests in CPython test suite which test embedded scenarios? If so, they should be extended with testing native module imports (like ctypes). It seems things got broken multiple times in the past because "python.exe works fine so it's all good".

[zooba]

Yes, but we don't test this scenario because it's not a supported one.

If you want to embed Python, embed all of it, or else configure the paths properly when you initialise it. Taking the executable and hoping it will find an existing install isn't the way.

[skochinsky]

assuming we want to rely on an existing full install, is there a howto/example of how to "configure the paths properly" which would not have been broken by this issue?

[zooba]

The example at https://docs.python.org/3/c-api/init_config.html#initialization-with-pyconfig basically shows it, and the rest of that page covers things in more detail.

For a full install though, your entry point is python.exe, not pythonXY.dll. Directly using the DLL from an install you're not running inside of isn't a supported path - it's totally supported for you to embed the DLL into your own app, but not to "borrow" it from an install.

So the other way that wouldn't have been broken is to run python.exe out of the install, rather than your own executable. If there's something you need to do as part of initialization that requires you to use your own executable, then you most likely should be embedding a full copy of Python. (For comparison, Blender embeds a full copy of Python, because they need to do this, while OBS Studio merely adds search paths for their own modules and so can run through a regular python.exe.)

[zooba]

If there's something you need to do as part of initialization that requires you to use your own executable ...

Incidentally, I'd love to hear about these if you have them, because I haven't come across anything before that really needed it (without also needing to control the rest of the Python runtime, i.e. bring their own version). Not knowing makes it hard to factor in the scenarios when we're looking at what can/should/might change.

[zooba]

Taking the fix as it stands, so we've at least got something.

[aundro]

If there's something you need to do as part of initialization that requires you to use your own executable ...

Incidentally, I'd love to hear about these if you have them, because I haven't come across anything before that really needed it (without also needing to control the rest of the Python runtime, i.e. bring their own version). Not knowing makes it hard to factor in the scenarios when we're looking at what can/should/might change.

Igor spotted this issue in the context of IDAPython.
By design, IDAPython does not use python.exe at all, but rather loads the installed Python DLL/so/dylib, precisely so that the user can use whatever method they want to control the packages that are installed on their system (including distribution packages wherever supported.)

[xyzzy42]

Igor spotted this issue in the context of IDAPython. By design, IDAPython does not use python.exe at all, but rather loads the installed Python DLL/so/dylib, precisely so that the user can use whatever method they want to control the packages that are installed on their system (including distribution packages wherever supported.)

I do this in my app as well. I don't see how I could possibly use the python executable, as I need the interpreter running inside the same address space as the rest of the code so I can have shared access to the same data.

Using an existing python library means the user can install Python and their packages any numbers of ways, apt, dnf, pip, msys2, anaconda, homebrew, etc.

Why would I want to get into the cross platform Python packaging and installation business?! That's crazy. And it's already been done.