ctypes loads wrong version of C runtime, leading to error message box from system

[dancol]

BPO	17213
Nosy	@amauryfa, @mdickinson, @abalkin, @ezio-melotti, @asvetlov, @meadori, @eryksun

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-25.11:39:24.389>
created_at = <Date 2013-02-16.05:33:20.824>
labels = ['ctypes', 'type-bug']
title = 'ctypes loads wrong version of C runtime, leading to error message box from system'
updated_at = <Date 2021-02-25.11:39:24.389>
user = 'https://bugs.python.org/dancol'

bugs.python.org fields:

activity = <Date 2021-02-25.11:39:24.389>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-25.11:39:24.389>
closer = 'eryksun'
components = ['ctypes']
creation = <Date 2013-02-16.05:33:20.824>
creator = 'dancol'
dependencies = []
files = []
hgrepos = []
issue_num = 17213
keywords = []
message_count = 9.0
messages = ['182212', '182700', '194562', '209307', '210095', '210102', '269992', '270036', '270041']
nosy_count = 13.0
nosy_names = ['amaury.forgeotdarc', 'mark.dickinson', 'belopolsky', 'ezio.melotti', 'asvetlov', 'cgohlke', 'meador.inge', 'palm.kevin', 'eryksun', 'dancol', 'Igor.Skochinsky', 'Artfunkel', 'Keno Fischer']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue17213'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[dancol]

Suppose we're running a Python program in an environment where PATH contains a directory that contains msvcr90.dll version A and that python27.exe is manifested to use msvcr90.dll version B, which is available in the SxS store but not on PATH.

Normally, python27.exe's side-by-side (SxS) manifest contains a description of *precisely* the correct version of msvcr90.dll to use, version B, so when python27.exe starts, the NT loader ignores msvcr90.dll version A and loads msvcr90.dll version B.

Everything works fine until somebody calls ctypes.CDLL("c"); uuid.py, which tries to find "uuid_generate_random" in libc on module load, is an example of a component that unexpectedly tries to load libc through ctypes.

Now, when someone tried to load "c", ctypes internally maps "c" to the C runtime library using ctypes.util.find_msvcrt, then calls _ctypes.LoadLibrary, which calls the Win32 API function LoadLibraryA. LoadLibraryA tries to find "msvcr90.dll", but WITHOUT CONSULTING THE SXS ACTIVATION CONTEXT, meaning that LoadLibrary finds msvcr90.dll version A, not version B. msvcr90.dll version A isn't loaded yet, so the NT loader does the usual loading and initialization; msvcr90.dll version A's DllMain runs, notices that it's not being loaded as part of an SxS manifest, and presents the user with an R6034 error message, "an application has made an attempt to laod the C runtime library incorrectly".

The overall result is that users of Python programs see error message popups from "Microsoft Visual C++ Runtime Library" that, in most cases, are completely benign. This problem doesn't occur if the correct version of msvcr90.dll happens to be in PATH.

One solution is to have _ctypes.LoadLibrary use the correct activation context; another is to use GetModuleHandleEx(GET_MODULE_HANDLE_EX_FLAG_FROM_ADDRESS, (LPCTSTR) &fopen, &module) to retrieve a handle to the current C runtime library without having to go through LoadLibrary at all.

[ezio-melotti]

Thanks for the report. Could you also provide a patch?

[IgorSkochinsky]

Just had this issue when using networkx (which imports uuid). One keyword that would help visibility is R6034 (the runtime error number). A couple of reports related to this:

https://projects.blender.org/tracker/index.php?func=detail&aid=27666
http://forums.boxee.tv/showthread.php?t=15425

The proper solution is to fix ctypes, but in the meantime we can at least fix uuid. The error seems to be triggered by this snippet:

    # The uuid_generate_* routines are provided by libuuid on at least
    # Linux and FreeBSD, and provided by libc on Mac OS X.
    for libname in ['uuid', 'c']:
        try:
            lib = ctypes.CDLL(ctypes.util.find_library(libname))
        except:
            continue
        if hasattr(lib, 'uuid_generate_random'):
            _uuid_generate_random = lib.uuid_generate_random
        if hasattr(lib, 'uuid_generate_time'):
            _uuid_generate_time = lib.uuid_generate_time

Since this code is useless on Windows, protecting it in "if os.name not in ['nt', 'ce']" does the trick.

BTW, instead of going all way with activation context etc., a simpler solution would be to add to ctypes something like this in CDLL.__init__:

if os.name in ['nt', 'ce'] and name == util.find_msvcrt():  # TODO: handle the extension
  self._handle = windll.kernel32.GetModuleHandleA(self._name)

i.e. use the already present runtime DLL instead of trying to load it again.

[Artfunkel]

Still an issue in 3.3.3. Igor's fix still works, thankfully.

[palmkevin]

+1

[palmkevin]

Reproducible for Py 3.2.5

[KenoFischer]

pyzmq has a similar issue. I believe one solution would be to call
_Py_ActivateActCtx in ctypes' load_library, i.e. do

#if HAVE_SXS
        ULONG_PTR cookie = _Py_ActivateActCtx();
#endif
        hMod = LoadLibraryW(name);
#if HAVE_SXS
        _Py_DeactivateActCtx(cookie);
#endif

in that function. I don't know enough about python or ctypes to say whether this patch works as is (_Py_ActivateActCtx seems to be an internal function, not sure if ctypes has access to that).

[eryksun]

This is only a problem for ctypes when python27.dll is used in an application that isn't manifested to load the "Microsoft.VC90.CRT" assembly. ctypes doesn't have this problem for a regular script run via the 2.7 version of python.exe, since the loader uses the python.exe activation context. Daniel claimed otherwise, but his claim is demonstrably false. Maybe he was embedding Python.

Keno, I don't think it's a good idea to arbitrarily set the Python DLL's activation context every time ctypes calls LoadLibrary. I proposed a more generic solution in bpo-24429, msg246754.

An alternative solution would be to implement a pure-Python activation context class that works as a Python context manager. For example, it could create and activate a context using the hModule and lpResourceName fields:

with ctypes.ACTCTX(module_handle=sys.dllhandle, 
                   resource_name=2):
    libc = ctypes.CDLL('msvcr90')

I have a Stack Overflow answer that demonstrates using activation contexts with ctypes:

http://stackoverflow.com/a/27392347/205580

[KenoFischer]

Yes, you are correct about it being only an issue in the embedding context.
I agree that it might not be a good idea to do do this for every library, but I wanted to revive the discussion since this kind of thing seems like something that comes up frequently when people embed python. Thanks for the reference to the other thread. A solution that allows the activation context to be specified would be great.