C Extension import limit

[KevinSchlossser]

BPO	38597
Nosy	@pfmoore, @scoder, @tjguk, @merwok, @zware, @zooba, @dstufft, @pganssle, @miss-islington, @JoeXinfa
PRs	bpo-38597: Never statically link extension initialization code on Windows #18724 [3.8] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) #18758 [3.7] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) #18759
Files	setup.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-03.18:08:03.183>
created_at = <Date 2019-10-26.19:48:20.615>
labels = ['3.8', '3.7', 'library', '3.9', 'performance']
title = 'C Extension import limit'
updated_at = <Date 2021-02-03.18:08:03.182>
user = 'https://bugs.python.org/KevinSchlossser'

bugs.python.org fields:

activity = <Date 2021-02-03.18:08:03.182>
actor = 'steve.dower'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-03.18:08:03.183>
closer = 'steve.dower'
components = ['Distutils']
creation = <Date 2019-10-26.19:48:20.615>
creator = 'Kevin Schlossser'
dependencies = []
files = ['48918']
hgrepos = []
issue_num = 38597
keywords = ['patch']
message_count = 23.0
messages = ['355425', '355578', '356824', '356887', '356892', '359314', '359395', '359403', '362529', '362549', '362557', '362653', '362756', '362793', '362795', '362796', '363067', '363080', '363157', '363222', '363225', '363226', '386268']
nosy_count = 13.0
nosy_names = ['paul.moore', 'scoder', 'tim.golden', 'eric.araujo', 'zach.ware', 'steve.dower', 'dstufft', 'p-ganssle', 'miss-islington', 'Kevin Schlossser', 'Yurii Leonov', 'xinfazhu', 'P12 Pfx']
pr_nums = ['18724', '18758', '18759']
priority = 'normal'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue38597'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[KevinSchlossser]

System
Windows 7 x64 SP2
Ram 16GB
6 Core AMD @ 3.2ghz

CPython 3.7.2

C Extension (pyd) import cap.

There seems to be a cap on the number of extensions that a package is able to contain. I am able to import 123 extension modules that my package has but when i go to import number 124 i get the following traceback

ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

these extension modules are part of my package, importing an extension module from another package does not change this behavior. it is only when I import the 124th extension that is in my package does it occur.

When I change the order of the imports the error does not follow the import. I end up getting the same error when the 124th extension gets loaded doesn't matter what extension it is.

I have tried to see if maybe it was a module limit and I spread the imports across multiple files and it still fails when the 124th gets loaded. I also tried imp.load_dynamic and importlib.import_module and the same error occurs.

If there is a way to work around this limitation it would be very helpful.

[zooba]

There is an implicit cap due to the C runtime (specifically vcruntime140.dll) allocating fibre local storage on load, which seems like the one you are hitting. However, we discovered this before the first 3.5 release and fixed it.

How are you compiling your packages? Or where are you getting them from?

It's possible that someone else is building and statically linking the C runtime, which will cause this, but distutils (and hence CPython) should not be doing it by default.

[YuriiLeonov]

Looks like I have same problem for Windows 10 (version 1809, build - 17763.864).

I created repository with steps for reproducing - https://github.com/Yuriy-Leonov/cython_imports_limit_issue

[zooba]

Could you share just one of your .pyd files?

Without being able to see whether they are compiled incorrectly, it's hard to be sure whether this is the same cause as before. It certainly looks like distutils is still going to link correctly.

[YuriiLeonov]

Done:
.pyd files are added in Yuriy-Leonov/cython_imports_limit_issue@2f9e7c0

folder with name "dist_example_with_error"

[KevinSchlossser]

Thank you msg356892 for spear heading this for me. I family things to attend to so I apologize for opening this bug report and then walking away..

As far as recreating this issue. It simple to do. you can either use cython or you can put together a quick script that will make say 150 extensions. put one line of code in it and compile it.. The results always end up being the same 123 extensions can be loaded on the 124th a TB happens.. It does not matter where the files are inside the package. it is always the 124th one that gets loaded for any one package. importing a pyd from another package does not change the number. The extensions have to be a direct descendant of one single package.

[zooba]

I haven't looked into _why_ yet, but the first PYD I grabbed from the GitHub link above has had the CRT statically linked. This is not the default (or it should not be), because when we made it the default this exact issue occurred :)

If somehow the default linking mode in distutils has changed, we should fix that. If it is being overridden by Setuptools or Cython then we should get those projects fixed.

Stefan/Paul - do Cython or Setuptools override compiler/linker settings like this at all? Most likely it's the /MT vs /MD option

[scoder]

Cython doesn't interfere with the C compiler setup in any way, that's left
to distutils/setuptools (and the user).

[JoeXinfa]

I have a similar issue. Do we have an estimate how long it may take to fix this bug? Thanks. I can help but would need some mentoring.

[zooba]

Okay, looking at _find_vcvarsall in distutils, I'm guessing that something about how your machines are set up means that the vcredist search is failing.

First, could you specify which versions of Visual Studio you have installed, and if possible which one is being found and used by your builds.

Then, if you can search your install to find both vcvarsall.bat and vcruntime140.dll (there will be a few of these) and post the paths, that may indicate if the layout isn't consistent.

Distutils will try and dynamically link to the runtime if you have the redist, and statically link it if you don't (though I don't remember why it doesn't just rely on the copy included with Python... probably future-proofing or licencing).

[JoeXinfa]

Thanks Steve. Here is what you requested.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ ./Installer/vswhere.exe
Visual Studio Locator version 2.7.1+180c706d56 [query version 2.3.2200.14893]
Copyright (C) Microsoft Corporation. All rights reserved.

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcvarsall.bat"
./2019/BuildTools/VC/Auxiliary/Build/vcvarsall.bat

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio
$ find . -name "vcruntime140.dll"
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/onecore/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x64/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Redist/MSVC/14.24.28127/x86/Microsoft.VC142.CRT/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx64/x86/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x64/vcruntime140.dll
./2019/BuildTools/VC/Tools/MSVC/14.24.28314/bin/Hostx86/x86/vcruntime140.dll
./Installer/resources/app/ServiceHub/Services/Microsoft.VisualStudio.Setup.Service/vcruntime140.dll
./Installer/vcruntime140.dll

xinfa@LAPTOP-71TBJKSA MINGW64 /c/Program Files (x86)/Microsoft Visual Studio

I want mention that I have 301 extension modules. I used setuptools in my setup.py
from setuptools import setup
from setuptools.extension import Extension
python setup.py build_ext
python setup.py bdist_wheel
I bring the whl file to another computer, pip install, then launch the app and I get this error
ImportError: DLL load failed: A dynamic link library (DLL) initialization routine failed.

I tried using distutils instead, but it says: error: invalid command 'bdist_wheel'
from distutils.core import setup
from distutils.extension import Extension

[zooba]

You should be able to install "wheel" without setuptools to get the bdist_wheel command.

Can you confirm that the build process is actually using that Visual Studio install? If it's going through the regular distutils detection process then it ought to be finding the right files.

[JoeXinfa]

I have had wheel installed. Following this SO answer (https://stackoverflow.com/a/50314071/7269441), I added "import setuptools" before distutils (my setup.py attached).

Following is the log, where we see the VS path it is using. BTW, I am trying to reduce my package to <120 modules and test it.

(base) C:\Users\xinfa\Documents\code\ezcad-dev\beta>C:\Users\xinfa\AppData\Local\Continuum\anaconda3\python.exe setup.py build_ext
Include directory: C:\Users\xinfa\Documents\code\ezcad-dev\beta
Number of ext modules = 225
running build_ext
building 'ezcad.__main__' extension
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT -IC:\Users\xinfa\Documents\code\ezcad-dev\beta -I. -IC:\Users\xinfa\AppData\Loca
l\Continuum\anaconda3\include -IC:\Users\xinfa\AppData\Local\Continuum\anaconda3\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\include" "-IC:\Program Files (x86)\Windows Kits
\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.1836
2.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tcezcad\main.c /Fobuild\temp.win-amd64-3.7\Release\ezcad\main.obj
__main__.c
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\bin\HostX86\x64\link.exe /nologo /INCREMENTAL:NO /LTCG /nodefaultlib:libucrt.lib ucrt.lib /DLL /MANIFEST:EMBED,ID=2 /MANIFESTUAC:NO /LIBPAT
H:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\libs /LIBPATH:C:\Users\xinfa\AppData\Local\Continuum\anaconda3\PCbuild\amd64 "/LIBPATH:C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.24.28314\l
ib\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\ucrt\x64" "/LIBPATH:C:\Program Files (x86)\Windows Kits\10\lib\10.0.18362.0\um\x64" build\temp.win-amd64-3.7\Release\ezcad\main.obj /OUT:build\lib.win
-amd64-3.7\ezcad\main.cp37-win_amd64.pyd
Creating library build\lib.win-amd64-3.7\ezcad\main.cp37-win_amd64.lib and object build\lib.win-amd64-3.7\ezcad\main.cp37-win_amd64.exp
Generating code
Finished generating code
version = 0.1.9

[JoeXinfa]

FYI when I reduced my package to 106 extension modules, I could run without the DLL error.

[zooba]

Thanks, it does seem like it's finding the correct MSVC, but is not finding the redistributable DLL:

cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MT ...

The "/MT" means to statically the CRT, and then we use link settings later to dynamically link the part installed in the OS (see https://stevedower.id.au/blog/building-for-python-3-5-part-two for more details).

The problem seems to be that this glob in distutils can't find the path listed below it:

"2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll"

"2019\BuildTools\VC\Redist\MSVC\14.24.28127\x64\Microsoft.VC142.CRT\vcruntime140.dll"

We swallow a lot of errors while doing this glob, so I assume there's something about your install that's affecting it (maybe related to it being BuildTools rather than Community or higher?)

Could you try running this code and see what result/error you get:

import glob
glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll", recursive=True)

[zooba]

Hmm, just ran it on my own and it's finding the OneCore file first, which is not a problem (yet) but could become one.

So we at least need to replace the "**" with a "*" in Lib/distutils/_msvccompiler.py#L109. Not sure if that will help in this other case or not.

[zooba]

In thinking about this, I think the best way forward is to just remove the logic that might statically link the initialization code, and instead commit to CPython releases always including vcruntime140.dll even if we switch to a newer version one day.

Hopefully third party distributions will do the same, though it should only matter for ABI3 modules that are not recompiled for the newer versions.

[JoeXinfa]

Steve, don't know if you still need it but here is what you requested. Sorry for the slow move (I was working on something else). Seems mine is finding the x64 before the OneCore, though I don't know the significance.

Python 3.7.1 (default, Dec 10 2018, 22:54:23) [MSC v.1915 64 bit (AMD64)] on win32
In[2]: import glob
In[3]: glob.glob(r"C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\redist\MSVC\**\x64\Microsoft.VC14*.CRT\vcruntime140.dll", recursive=True)
Out[3]:
['C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll',
'C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\BuildTools\\VC\\redist\\MSVC\\14.24.28127\\onecore\\x64\\Microsoft.VC142.CRT\\vcruntime140.dll']

[zooba]

don't know if you still need it but here is what you requested.

Thanks. That basically confirms that something is interfering with distutils. Skipping the check entirely should avoid it.

[zooba]

New changeset ce3a498 by Steve Dower in branch 'master':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
ce3a498

[miss-islington]

New changeset 8a5f7ad by Steve Dower in branch '3.7':
[3.7] bpo-38597: Never statically link extension initialization code on Windows (GH-18724) (GH-18759)
8a5f7ad

[miss-islington]

New changeset 0d20364 by Miss Islington (bot) in branch '3.8':
bpo-38597: Never statically link extension initialization code on Windows (GH-18724)
0d20364

[zooba]

Distutils is now deprecated (see PEP-632) and all tagged issues are being closed. From now until removal, only release blocking issues will be considered for distutils.

If this issue does not relate to distutils, please remove the component and reopen it. If you believe it still requires a fix, most likely the issue should be re-reported at https://github.com/pypa/setuptools