Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

import limit cap? #3211

Open
kdschlosser opened this issue Oct 26, 2019 · 5 comments

Comments

@kdschlosser
Copy link

@kdschlosser kdschlosser commented Oct 26, 2019

I have a project I am working on. it has close to 200 modules. This is the issue I am running into.

I need to import 132 of the modules into a single module. this single module is python code and not a compiled c extension. the 132 modules I am needing to import are c extensions compiled with Cython.

no matter the arrangement of the imports it always fails at import number 108. It simply spits out an error stating module failed to initialize and the program exits. it's the same error no matter what module I put as the 108th import.

what could be causing this problem? and what would be a solution?

@kdschlosser

This comment has been minimized.

Copy link
Author

@kdschlosser kdschlosser commented Oct 26, 2019

As it turns out there is some kind of a limit. The limit is in Python. I do not know what the limit is. I know there is a limit in place because I removed one of the imports of a pyd right at the beginning of my module import and the error moved onto the next import.

I do not know if it is a limit imposed on a per library bases. or if it is for python as a whole. I am going to import ctypes and see if it is python as a whole.

@scoder

This comment has been minimized.

Copy link
Contributor

@scoder scoder commented Oct 27, 2019

@kdschlosser

This comment has been minimized.

Copy link
Author

@kdschlosser kdschlosser commented Oct 27, 2019

The code runs fine if i reduce the number of pyd files to 123. as soon as I add the 124th it get an error stating the dll init failed. and it does not matter what extension it is. so it is not a circular import problem.

I am compiling python code into extensions only to gain speed of code execution. so i have the ability to swap out the pyd files for py files. I thought maybe it was because of the number of imports taking place in a single file. so i tried spreading the imports across multiple files.. it still failed on number 124..

I then put into place a lazy import mechanism which only would import files as they are needed. this moved the problem to much later on in the programs execution but it still happened on the 124th file.

I also tried using importlib.import_module and imp.load_dynamic. still the same result.

@Yuriy-Leonov

This comment has been minimized.

Copy link

@Yuriy-Leonov Yuriy-Leonov commented Nov 17, 2019

@kdschlosser @scoder
I've created repository which could be used for reprodusing such issue - https://github.com/Yuriy-Leonov/cython_imports_limit_issue

Reproducing steps are described in README.

@Yuriy-Leonov

This comment has been minimized.

Copy link

@Yuriy-Leonov Yuriy-Leonov commented Nov 17, 2019

Looks like this is an issue not related to cython itself, but for windows - https://bugs.python.org/issue38597

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.