Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Regression for no attribute '__reduce_cython__' in Cython 0.29 #2985

Open
mobiusklein opened this issue Jun 5, 2019 · 4 comments

Comments

Projects
None yet
2 participants
@mobiusklein
Copy link

commented Jun 5, 2019

Platform: Windows 10
Python Version: 3.6.7

When running a program built with PyInstaller, cdef class types built with Cython 0.29 or later fail during extension module initialization with
AttributeError: type object '<type>' has no attribute '__reduce_cython__'.

I can confirm that the fix in Cython 0.28.3-2.8.6 still works.

The program works without issue when run from Python instead of the PyInstaller bundle.

@mobiusklein mobiusklein changed the title Regression in for no attribute '__reduce_cython__' in Cython 0.29 Regression for no attribute '__reduce_cython__' in Cython 0.29 Jun 5, 2019

@scoder

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

I assume you read #1953, especially #1953 (comment)

Is the Cython version the only thing you changed that made it fail? (Same CPython, same PyInstaller version, …)

Did you verify that the generated C code that fails comes from a 0.29.x version? (First line in the file.)

Did you verify that your code does not have circular imports?

I have no experience with PyInstaller and cannot say what it does internally. To debug this, you can search for Re-initialisation is not supported in the Cython generated C file and add a couple of C printf() debug statements here and there, to see what code path it takes and how often.

@mobiusklein

This comment has been minimized.

Copy link
Author

commented Jun 7, 2019

Cython version is the only variable.

Building a toy example does not encounter this problem, but when I include a large module with many C-extensions, I see __pyx_pymod_exec_<module> gets called twice from PyInstaller, but not Python. I've checked that all of the generated C files for that module were produced by Cython 0.29. To my knowledge, there are no circular imports, though the extension module initialization order does differ when run through PyInstaller compared to Python.

Curiously, when passing through the __pyx_pymod_exec_<module> function, the re-initialization is not detected:

static CYTHON_SMALL_CODE int __pyx_pymod_exec_<module>(PyObject *__pyx_pyinit_module)
#endif
#endif
{
  PyObject *__pyx_t_1 = NULL;
  int __pyx_t_2;
  static PyThread_type_lock __pyx_t_3[8];
  __Pyx_RefNannyDeclarations
  printf("Begin exec_peak_statistics\n");
  #if CYTHON_PEP489_MULTI_PHASE_INIT
  printf("BEGIN MULTIPHASE_INIT\n");
  if (__pyx_m) { //<<< Never True
    if (__pyx_m == __pyx_pyinit_module) return 0;
    PyErr_SetString(PyExc_RuntimeError, "Module '<module>' has already been imported. Re-initialisation is not supported.");
    printf("Module '<module>' has already been imported. Re-initialisation is not supported.\n");
    return -1;
  }
  #elif PY_MAJOR_VERSION >= 3
  if (__pyx_m) return __Pyx_NewRef(__pyx_m);
  #endif

When the program is run with PyInstaller, my print calls run like so:

Begin exec_<module>
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...
Begin exec_<module>
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...

When run with Python, instead I get:

Begin exec_<module>
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...

So my print call in the block that tests to see if the module has been re-initialized is never executed. During both passes through the exec_<module> function, __pyx_m is NULL and __pyx_pyinit_module is some arbitrary pointer. __pyx_pyinit_module does not point to the same location when PyInstaller calls exec_<module> for the second time:

Begin exec_<module>
BEGIN MULTIPHASE_INIT
__pyx_m = 0000000000000000
__pyx_pyinit_module = 000002ACCC241AE8
Initializing module...
Caching module...
Begin exec_<module>
BEGIN MULTIPHASE_INIT
__pyx_m = 0000000000000000
__pyx_pyinit_module = 000002ACCC2206D8
Initializing module...
Caching module...
@scoder

This comment has been minimized.

Copy link
Contributor

commented Jun 7, 2019

Thanks for the investigations. Could you try a couple of more things? There is a function called __pyx_module_cleanup. Please add debug prints to that, too, to see if it is called in between the imports. Also, please add a printf() at the end of the exec function to see if the second import happens before or after terminating it. If it's within the first call, that would suggest that something in your module code triggers a re-import of the module. If it happens afterwards, that could suggest that PyInstaller is triggering it itself.

__pyx_pyinit_module is the module object created for the import, so it's normal that it refers to different objects on different imports.

Could it be that you are importing the module in different ways in your code? E.g. with absolute and relative imports, or with both import pkg.module and from pkg import module? It might be that PyInstaller is unable to detect that both refer to the same module, and then it might end up asking CPython to import the same module twice, which could make it re-initialise the same extension module.

It is weird, though, that __pyx_m is NULL in both cases. I assume it's being set where it says Caching module in your output? After that, it should have a non-NULL value…

Also, could you switch to 0.29.10 please, or even give the latest master a try? Just to be sure we're not chasing zombies.

@mobiusklein

This comment has been minimized.

Copy link
Author

commented Jun 7, 2019

I switched to 0.29.10:

Unfortunately, I can't find any symbol with the word "cleanup" in it.

Here's an expanded log for PyInstaller:

Begin exec_<module>
__pyx_m = 0000000000000000
__pyx_pyinit_module = 000001D45C300AE8
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...
Begin populating globals...
An error occured? __pyx_lineno 1
__pyx_clineno 34110
Py_CLEAR(__pyx_m)
Returning -1
Begin exec_<module>
__pyx_m = 0000000000000000
__pyx_pyinit_module = 000001D45C2E06D8
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...
Begin populating globals...
An error occured? __pyx_lineno 621
__pyx_clineno 33939
Py_CLEAR(__pyx_m)
Returning -1

and for Python:

Begin exec_<module>
__pyx_m = 0000000000000000
__pyx_pyinit_module = 000001BCC3E7BF48
BEGIN MULTIPHASE_INIT
Initializing module...
Caching module...
Begin populating globals...
NumPy C-API Initialized
...Done populating globals
__pyx_m = 000001BCC3E7BF48
Returning 0

This shows the PyInstaller ran module initialization is quitting somewhere in the init code and going to the error handling bock. It looks like the failure state differs between passes through the initialization block, but the module globals might have been partially initialized.

The first error happened in __Pyx_modinit_type_import_code, on a line importing a C module which no Python code imports directly. This made it a "hidden import", which I have to tell PyInstaller about explicitly. However, the explicit notification method I used was written for Python 2, and there was a small change required for Python 3. Without it , the relevant .so/.pyd isn't included in the bundle, and attempting to import it should error out. I would have expected that to bubble up to the top immediately with an ImportError, as it does on Py2, but alas it instead just leaves the module in a partially incomplete state that the next import attempt runs into. When I properly specify the hook for Py3, the bundle works without issue.

So the problem appears to be not that the generated __reduce__ methods were missing, but that when they were queried and failed was somehow the next moment that Python errors could be raised?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.