Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler crashes while analyzing fused method with optional arguments only on Python 3.11 #5894

Closed
mobiusklein opened this issue Dec 5, 2023 · 2 comments · Fixed by #5896
Milestone

Comments

@mobiusklein
Copy link

Describe the bug

I've observed that Cython crashes when analyzing a cdef class with a cdef method taking fused type arguments and optional arguments, but not always consistently, and only on Python 3.11. It seemed to affect all versions of Cython 3, including the current master branch.

Error compiling Cython file:
------------------------------------------------------------
...
        return self._discover_peaks(mz_array, intensity_array, _start_mz, _stop_mz, _start_index, _stop_index)

    @cython.nonecheck(False)
    @cython.cdivision(True)
    @cython.boundscheck(False)
    cpdef size_t _discover_peaks(self, np.ndarray[cython.floating, ndim=1, mode='c'] mz_array, np.ndarray[cython.floating, ndim=1, mode='c'] intensity_array,
          ^
------------------------------------------------------------

src\ms_peak_picker\_c\peak_picker.pyx:248:10: Compiler crash in AnalyseDeclarationsTransform

ModuleNode.body = StatListNode(peak_picker.pyx:1:0)
StatListNode.stats[22] = StatListNode(peak_picker.pyx:140:5)
StatListNode.stats[0] = CClassDefNode(peak_picker.pyx:140:5,
    as_name = 'PeakProcessor',
    class_name = 'PeakProcessor',
    module_name = '',
    punycode_class_name = 'PeakProcessor',
    visibility = 'private')
CClassDefNode.body = StatListNode(peak_picker.pyx:142:4)
StatListNode.stats[8] = CompilerDirectivesNode(peak_picker.pyx:248:10)
CompilerDirectivesNode.body = StatListNode(peak_picker.pyx:248:10)
StatListNode.stats[0] = CFuncDefNode(peak_picker.pyx:248:10,
    args = [...]/7,
    doc = 'Carries out the peak picking process on `mz_array` and `intensity_array`. All\n        peaks picked are appended to :attr:`peak_data`.\n\n        Parameters\n        ----------\n        mz_array : np.ndarray\n            The m/z values to pick peaks from\n        intensity_array : np.ndarray\n            The intensity values to pick peaks from\n        start_mz : float, optional\n            The minimum m/z to pick peaks above\n        stop_mz : float, optional\n        
    The maximum m/z to pick peaks below\n\n        Returns\n        -------\n        int\n            The current number of peaks accumulated\n        ',
    has_fused_arguments = True,
    is_c_class_method = 1,
    modifiers = [...]/0,
    outer_attrs = [...]/2,
    overridable = 1,
    visibility = 'private')
File 'FusedNode.py', line 60, in __init__: FusedCFuncDefNode(peak_picker.pyx:248:10,
    fused_compound_types = [...]/1,
    match = "dest_sig[{{dest_sig_idx}}] = '{{specialized_type_name}}'",
    no_match = 'dest_sig[{{dest_sig_idx}}] = None',
    nodes = [...]/2)
File 'FusedNode.py', line 201, in copy_cdef: FusedCFuncDefNode(peak_picker.pyx:248:10,
    fused_compound_types = [...]/1,
    match = "dest_sig[{{dest_sig_idx}}] = '{{specialized_type_name}}'",
    no_match = 'dest_sig[{{dest_sig_idx}}] = None',
    nodes = [...]/2)

Compiler crash traceback from this point on:
  File "c:\users\joshua\dev\cython\Cython\Compiler\FusedNode.py", line 200, in copy_cdef
    cindex = env.cfunc_entries.index(self.node.entry)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\joshua\dev\cython\Cython\Compiler\Symtab.py", line 255, in __repr__
    return "%s(<%x>, name=%s, type=%s)" % (type(self).__name__, id(self), self.name, self.type)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  File "c:\users\joshua\dev\cython\Cython\Compiler\PyrexTypes.py", line 299, in __str__
    return self.declaration_code("", for_display = 1).strip()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\users\joshua\dev\cython\Cython\Compiler\PyrexTypes.py", line 3297, in declaration_code
    arg_decl_list.append(self.op_arg_struct.declaration_code(Naming.optional_args_cname))
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'declaration_code'

As the traceback seems to show, the op_arg_struct attribute of the CFuncType node behind the declaration is left as None. I was not able to trace out where it should have been set.

The method signature in question is

    @cython.nonecheck(False)
    @cython.cdivision(True)
    @cython.boundscheck(False)
    cpdef size_t _discover_peaks(self, np.ndarray[cython.floating, ndim=1, mode='c'] mz_array, np.ndarray[cython.floating, ndim=1, mode='c'] intensity_array,
                                 double start_mz, double stop_mz, Py_ssize_t start_index=0, Py_ssize_t stop_index=0) noexcept:
        """Carries out the peak picking process on `mz_array` and `intensity_array`. All
        peaks picked are appended to :attr:`peak_data`.

If the default arguments are removed, the code compiles without issue.

Code to reproduce the behaviour:

No response

Expected behaviour

No response

OS

Windows, Linux

Python version

3.11.5

Cython version

<= 3.0.6

Additional context

The project compiles without issue on Python 3.8-3.10 without issue. The full project is https://github.com/mobiusklein/ms_peak_picker, with the tag for the last released version with the defect https://github.com/mobiusklein/ms_peak_picker/releases/tag/v0.1.43

I was unable reduce this to a minimal example as it seemed to interact with Cython.Compiler.FusedNode the env context variable passed through much of Cython which holds state beyond just the class.

@mobiusklein mobiusklein changed the title [BUG] Compiler crashes while analyzing fused method with optional arguments only on Python 3.11 Dec 5, 2023
@da-woods
Copy link
Contributor

da-woods commented Dec 5, 2023

This actually turns out to be a duplicate of something we thought was a PyPy specific bug: #5588 (comment). I'll fix it properly now

da-woods added a commit to da-woods/cython that referenced this issue Dec 5, 2023
In some Python versions, generating the error message when .index
fails to find an index leads to a compiler crash.

Fix this by not relying on type being fully set up while generating
__str__.

Fixes cython#5894 and cython#5588
@da-woods
Copy link
Contributor

da-woods commented Dec 5, 2023

I've reported it as a bug to CPython, because they have changed the behaviour when generating certain exceptions. I plan to fix it here anyway, whatever they decide to do in CPython.

da-woods added a commit that referenced this issue Dec 8, 2023
* Fix an issue trying list.index indexing in FusedNode

In some Python versions, generating the error message when .index
fails to find an index leads to a compiler crash.

Fix this by not relying on type being fully set up while generating
__str__.

Fixes #5894 and #5588

* Check "in" before indexing
da-woods added a commit that referenced this issue Dec 8, 2023
* Fix an issue trying list.index indexing in FusedNode

In some Python versions, generating the error message when .index
fails to find an index leads to a compiler crash.

Fix this by not relying on type being fully set up while generating
__str__.

Fixes #5894 and #5588

* Check "in" before indexing
da-woods added a commit that referenced this issue Dec 8, 2023
* Fix an issue trying list.index indexing in FusedNode

In some Python versions, generating the error message when .index
fails to find an index leads to a compiler crash.

Fix this by not relying on type being fully set up while generating
__str__.

Fixes #5894 and #5588

* Check "in" before indexing
@da-woods da-woods added this to the 0.29.37 milestone Dec 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants