Mysterious CI crash

[scoder]

Seems to occur randomly in Py3.9, e.g.
https://github.com/cython/cython/runs/4569558060?check_suite_focus=true
https://github.com/cython/cython/runs/4571038859?check_suite_focus=true

Error compiling Cython file:
------------------------------------------------------------
...
        self.reductions = set()

        self.in_inplace_assignment = False
        self.env_stack = []
        self.env = node.scope
        self.stack = []
                    ^
------------------------------------------------------------

Cython/Compiler/FlowControl.py:687:21: Compiler crash in AnalyseExpressionsTransform

ModuleNode.body = StatListNode(FlowControl.py:4:0)
StatListNode.stats[31] = StatListNode(FlowControl.py:674:0)
StatListNode.stats[0] = CClassDefNode(FlowControl.py:674:0,
    class_name = 'ControlFlowAnalysis',
    punycode_class_name = 'ControlFlowAnalysis',
    visibility = 'private')
CClassDefNode.body = StatListNode(FlowControl.py:676:4)
StatListNode.stats[0] = DefNode(FlowControl.py:676:4,
    is_cyfunction = True,
    modifiers = [...]/0,
    name = 'visit_ModuleNode',
    np_args_idx = [...]/0,
    num_required_args = 2,
    outer_attrs = [...]/2,
    py_wrapper_required = True,
    reqd_kw_flags_cname = '0',
    used = True)
File 'Nodes.py', line 396, in analyse_expressions: StatListNode(FlowControl.py:677:8,
    is_terminator = True)
File 'Nodes.py', line 5725, in analyse_expressions: SingleAssignmentNode(FlowControl.py:687:21)
File 'Nodes.py', line 5845, in analyse_types: SingleAssignmentNode(FlowControl.py:687:21)
File 'ExprNodes.py', line 8227, in analyse_types: ListNode(FlowControl.py:687:21,
    constant_result = [...]/0,
    is_sequence_constructor = 1,
    is_temp = 1,
    obj_conversion_errors = [...]/0,
    original_args = [...]/0,
    result_is_used = True,
    use_managed_ref = True)

Compiler crash traceback from this point on:
  File "/home/runner/work/cython/cython/Cython/Compiler/ExprNodes.py", line 8227, in analyse_types
    node = SequenceNode.analyse_types(self, env)
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/home/runner/work/cython/cython/Cython/Compiler/Errors.py", line 260, in local_errors
    release_errors(ignore=ignore)
  File "/home/runner/work/cython/cython/Cython/Compiler/Errors.py", line 241, in release_errors
    held_errors = error_stack.pop()
IndexError: pop from empty list

[da-woods]

Given where it's happening, my eyes are drawn to the lines:

cython/Cython/Compiler/Errors.py

Lines 263 to 264 in ba37c35

	# this module needs a redesign to support parallel cythonisation, but
	# for now, the following works at least in sequential compiler runs

It certainly seems like a plausible cause of this issue

[scoder]

https://github.com/cython/cython/blob/ba37c35ca4da7edd099ffa6832e23764b0bf9bd9/Cython/Compiler/Errors.py#L263-L264 It certainly seems like a plausible cause of this issue

Not ruling it out completely, but IIRC we don't use threads but processes to parallelise.

[da-woods]

The thing that makes me think it might be: it looks like distutils is doing something in parallel itself in the traceback:

 File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/distutils/command/build_ext.py", line 447, in build_extensions
    self._build_extensions_parallel()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/distutils/command/build_ext.py", line 469, in _build_extensions_parallel
    fut.result()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/concurrent/futures/_base.py", line 438, in result
    return self.__get_result()
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/concurrent/futures/_base.py", line 390, in __get_result
    raise self._exception
  File "/opt/hostedtoolcache/Python/3.9.9/x64/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)

and looking at distutils, it seems to use threading internally: https://github.com/python/cpython/blob/f4c03484da59049eb62a9bf7777b963e2267d187/Lib/distutils/command/build_ext.py#L451-L458 (link goes to Python 3.9).

I'm not absolutely convinced - basically the same distutils code is in Python 3.8 and we don't seem to have the problem there.

[da-woods]

If I modify Errors.py to check for the number of running threads and call python3.9 setup.py build_ext --inplace -j4 --force

def release_errors(ignore=False):
    import threading
    if threading.active_count() > 1:
        raise RuntimeError("Oh no!", threading.active_count())
    held_errors = error_stack.pop()
    if not ignore:
        for err in held_errors:
            report_error(err)

I get a bunch of exceptions claiming between 2 and 5 threads. I don't know whether those threads are trying to run Cython in parallel, but I think distutils might be adding threading in addition to Cython using multiprocessing.

[da-woods]

Final comment on this for the moment: if I edit Cython/Compiler/Main.py to put the whole pipeline in a local_errors() block

with Errors.local_errors():
        err, enddata = Pipeline.run_pipeline(pipeline, source)

I can reproduce the crash predictably locally with python3.9 setup.py build_ext --inplace --force -j2. Without -j2 doesn't seem to crash. (I haven't tried other Python versions)