Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TSK: Make NumPy buildable on the nogil branch #26161

Closed
Tracked by #26157
ngoldbaum opened this issue Mar 28, 2024 · 10 comments
Closed
Tracked by #26157

TSK: Make NumPy buildable on the nogil branch #26161

ngoldbaum opened this issue Mar 28, 2024 · 10 comments
Assignees
@ngoldbaum
Labels
17 - Task 39 - no-GIL PRs and issues related to support for free-threaded CPython (a.k.a. no-GIL, PEP 703)
Milestone
2.1.0 release

Comments

@ngoldbaum
Copy link
Member

ngoldbaum commented Mar 28, 2024 •
edited
Loading

Currently the build fails with internal errors in cython (compiled using the latest cython master branch):

FAILED: numpy/random/_sfc64.cpython-313t-darwin.so.p/numpy/random/_sfc64.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/_sfc64.pyx -o numpy/random/_sfc64.cpython-313t-darwin.so.p/numpy/random/_sfc64.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() keywords must be strings
[284/309] Compiling Cython source /Users/goldbaum/Documents/numpy/numpy/random/_mt19937.pyx
FAILED: numpy/random/_mt19937.cpython-313t-darwin.so.p/numpy/random/_mt19937.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/_mt19937.pyx -o numpy/random/_mt19937.cpython-313t-darwin.so.p/numpy/random/_mt19937.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() got an unexpected keyword argument 'code'
[285/309] Compiling Cython source /Users/goldbaum/Documents/numpy/numpy/random/_philox.pyx
FAILED: numpy/random/_philox.cpython-313t-darwin.so.p/numpy/random/_philox.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/_philox.pyx -o numpy/random/_philox.cpython-313t-darwin.so.p/numpy/random/_philox.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() keywords must be strings
[286/309] Compiling Cython source /Users/goldbaum/Documents/numpy/numpy/random/bit_generator.pyx
FAILED: numpy/random/bit_generator.cpython-313t-darwin.so.p/numpy/random/bit_generator.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/bit_generator.pyx -o numpy/random/bit_generator.cpython-313t-darwin.so.p/numpy/random/bit_generator.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() got an unexpected keyword argument 'code'
[287/309] Compiling Cython source /Users/goldbaum/Documents/numpy/numpy/random/_pcg64.pyx
FAILED: numpy/random/_pcg64.cpython-313t-darwin.so.p/numpy/random/_pcg64.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/_pcg64.pyx -o numpy/random/_pcg64.cpython-313t-darwin.so.p/numpy/random/_pcg64.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() keywords must be strings
[288/309] Compiling Cython source numpy/random/_bounded_integers.pyx
FAILED: numpy/random/_bounded_integers.cpython-313t-darwin.so.p/numpy/random/_bounded_integers.pyx.c
cython -M --fast-fail -3 numpy/random/_bounded_integers.pyx -o numpy/random/_bounded_integers.cpython-313t-darwin.so.p/numpy/random/_bounded_integers.pyx.c
[289/309] Compiling Cython source numpy/random/mtrand.pyx
FAILED: numpy/random/mtrand.cpython-313t-darwin.so.p/numpy/random/mtrand.pyx.c
cython -M --fast-fail -3 numpy/random/mtrand.pyx -o numpy/random/mtrand.cpython-313t-darwin.so.p/numpy/random/mtrand.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 342, in generate_function_definitions
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() keywords must be strings
[290/309] Compiling Cython source /Users/goldbaum/Documents/numpy/numpy/random/_common.pyx
FAILED: numpy/random/_common.cpython-313t-darwin.so.p/numpy/random/_common.pyx.c
cython -M --fast-fail -3 /Users/goldbaum/Documents/numpy/numpy/random/_common.pyx -o numpy/random/_common.cpython-313t-darwin.so.p/numpy/random/_common.pyx.c
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/bin/cython", line 8, in <module>
    sys.exit(setuptools_main())
             ~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 740, in setuptools_main
    return main(command_line = 1)
           ~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 768, in main
    result = compile(sources, options)
             ~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 659, in compile
    return compile_multiple(source, options)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 633, in compile_multiple
    result = run_pipeline(source, options,
             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
                          full_module_name=options.module_name,
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                          context=context)
                          ^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Main.py", line 525, in run_pipeline
    err, enddata = Pipeline.run_pipeline(pipeline, source)
                   ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 404, in run_pipeline
    data = run(phase, data)
           ~~~^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 386, in run
    return phase(data)
           ~~~~~^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Pipeline.py", line 50, in generate_pyx_code_stage
    module_node.process_implementation(options, result)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 221, in process_implementation
    self.generate_c_code(env, options, result)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/ModuleNode.py", line 514, in generate_c_code
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 342, in generate_function_definitions
    self.body.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 397, in generate_function_definitions
    stat.generate_function_definitions(env, code)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2321, in generate_function_definitions
    align_error_path_gil_to_success_path()
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-nogil/lib/python3.13/site-packages/Cython/Compiler/Nodes.py", line 2305, in align_error_path_gil_to_success_path
    assure_gil('error', code=code)
    ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
TypeError: FuncDefNode.generate_function_definitions.<locals>.assure_gil() got an unexpected keyword argument 'code'
[291/309] Compiling Cython source numpy/random/_generator.pyx
FAILED: numpy/random/_generator.cpython-313t-darwin.so.p/numpy/random/_generator.pyx.c
cython -M --fast-fail -3 numpy/random/_generator.pyx -o numpy/random/_generator.cpython-313t-darwin.so.p/numpy/random/_generator.pyx.c
ninja: build stopped: subcommand failed.

We can probably hack around this by disabling np.random for nogil python.
@ngoldbaum ngoldbaum mentioned this issue Mar 28, 2024
Open
12 tasks
@ngoldbaum ngoldbaum changed the title Make NumPy buildable on the nogil branch TSK: Make NumPy buildable on the nogil branch Mar 29, 2024
@ngoldbaum
Copy link
Member Author

@lysnikolaou has some WIP changes that lets me build np.random on this branch. It looks like there are some test failures that are specific to a --disable-gil python build:

FAILED build-install/usr/lib/python3.13/site-packages/numpy/_core/tests/test_nditer.py::test_iter_refcount - AssertionError
FAILED build-install/usr/lib/python3.13/site-packages/numpy/_core/tests/test_nditer.py::test_buffered_cast_error_paths_unraisable - subprocess.CalledProcessError: Command '['/Users/goldbaum/.pyenv/versions/3...
FAILED build-install/usr/lib/python3.13/site-packages/numpy/linalg/tests/test_linalg.py::test_sdot_bug_8577 - subprocess.CalledProcessError: Command '['/Users/goldbaum/.pyenv/versions/3...
FAILED build-install/usr/lib/python3.13/site-packages/numpy/random/tests/test_extending.py::test_cython - subprocess.CalledProcessError: Command '['meson', 'setup', '/private/var/fo...
FAILED build-install/usr/lib/python3.13/site-packages/numpy/tests/test_public_api.py::test_import_lazy_import[testing] - subprocess.CalledProcessError: Command '('/Users/goldbaum/.pyenv/versions/3...
FAILED build-install/usr/lib/python3.13/site-packages/numpy/tests/test_reloading.py::test_full_reimport - AssertionError: Non-zero return code: 1
FAILED build-install/usr/lib/python3.13/site-packages/numpy/tests/test_scripts.py::test_pep338 - subprocess.CalledProcessError: Command '['/Users/goldbaum/.pyenv/versions/3...

Reloading will probably be fixed upstream but the others look like issues in numpy that bear a closer look.

Additionally, all the cython and most of the f2py tests are failing due to a meson issue on 3.13 that meson is waiting on an upstream fix for.

I'd like to get these tests fixed and add add 3.13 CI. I see a build matrix including a normal 3.13 build, and two 3.13 test runs with --disable-gil, one with PYTHON_GIL=1, and eventually another with PYTHON_GIL=0.

We might need to xfail or skip the cython or f2py tests for now.

@rgommers
Copy link
Member

rgommers commented Apr 3, 2024

Additionally, all the cython and most of the f2py tests are failing due to a meson issue on 3.13 that meson is waiting on an upstream fix for.

This is the importlib.resources API removal that is about to be reverted I assume? If so, that should be done soon, this was posted about 24 hours ago: https://discuss.python.org/t/deprecating-importlib-resources-legacy-api/11386/41. The PR linked there (python/cpython#116609) is approved, so this should happen soon. I wouldn't bother with xfailing yet, unless it becomes too annoying to wait for that PR.

@lysnikolaou lysnikolaou mentioned this issue Apr 8, 2024
Merged
@ngoldbaum
Copy link
Member Author

python/cpython#116609 is merged, which fixed all the tests that depend on meson. I also just merged #26207 which fixes one of the tests and marked the limited API test as an xfail in #26229.

I'm going to close this for and open followup issues about test failures with the GIL disabled (awesome we've gotten that far already!). I'm also going to set up bleeding edge CPython CI that can be converted into CPython prerelease tests once the first beta is cut.

@rgommers
Copy link
Member

rgommers commented Apr 9, 2024

Awesome, nice work!

I'm also going to set up bleeding edge CPython CI that can be converted into CPython prerelease tests once the first beta is cut.

On a fork for now rather than on this repo? This is probably going to be failing regularly for the next weeks, with changes in CPython and Cython.

@rgommers rgommers added this to the 2.1.0 release milestone Apr 9, 2024
@seberg
Copy link
Member

seberg commented Apr 9, 2024

Wasn't there a way to say "failure is OK" so that it shows up as orange and not as a failure? that would seem fine also.

@rgommers
Copy link
Member

rgommers commented Apr 9, 2024

If there is, I'm not familiar with it. My understanding is that there's a feature to say which checks are required for merging, which we don't use. If you do use it, failing non-required jobs still look red but won't block merging. For us, I think any failing job will simply be red.

@lysnikolaou
Copy link
Contributor

lysnikolaou commented Apr 9, 2024 •
edited
Loading

Wasn't there a way to say "failure is OK" so that it shows up as orange and not as a failure? that would seem fine also.

There's no official feature for this, but there's a workaround explained on the feature request ticket.

@rgommers
Copy link
Member

rgommers commented Apr 9, 2024

Oh wait, I read the proposal as "a new CI job for the no-GIL build", but on re-reading I think you meant one for the default CPython build. That seems fine by now; with beta 1 arriving in a couple of weeks I hope there won't be more than 1 or 2 more regressions that affect numpy anymore.

@seberg
Copy link
Member

seberg commented Apr 9, 2024

Yeah, I guess allow-failure still looks too much like a failure, seems I probably had thought of a Travis CI feature. I would be OK with trying if it fails too often, but maybe a branch is best then.

@ngoldbaum
Copy link
Member Author

Oh wait, I read the proposal as "a new CI job for the no-GIL build", but on re-reading I think you meant one for the default CPython build. That seems fine by now; with beta 1 arriving in a couple of weeks I hope there won't be more than 1 or 2 more regressions that affect numpy anymore.

That's right, just vanilla 3.13 CI. Nogil CI will have to wait until more stuff is stabilized.

@rgommers rgommers added the 39 - no-GIL PRs and issues related to support for free-threaded CPython (a.k.a. no-GIL, PEP 703) label Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ngoldbaum ngoldbaum

Labels
17 - Task 39 - no-GIL PRs and issues related to support for free-threaded CPython (a.k.a. no-GIL, PEP 703)
Projects
None yet
Milestone
2.1.0 release
Development

No branches or pull requests
4 participants
@seberg @rgommers @ngoldbaum @lysnikolaou