Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cythonizing scipy master broken on cython master #3113

Open
tacaswell opened this issue Aug 29, 2019 · 2 comments

Comments

@tacaswell
Copy link
Contributor

commented Aug 29, 2019

Trying to install scipy master with the current cython master (00c1dc9) results in errors that look like:

(syspy) ✔  ~/source/other_source/scipy [master {origin/master}|✚ 1]
jupiter@00:01  ➤  pip install -v .
Created temporary directory: /tmp/pip-ephem-wheel-cache-v8igo31t
Created temporary directory: /tmp/pip-req-tracker-sdh92sme
Created requirements tracker '/tmp/pip-req-tracker-sdh92sme'
Created temporary directory: /tmp/pip-install-znwt6o5y
Processing /home/tcaswell/source/other_source/scipy
  Created temporary directory: /tmp/pip-req-build-p5nr1lu0
  Added file:///home/tcaswell/source/other_source/scipy to build tracker '/tmp/pip-req-tracker-sdh92sme'
    Running setup.py (path:/tmp/pip-req-build-p5nr1lu0/setup.py) egg_info for package from file:///home/tcaswell/source/other_source/scipy
    Running command python setup.py egg_info
    running egg_info
    creating pip-egg-info/scipy.egg-info
    writing pip-egg-info/scipy.egg-info/PKG-INFO
    writing dependency_links to pip-egg-info/scipy.egg-info/dependency_links.txt
    writing top-level names to pip-egg-info/scipy.egg-info/top_level.txt
    writing manifest file 'pip-egg-info/scipy.egg-info/SOURCES.txt'
    reading manifest file 'pip-egg-info/scipy.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'pyproject.toml'
    warning: no files found matching 'cythonize.dat'
    warning: no previously-included files matching '*_subr_*.f' found under directory 'scipy/linalg/src/id_dist/src'
    no previously-included directories found matching 'benchmarks/env'
    no previously-included directories found matching 'benchmarks/results'
    no previously-included directories found matching 'benchmarks/html'
    no previously-included directories found matching 'benchmarks/scipy'
    no previously-included directories found matching 'doc/build'
    no previously-included directories found matching 'doc/source/generated'
    no previously-included directories found matching '*/__pycache__'
    warning: no previously-included files matching '*.pyc' found anywhere in distribution
    warning: no previously-included files matching '*~' found anywhere in distribution
    warning: no previously-included files matching '*.bak' found anywhere in distribution
    warning: no previously-included files matching '*.swp' found anywhere in distribution
    warning: no previously-included files matching '*.pyo' found anywhere in distribution
    writing manifest file 'pip-egg-info/scipy.egg-info/SOURCES.txt'
  Source in /tmp/pip-req-build-p5nr1lu0 has version 1.4.0.dev0+d3fb55c, which satisfies requirement scipy==1.4.0.dev0+d3fb55c from file:///home/tcaswell/source/other_source/scipy
  Removed scipy==1.4.0.dev0+d3fb55c from file:///home/tcaswell/source/other_source/scipy from build tracker '/tmp/pip-req-tracker-sdh92sme'
Installing collected packages: scipy
  Created temporary directory: /tmp/pip-record-d0stn8dj
  Running setup.py install for scipy ...     Running command /tmp/syspy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-p5nr1lu0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-d0stn8dj/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/syspy/include/site/python3.7/scipy

    Note: if you need reliable uninstall behavior, then install
    with pip instead of using `setup.py install`:

      - `pip install .`       (from a git repo or downloaded source
                               release)
      - `pip install scipy`   (last SciPy release on PyPI)


    Cythonizing sources
    Running scipy/special/_generate_pyx.py
    Running scipy/linalg/_generate_pyx.py
    Processing scipy/ndimage/src/_ni_label.pyx
    Processing scipy/ndimage/src/_cytest.pyx
    Processing scipy/signal/_max_len_seq_inner.pyx
    Processing scipy/signal/_upfirdn_apply.pyx
    Processing scipy/signal/_peak_finding_utils.pyx
    Processing scipy/signal/_spectral.pyx
    Processing scipy/optimize/_group_columns.pyx
    Processing scipy/optimize/_bglu_dense.pyx
    Processing scipy/optimize/_lsq/givens_elimination.pyx
    Processing scipy/optimize/_trlib/_trlib.pyx

    Error compiling Cython file:
    ------------------------------------------------------------
    ...
        iti = <PyArrayIterObject *> _iti
        itstruct = <PyArrayIterObject *> _itstruct

        # we only process this many neighbors from the itstruct iterator before
        # reaching the center, where we stop
        num_neighbors = (structure.size / 3) // 2
                                       ^
    ------------------------------------------------------------

    _ni_label.pyx:274:36: Compiler crash in AnalyseExpressionsTransform

    ModuleNode.body = StatListNode(_ni_label.pyx:6:0)
    StatListNode.stats[21] = StatListNode(_ni_label.pyx:204:6)
    StatListNode.stats[0] = CFuncDefNode(_ni_label.pyx:204:6,
        args = [...]/3,
        modifiers = [...]/0,
        outer_attrs = [...]/1,
        overridable = 1,
        visibility = 'private')
    File 'Nodes.py', line 437, in analyse_expressions: StatListNode(_ni_label.pyx:210:4,
        is_terminator = True)
    File 'Nodes.py', line 437, in analyse_expressions: StatListNode(_ni_label.pyx:274:41)
    File 'Nodes.py', line 5319, in analyse_expressions: SingleAssignmentNode(_ni_label.pyx:274:41)
    File 'Nodes.py', line 5439, in analyse_types: SingleAssignmentNode(_ni_label.pyx:274:41)
    File 'ExprNodes.py', line 10992, in analyse_types: DivNode(_ni_label.pyx:274:41,
        infix = True,
        operator = '//',
        result_is_used = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 10994, in analyse_types: DivNode(_ni_label.pyx:274:36,
        ctruedivision = True,
        infix = True,
        operator = '/',
        result_is_used = True,
        truedivision = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 11485, in analyse_operation: DivNode(_ni_label.pyx:274:36,
        ctruedivision = True,
        infix = True,
        operator = '/',
        result_is_used = True,
        truedivision = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 11012, in analyse_operation: DivNode(_ni_label.pyx:274:36,
        ctruedivision = True,
        infix = True,
        operator = '/',
        result_is_used = True,
        truedivision = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 11193, in analyse_c_operation: DivNode(_ni_label.pyx:274:36,
        ctruedivision = True,
        infix = True,
        operator = '/',
        result_is_used = True,
        truedivision = True,
        use_managed_ref = True)
    File 'ExprNodes.py', line 11501, in compute_c_result_type: DivNode(_ni_label.pyx:274:36,
        ctruedivision = True,
        infix = True,
        operator = '/',
        result_is_used = True,
        truedivision = True,
        use_managed_ref = True)

    Compiler crash traceback from this point on:
      File "/tmp/syspy/lib/python3.7/site-packages/Cython/Compiler/ExprNodes.py", line 11501, in compute_c_result_type
        widest_type = PyrexTypes.widest_numeric_type(type1, PyrexTypes.c_double_type)
      File "/tmp/syspy/lib/python3.7/site-packages/Cython/Compiler/PyrexTypes.py", line 4500, in widest_numeric_type
        elif type1.rank < type2.rank:
    AttributeError: 'CFuncType' object has no attribute 'rank'
    Processing scipy/optimize/cython_optimize/_zeros.pyx.in
    Traceback (most recent call last):
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 307, in <module>
        main()
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 303, in main
        find_process_files(root_dir)
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 292, in find_process_files
        for result in pool.imap_unordered(lambda args: process(*args), jobs):
      File "/usr/lib/python3.7/multiprocessing/pool.py", line 748, in next
        raise value
      File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
        result = (True, func(*args, **kwds))
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 292, in <lambda>
        for result in pool.imap_unordered(lambda args: process(*args), jobs):
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 231, in process
        processor_function(fromfile, tofile, cwd=path)
      File "/tmp/pip-req-build-p5nr1lu0/tools/cythonize.py", line 96, in process_pyx
        raise Exception('Cython failed')
    Exception: Cython failed
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-req-build-p5nr1lu0/setup.py", line 540, in <module>
        setup_package()
      File "/tmp/pip-req-build-p5nr1lu0/setup.py", line 524, in setup_package
        generate_cython()
      File "/tmp/pip-req-build-p5nr1lu0/setup.py", line 292, in generate_cython
        raise RuntimeError("Running cythonize failed!")
    RuntimeError: Running cythonize failed!
error
Cleaning up...
  Removing source in /tmp/pip-req-build-p5nr1lu0
Removed build tracker '/tmp/pip-req-tracker-sdh92sme'
Command "/tmp/syspy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-p5nr1lu0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-d0stn8dj/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/syspy/include/site/python3.7/scipy" failed with error code 1 in /tmp/pip-req-build-p5nr1lu0/
Exception information:
Traceback (most recent call last):
  File "/tmp/syspy/lib/python3.7/site-packages/pip/_internal/cli/base_command.py", line 179, in main
    status = self.run(options, args)
  File "/tmp/syspy/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 393, in run
    use_user_site=options.use_user_site,
  File "/tmp/syspy/lib/python3.7/site-packages/pip/_internal/req/__init__.py", line 57, in install_given_reqs
    **kwargs
  File "/tmp/syspy/lib/python3.7/site-packages/pip/_internal/req/req_install.py", line 945, in install
    spinner=spinner,
  File "/tmp/syspy/lib/python3.7/site-packages/pip/_internal/utils/misc.py", line 761, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip._internal.exceptions.InstallationError: Command "/tmp/syspy/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-req-build-p5nr1lu0/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-d0stn8dj/install-record.txt --single-version-externally-managed --compile --install-headers /tmp/syspy/include/site/python3.7/scipy" failed with error code 1 in /tmp/pip-req-build-p5nr1lu0/


This bisects back to 3062852 which is the merge commit in #3095

Bisect log

# bad: [00c1dc96af0e44741dc777f96a2eb5769f572bb3] Utility/CppConvert: fix Cython warning (GH-3108)
# good: [d4610f39a43a658cee1862c899a9696d8a9c9f5f] Prepare release of 0.29.13.
git bisect start 'master' '0.29.13'
# good: [e3252209a60733850557cd22c46b96f1aa8574c4] Remove unused import.
git bisect good e3252209a60733850557cd22c46b96f1aa8574c4
# good: [b330b6854bcfdd02f95ea21ece430d348b665aa4] Call ".tp_repr()" instead of ".tp_str()" when formatting builtin types, since it avoids an indirection in Py3.8.
git bisect good b330b6854bcfdd02f95ea21ece430d348b665aa4
# good: [82a0ed43b31f0ad263256d2725d762ee82cd2179] Include: cpython.pystate: Make PyGILState_STATE definition usable (GH-2997)
git bisect good 82a0ed43b31f0ad263256d2725d762ee82cd2179
# good: [8cf109c0f19ad8c36b8a5368d138912495963387] Fix PY_HEX_VERSION -> PY_VERSION_HEX (GH-3071)
git bisect good 8cf109c0f19ad8c36b8a5368d138912495963387
# good: [2001612cfe4ac4f89c071da41839b2e2d0ac1a41] Update lexicon from latest CPython 3.8.
git bisect good 2001612cfe4ac4f89c071da41839b2e2d0ac1a41
# bad: [5cfa3bd4d68d0b0e7451ca513eec90f64528af18] Use METH_FASTCALL on CPython >= 3.7 (GH-3021)
git bisect bad 5cfa3bd4d68d0b0e7451ca513eec90f64528af18
# good: [4e02338e8c282b30d42564c52c4fba6de69ed984] Add an assertion for the assumption that all identifier start characters are also valid continuation characters, so that we notice it if that ever changes.
git bisect good 4e02338e8c282b30d42564c52c4fba6de69ed984
# good: [3ac72f5ff9388bbd952e8764ecaad230fcd03991] Always enable C line in traceback for tests (GH-3098)
git bisect good 3ac72f5ff9388bbd952e8764ecaad230fcd03991
# bad: [306285235bdc8a17ed291104e96ba7b3ceb54806] Support wider use of @property decorator on CClassDef AttributeNodes (GH-3095)
git bisect bad 306285235bdc8a17ed291104e96ba7b3ceb54806
# good: [2ffd5060f5a008684728dfc0574f57e422fda42d] __Pyx_GetNameInClass() should get raw attribute from __dict__ (GH-3100)
git bisect good 2ffd5060f5a008684728dfc0574f57e422fda42d
# first bad commit: [306285235bdc8a17ed291104e96ba7b3ceb54806] Support wider use of @property decorator on CClassDef AttributeNodes (GH-3095)

bisect script

pip uninstall cython --yes
git clean -xfd
pip install .
retVal=$?
if [ $retVal -ne 0 ]; then
    exit 125
fi
pushd ../scipy
git clean -xfd
python tools/cythonize.py scipy

[edit, sorry for the terrible first version, hit ctl-enter by accident]

@mattip

This comment has been minimized.

Copy link
Contributor

commented Aug 29, 2019

Thanks for the heads-up, sorry for the breakage. I am working on a fix.

The root problem is that structure.size is currently converted to PyLong_FromLong(getattr(structure, 'size') using equivalent C-API calls, when it should become PyArray_SIZE(structure) in pure C, never needing to call the C-API.

Bottom line: when it is fixed, the new code should use fewer C-API calls than the old.

@tacaswell

This comment has been minimized.

Copy link
Contributor Author

commented Aug 29, 2019

@mattip Finding these bugs is why I build master-everything ;)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.