Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"AttributeError: cython_sources" with Cython 3.0.0a10 #601

Open
The-Compiler opened this issue Jan 12, 2022 · 104 comments
Open

"AttributeError: cython_sources" with Cython 3.0.0a10 #601

The-Compiler opened this issue Jan 12, 2022 · 104 comments
Assignees

Comments

@The-Compiler
Copy link
Contributor

The-Compiler commented Jan 12, 2022

When attempting to install PyYAML from sources with Cython 3.0.0a10, e.g. like so:

pip install --pre -v git+https://github.com/yaml/pyyaml.git

the install fails with:

Using pip 21.3.1 from /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip (python 3.10)
Collecting git+https://github.com/yaml/pyyaml.git
  Cloning https://github.com/yaml/pyyaml.git to /tmp/pip-req-build-rdv2sw86
  Running command git version
  git version 2.34.1
  Running command git clone --filter=blob:none -q https://github.com/yaml/pyyaml.git /tmp/pip-req-build-rdv2sw86
  Running command git rev-parse HEAD
  8cdff2c80573b8be8e8ad28929264a913a63aa33
  Resolved https://github.com/yaml/pyyaml.git to commit 8cdff2c80573b8be8e8ad28929264a913a63aa33
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /tmp/pip-standalone-pip-7sp3xayz/__env_pip__.zip/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-az2k859x/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple --pre -- setuptools wheel Cython
  Collecting setuptools
    Using cached setuptools-60.5.0-py3-none-any.whl (958 kB)
  Collecting wheel
    Using cached wheel-0.37.1-py2.py3-none-any.whl (35 kB)
  Collecting Cython
    Using cached Cython-3.0.0a10-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl (2.2 MB)
  Installing collected packages: wheel, setuptools, Cython
  Successfully installed Cython-3.0.0a10 setuptools-60.5.0 wheel-0.37.1
  Installing build dependencies ... done
  Running command /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l
  running egg_info
  creating lib/PyYAML.egg-info
  writing manifest file 'lib/PyYAML.egg-info/SOURCES.txt'
  Traceback (most recent call last):
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
      main()
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 130, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 162, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 143, in _get_build_requires
      self.run_setup()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 158, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 288, in <module>
      setup(
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 155, in setup
      return distutils.core.setup(**attrs)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 148, in setup
      return run_commands(dist)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 163, in run_commands
      dist.run_commands()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 967, in run_commands
      self.run_command(cmd)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 986, in run_command
      cmd_obj.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 299, in run
      self.find_sources()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 306, in find_sources
      mm.run()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 541, in run
      self.add_defaults()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 578, in add_defaults
      sdist.add_defaults(self)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 228, in add_defaults
      self._add_defaults_ext()
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 312, in _add_defaults_ext
      self.filelist.extend(build_ext.get_source_files())
    File "setup.py", line 204, in get_source_files
      self.cython_sources(ext.sources, ext)
    File "/tmp/pip-build-env-az2k859x/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 103, in __getattr__
      raise AttributeError(attr)
  AttributeError: cython_sources
  Getting requirements to build wheel ... error
WARNING: Discarding git+https://github.com/yaml/pyyaml.git. Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.
ERROR: Command errored out with exit status 1: /home/florian/tmp/toxtest/.venv/bin/python3 /home/florian/tmp/toxtest/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpqirb9t5l Check the logs for full command output.

I'm not sure if this is an intended change in Cython 3, or a bug - but given that "pyyaml setup is still relying on a bunch of ancient custom distutils bits" (#584) I thought I'd report it here first. Please let me know if you think I should rather report it there.

The-Compiler added a commit to qutebrowser/qutebrowser that referenced this issue Jan 12, 2022
@nitzmahone
Copy link
Member

It's definitely a regression between Cython 3.0.0a9 and a10; every previous 3.0.0 alpha has been able to build the pyyaml extension fine. This PR (included in a10) swapped out the base class on Cython's custom build_ext distutils command (which pyyaml further subclasses to customize some stuff). I haven't dug down all the way yet to see if it's something we should try to address in pyyaml's customization or if it's just a bug in the new base class.

Thanks for the report- regardless, we'll want to get this sorted one way or another, so I'll figure out where the problem is and work with the Cython/setuptools folks if necessary to get it working.

@tomplex
Copy link

tomplex commented Jan 15, 2022

I encountered this error when attempting to lock my dependencies with pipenv; I was able to fix it in my case by setting

[pipenv]
allow_prereleases = false

just figured I'd share this in case it helps someone else.

@nitzmahone
Copy link
Member

allow_prereleases = false

That'll only work until Cython 3.0 releases, unless we're able to get a change made over there (which it sounds like they're amenable to, if it's not too gnarly)

@RobertoPrevato
Copy link

RobertoPrevato commented Feb 7, 2022

I don't know if this can be of interest for anyone reading here, but I had the same issue in a GitHub Workflow, only happening on Python 3.10. The failed build can be seen here: https://github.com/Neoteroi/BlackSheep/runs/5098546591?check_suite_focus=true

In my case, PyYAML is a dependency and I was testing another wheel with this command:

pip install --pre blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

In my case, removing the --pre option fixed the problem.

pip install blacksheep -f "file:///${GITHUB_WORKSPACE}/dist"

@nitzmahone
Copy link
Member

Yep, unfortunately --pre seems to affect all deps and build deps as well under pip.

twigleingrid pushed a commit to twigleingrid/qutebrowser that referenced this issue May 13, 2022
crides pushed a commit to crides/qutebrowser that referenced this issue Sep 19, 2022
dairiki added a commit to dairiki/marshmallow_dataclass that referenced this issue Nov 4, 2022
Currently PyYAML fails to build with Cython 3.0.0a, thus
it fails to install from sdist using `pip install --pre`.

See yaml/pyyaml#601
dairiki added a commit to dairiki/marshmallow_dataclass that referenced this issue Nov 4, 2022
Currently PyYAML fails to build with Cython 3.0.0a, thus
it fails to install from sdist using `pip install --pre`.

See yaml/pyyaml#601
lovasoa pushed a commit to lovasoa/marshmallow_dataclass that referenced this issue Nov 5, 2022
…ons (#220)

* Test under Python 3.11

* Do not set PYTHONWARNINGS=error during CI pytest

That catches, e.g., deprecation warnings from any package (beyond our
control).

Note that we do configure a pytest warning filter in pyproject.toml
which will generate errors for any warnings generated from within our
code.

* CI: update to actions/checkout@v3, actions/setup-python@v4, actions/cache@v3

* Hack installation of PyYAML on pypy

Currently PyYAML fails to build with Cython 3.0.0a, thus
it fails to install from sdist using `pip install --pre`.

See yaml/pyyaml#601

* Update pre-commit plugin versions to latest
@mgorny
Copy link

mgorny commented Mar 3, 2023

Ping. Cython 3.0.0 has already reached beta releases, so we need some solution sooner than later.

@nitzmahone
Copy link
Member

Cool- we've been poking at doing another release, and since we sat on it long enough, setuptools and pip now fully support PEP517/518 config passthru from the CLI, which should make the replacement of the dynamic bits a whole lot easier.

@jmakov
Copy link

jmakov commented Mar 15, 2023

Stumbled upon this bug while trying to install Pycaret:

Collecting PyYAML~=5.1                                                                                                                   
#0 30.24   Downloading PyYAML-5.4.1.tar.gz (175 kB)                                                                                               #0 30.24      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 175.1/175.1 kB 34.3 MB/s eta 0:00:00
#0 30.32   Installing build dependencies: started                                                                                                 
#0 33.08   Installing build dependencies: finished with status 'done'                                                                             
#0 33.08   Getting requirements to build wheel: started                  
#0 33.19   Getting requirements to build wheel: finished with status 'error'                                                                      #0 33.20   error: subprocess-exited-with-error                                                                                                    
#0 33.20                                                                                                                                          
#0 33.20   × Getting requirements to build wheel did not run successfully.
#0 33.20   │ exit code: 1                                                                                                                         
#0 33.20   ╰─> [50 lines of output]                                      
#0 33.20       /tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:516: SetuptoolsDeprecationWarning: T
he license_file parameter is deprecated, use license_files instead.
#0 33.20         warnings.warn(msg, warning_class)                                                                                                
#0 33.20       running egg_info                                          
#0 33.20       writing lib3/PyYAML.egg-info/PKG-INFO                                                                                              
#0 33.20       writing dependency_links to lib3/PyYAML.egg-info/dependency_links.txt                                                              
#0 33.20       writing top-level names to lib3/PyYAML.egg-info/top_level.txt                                                                      
#0 33.20       Traceback (most recent call last):
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <
module>
#0 33.20           main()
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in m
ain
#0 33.20           json_out['return_val'] = hook(**hook_input['kwargs'])
#0 33.20         File "/opt/conda/envs/rapids/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in g
et_requires_for_build_wheel
#0 33.20           return hook(config_settings)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_b
uild_wheel
#0 33.20           return self._get_build_requires(config_settings, requirements=['wheel'])
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 320, in _get_build_require
s
#0 33.20           self.run_setup()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 335, in run_setup
#0 33.20           exec(code, locals())
#0 33.20         File "<string>", line 271, in <module>
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 108, in setup
#0 33.20           return distutils.core.setup(**attrs)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
#0 33.20           return run_commands(dist)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
#0 33.20           dist.run_commands()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
#0 33.20           self.run_command(cmd)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 1221, in run_command
#0 33.20           super().run_command(command)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
#0 33.20           cmd_obj.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 318, in run
#0 33.20           self.find_sources()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 326, in find_sources
#0 33.20           mm.run()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 570, in run
#0 33.20           self.add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/egg_info.py", line 608, in add_defaults
#0 33.20           sdist.add_defaults(self)
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/command/sdist.py", line 106, in add_defaults
#0 33.20           super().add_defaults()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 251, in add_defaults
#0 33.20           self._add_defaults_ext()
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/sdist.py", line 336, in _add_defaults_ext
#0 33.20           self.filelist.extend(build_ext.get_source_files())
#0 33.20         File "<string>", line 201, in get_source_files
#0 33.20         File "/tmp/pip-build-env-_n6pb0oj/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 107, in __getattr__
#0 33.20           raise AttributeError(attr)
#0 33.20       AttributeError: cython_sources
#0 33.20       [end of output]

@nitzmahone
Copy link
Member

nitzmahone commented Mar 16, 2023

@jmakov the likely short-term fix will be to pin the Cython PEP518 build deps to pre-3.0 Cython in a 6.x release (see #702), but we generally consider the 5.x release branch dead, so hadn't planned on re-publishing the build change back there. You might want to try and get the Pycaret folks (or whoever's ultimately pinning PyYAML to ~=5.1) to update their deps.

@jmakov
Copy link

jmakov commented Mar 16, 2023

@nitzmahone thank you so much for pointing out the solution! Will link your answer to an issue in their repo.

@benorgera
Copy link

Any update on this?

m-kuhn added a commit to open-vcpkg/python-registry that referenced this issue May 12, 2024
m-kuhn added a commit to open-vcpkg/python-registry that referenced this issue May 12, 2024
m-kuhn added a commit to open-vcpkg/python-registry that referenced this issue May 12, 2024
m-kuhn added a commit to open-vcpkg/python-registry that referenced this issue May 12, 2024
guits pushed a commit to guits/ceph that referenced this issue May 23, 2024
Move to 6.0.1 to overcome yaml/pyyaml#601

Fixes: https://tracker.ceph.com/issues/63591

Signed-off-by: Brad Hubbard <bhubbard@redhat.com>
@RubTalha
Copy link

lsf37 added a commit to seL4/seL4 that referenced this issue Jul 12, 2024
Require cypthon < 3, because >=3 breaks the install for pyyaml at
build/install time for more recent python version.

See also <yaml/pyyaml#601>

Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
lsf37 added a commit to seL4/seL4 that referenced this issue Jul 12, 2024
Require cython < 3, because >=3 breaks the install for pyyaml < 5.4 at
build/install time for more recent python version.

pyyaml 5.4 also breaks, but is not fixed by downgrading cython, so we
also require pyyaml < 5.4.

See also <yaml/pyyaml#601>

Signed-off-by: Gerwin Klein <gerwin.klein@proofcraft.systems>
TeddyAndrieux added a commit to scality/metalk8s that referenced this issue Jul 24, 2024
Molter73 added a commit to stackrox/stackrox that referenced this issue Jul 29, 2024
@mathomp4
Copy link

Query: is this issue now moot with 6.0.2? The 6.0.2 release notes say:

Support for Cython 3.x and Python 3.13.

and in my testing to update the spack package for pyyaml, I did test with cython3 and it seems to work, see spack/spack#45716

@henryiii
Copy link

henryiii commented Aug 13, 2024

This issue has been moot for a year or two. It's only affecting people stuck on old versions. (<5, IIRC) (Though, I don't remember if it used to be fixed by just limiting Cython - that's gone in version 6 if it used to be true before)

gotmax23 added a commit to gotmax23/fedrq-mirror that referenced this issue Sep 20, 2024
pyyaml fails to build from source, and there are no 3.12 wheels on PyPI.

Relates: yaml/pyyaml#601
SirAionTech added a commit to SirAionTech/oca-addons-repo-template that referenced this issue Sep 26, 2024
vergenzt added a commit to vergenzt/runcached that referenced this issue Oct 4, 2024
to work around yaml/pyyaml#601 on install
tushar5526 pushed a commit to tushar5526/lpci-mirror that referenced this issue Oct 15, 2024
... in order to avoid build issues, see
yaml/pyyaml#601.
larsks pushed a commit to larsks/openapi2jsonschema that referenced this issue Dec 11, 2024
Fix `AttributeError: cython_sources`
e.g. ref yaml/pyyaml#601
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: In Progress