Pin Cython build constraint to < 3.0 #702

nitzmahone · 2023-03-03T23:24:35Z

Temporary workaround for #601.

This is just a stopgap measure to ensure the libyaml extension build continues to function whenever Cython 3.0 is released (date TBD). We'll want to prioritize the extension build rewrite to ensure that PyYAML can support Python > 3.11, as I'd be amazed if they backport things to the Cython 0.x branch once 3.0 has released.

We should discuss if we want to just release this as 6.0.post1 or what, since there are no code changes to the actual library, just build/packaging metadata...

keith · 2023-07-17T17:15:02Z

Merging and releasing this would fix #601

rahul-theorem · 2023-07-17T18:23:21Z

Would it be possible to backport this fix to 5.x & release a hotfix as well? We're currently blocked on upgrading to pyyaml v6 (which we're hoping to resolve this week), but it'd be helpful for a few reasons to be able to pick this up in 5.4.2

hssyoo · 2023-07-17T18:35:09Z

Hi @nitzmahone, the AWS CLI is also currently impacted by this (aws/aws-cli#8036). If we can get this PR merged and released as 6.0.1, it will resolve the issues faced by the AWS CLI and other impacted packages.

DeanWay · 2023-07-17T19:51:23Z

This is just a stopgap measure to ensure the libyaml extension build continues to function whenever Cython 3.0 is released (date TBD).

Cython just released 3.0 so everything that has any kind of transitive dependency on PyYAML will now fail to install. This is now a necessary change.

mharding-hpe · 2023-07-17T19:54:58Z

This is causing a number of products in our organization to be unbuildable. Please consider this another plea for a quick merge and release of this now critical PR.

addyess

This is just what we all need from pyyaml

enmatt · 2023-07-17T20:11:52Z

We've also been wrestling with this dependency issue this morning during our builds.

cgoessling

This would be urgently appreciated.

addyess · 2023-07-17T20:31:36Z

@ingydotnet, @xitology, and @perlpunk thanks for your quick review on this

nitzmahone · 2023-07-17T21:31:30Z

Yep, I'll be doing a 6.0.1 release imminently to cover this (grumblesmurf that I somehow missed the Cython 3.0 release was actually happening since they've been in beta for so long).

No commitment to a 5.x backport/release, but if I can easily resurrect all the necessary wheel builds for it, I might consider it.

antoniogomezalvarado · 2023-07-17T21:46:58Z

For anyone that needs a quick fix for this do the following instead of installing PyYAML directly

git clone https://github.com/yaml/pyyaml.git && \
cd pyyaml && \
git checkout release/5.4.1 && \
sed -i.bak 's/Cython/Cython<3.0/g' pyproject.toml && \
python setup.py sdist && \
pip install --pre dist/PyYAML-5.4.1.tar.gz

* Bump pyyaml to 6.0.1 Temporary workaround for wheel build issue on Cython versions >= 3 (yaml/pyyaml#702) * Pin black to prevent breakage on updates (cherry picked from commit 383dfbd)

Temporary workaround for wheel build issue on Cython versions >= 3 (yaml/pyyaml#702) (cherry picked from commit 383dfbd) * Remove twine dependency to support older PyPy

The 6.0 release contains a temporary workaround to a regression in Cython 3.10.0a10 that throws an attribute error. See more details upstream at yaml/pyyaml#601 and yaml/pyyaml#702.

* keeps libyaml extension build functional once Cython 3.0 releases; stopgap measure until we can rewrite the extension build to eliminate all the ancient deprecated distutils magic (cherry picked from commit ae08bdc)

The new release includes yaml/pyyaml#702

…bot#7738) Under `pip-tools` `7.0.0`, the default flipped from `--resolver=legacy` to `--resolver=backtracking`. As a result, the error message changed. Furthermore, the new error message is hidden behind an exception thrown by `cython>=3.0.0`/`pyyaml<6.0.1` conflict (details in yaml/pyyaml#702). I already fixed a similar error here: dependabot@0a8e0b4 So the first thing I did was bump `pyyaml` to `6.0.1`. While not strictly necessary, it'll make later debugging easier when / if someone is trying to repro this error under the new `backtracking` resolver. Secondly, I specified `--resolver=legacy` to preserve the old behavior even after we upgrade `pip-tools` (dependabot#7711). This is acceptable for the purposes of this test because this isn't testing the resolver behavior, it's testing that we properly convert the `pip-tools` native error to a well-known `Dependabot::DependencyFileNotResolvable` error. I did take a peek at the `pip-tools` internals to see if this error messsage was specific solely to the `legacy` resolver, but the logic looks pretty generic... ie, this likely could be raised under the new resolver: https://github.com/jazzband/pip-tools/blob/9e4aeff8f3b5518586091700bec9bbb570154a38/piptools/exceptions.py#L53 This is admittedly a short-term solution because the `legacy` resolver is deprecated and will be removed at some point in the future... so at that point we'll need to find another way to reproduce the error. But enough other things could change upstream in `pip`/`pip-tools` (such as they might switch the error or remove it when they remove the legacy resolver) so we can worry about that when we get there rather than prematurely optimizing. Technically I suppose we could mock the response from calling `pip-compile`, but I hate mocking what I don't own... much prefer to simply coerce it to return the expected error message. For future reference, here's the error message under `--resolver=legacy`: ```shell ```shell [dependabot-core-dev] ~/python/spec/fixtures/pip_compile_files $ pyenv exec pip-compile --build-isolation --output-file=../requirements/incompatible_versions.txt --no-emit-index-url -P pyyaml==6.0.1 --resolver=legacy incompatible_versions.in WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`. Could not find a version that matches pyyaml<5.4,==6.0.1,>=3.10 Tried: 3.10, 3.10, 3.11, 3.11, 3.12, 3.12, 3.13, 5.1, 5.1.1, 5.1.2, 5.2, 5.3, 5.3.1, 5.4, 5.4.1, 6.0, 6.0, 6.0.1, 6.0.1 Skipped pre-versions: 3.13b1, 3.13rc1, 4.2b1, 4.2b2, 4.2b4, 5.1b1, 5.1b3, 5.1b5, 5.2b1, 5.3b1, 5.4b1, 5.4b2, 6.0b1 There are incompatible versions in the resolved dependencies: pyyaml==6.0.1 PyYAML<5.4,>=3.10 (from awscli==1.18.198->-r incompatible_versions.in (line 3)) pyyaml (from jinja2-cli[yaml]==0.7.0->-r incompatible_versions.in (line 2)) ``` And here's the error message under `--resolver=backtracking`: ```shell [dependabot-core-dev] ~/python/spec/fixtures/pip_compile_files $ pyenv exec pip-compile --build-isolation --output-file=../requirements/incompatible_versions.txt --no-emit-index-url -P pyyaml==6.0.1 --resolver=backtracking incompatible_versions.in ERROR: Cannot install -r incompatible_versions.in (line 3) and jinja2-cli[yaml]==0.7.0 because these package versions have conflicting dependencies. Traceback (most recent call last): File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 316, in _backjump name, candidate = broken_state.mapping.popitem() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ KeyError: 'dictionary is empty' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 92, in resolve result = self._result = resolver.resolve( ^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 546, in resolve state = resolution.resolve(requirements, max_rounds=max_rounds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 434, in resolve success = self._backjump(causes) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_vendor/resolvelib/resolvers.py", line 318, in _backjump raise ResolutionImpossible(causes) pip._vendor.resolvelib.resolvers.ResolutionImpossible: [RequirementInformation(requirement=SpecifierRequirement('pyyaml; extra == "yaml"'), parent=ExtrasCandidate(base=LinkCandidate('https://files.pythonhosted.org/packages/83/52/c1bb249f49b204c14bf603e1b1a6dc2db8c3b631c4fe00a2872360085cd6/jinja2_cli-0.7.0-py2.py3-none-any.whl (from https://pypi.org/simple/jinja2-cli/)'), extras=frozenset({'yaml'}))), RequirementInformation(requirement=SpecifierRequirement('PyYAML<5.4,>=3.10; python_version != "3.4"'), parent=LinkCandidate('https://files.pythonhosted.org/packages/4e/68/35522bf3b5ef9186f3cd7cfdc00fe708c019fada077a21009ff78757b0f9/awscli-1.18.198-py2.py3-none-any.whl (from https://pypi.org/simple/awscli/)'))] The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/local/.pyenv/versions/3.11.4/bin/pip-compile", line 8, in <module> sys.exit(cli()) ^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/click/core.py", line 1157, in __call__ return self.main(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, **ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/click/core.py", line 783, in invoke return __callback(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func return f(get_current_context(), *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/piptools/scripts/compile.py", line 657, in cli results = resolver.resolve(max_rounds=max_rounds) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/piptools/resolver.py", line 604, in resolve is_resolved = self._do_resolve( ^^^^^^^^^^^^^^^^^ File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/piptools/resolver.py", line 636, in _do_resolve resolver.resolve( File "/usr/local/.pyenv/versions/3.11.4/lib/python3.11/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 101, in resolve raise error from e pip._internal.exceptions.DistributionNotFound: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts ``` Furthermore, the `cython>=3.0.0`/`pyyaml<=6.0.1` conflict only appears when I do `--resolver=backtracking`, otherwise under the legacy resolver I don't see that error, even on older version of pyyaml: ```shell [dependabot-core-dev] ~/python/spec/fixtures/pip_compile_files $ pyenv exec pip-compile --build-isolation --output-file=../requirements/incompatible_versions.txt --no-emit-index-url -P pyyaml==5.4 --resolver=legacy incompatible_versions.in WARNING: the legacy dependency resolver is deprecated and will be removed in future versions of pip-tools. Using legacy resolver. Consider using backtracking resolver with `--resolver=backtracking`. Could not find a version that matches pyyaml<5.4,==5.4,>=3.10 Tried: 3.10, 3.10, 3.11, 3.11, 3.12, 3.12, 3.13, 5.1, 5.1.1, 5.1.2, 5.2, 5.3, 5.3.1, 5.4, 5.4.1, 6.0, 6.0, 6.0.1, 6.0.1 Skipped pre-versions: 3.13b1, 3.13rc1, 4.2b1, 4.2b2, 4.2b4, 5.1b1, 5.1b3, 5.1b5, 5.2b1, 5.3b1, 5.4b1, 5.4b2, 6.0b1 There are incompatible versions in the resolved dependencies: pyyaml==5.4 PyYAML<5.4,>=3.10 (from awscli==1.18.198->-r incompatible_versions.in (line 3)) pyyaml (from jinja2-cli[yaml]==0.7.0->-r incompatible_versions.in (line 2)) ```

To fix an installation error caused by pyyaml cython requirement: AttributeError: cython_sources https://github.com/yaml/pyyaml/blob/c42fa3bff1eabdb64763bb1526d9ea1ccb708479/CHANGES#L9 yaml/pyyaml#702

Fixes an issue I am seeing on Fedora 39. For more details about the upstream fix see yaml/pyyaml#702 Signed-off-by: Christian Gmeiner <cgmeiner@igalia.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/25878>

Refer: yaml/pyyaml#702

- pyyaml <= 6.0.0 breaks when cython >= 3.0.0 - release 6.0.1 sets a cython build-time constraint - [PR](yaml/pyyaml#702) - fix will not be [backported to 5.4 or lower](yaml/pyyaml#751) - jinja major version constraint added

* Remove install_dependencies from buld.sh call * remove sudo from folly build * Align cpp wf file with cpu wf file * Make yaml statically linked dependency * move cpp binaries to location based on ts.__file__ * Add FAQ to cpp readme * Update readme to expect installed TS * Fix libtorch install on mac * Add cpp build faqs * Remove llama.cpp as submodule and add through fetch_content * Disable metal in llamacpp example * Fix dangling pointer error in cpp worker * Remove kineto from mac build * Add llvm as mac cpp dependencies (req for clang-tidy) * Enable fPIC for llama.cpp example * Add sudo to clang-tidy link creation * Add undefined symbol faq * fix llv typo * Add install from source to cpp_ci * Correct install from source to cpp_ci * bump up pyyaml version to avoid cython 3 issue yaml/pyyaml#702 * Move cpp ci to M1 mac * Run workflow on self hosted runner * Disable mac ci for cpp * Fix workflow syntax * Run on cpp-ci * Remove sudo * Add apt update for act docker * print library_path in print_env_info.py * print end in cpp ci workflow * Run on github runner * Add upterm session to workflow * Move post mortem upterm session before build * Remove first upterm session * ci debugging * Observe disk space * move _build to /mnt on github runner * fix permission denied * use mount instead of ln * Adjust llamacpp api * Reactivate set -e * Remove note on env variable in cpp readme * Fix linting issue in print_env_info.py * Cleanup ci-cpu-cpp.yml * quieten wget * Add build clean section in cpp/readme * Readjust to llama.cpp api

This updates 2 python packages to versions compatible with python3.12, fixing the following errors when running `./scripts/examples/gn_build_example.sh examples/chip-tool BUILDFOLDER`: 1. `pyyaml` 6.0 build error: `AttributeError: cython_sources`, fixed [here](yaml/pyyaml#702) 2. `construct` import error: ``` File "/tmp/pip-install-ifgk4tul/construct_6d60304f85e249759f6c184ea89f1317/construct/core.py", line 3, in <module> import struct, io, binascii, itertools, collections, pickle, sys, os, tempfile, hashlib, importlib, imp ModuleNotFoundError: No module named 'imp' ``` this was fixed [here](construct/construct@91cc0c6), in version `v2.10.57` of `construct`. I'm just updating to latest though. Now running `./scripts/examples/gn_build_example.sh examples/chip-tool BUILDFOLDER` succeeds when the host python is python3.12.

nitzmahone changed the base branch from release/6.0 to master March 3, 2023 23:27

nitzmahone mentioned this pull request Mar 16, 2023

"AttributeError: cython_sources" with Cython 3.0.0a10 #601

Open

mcdonnnj mentioned this pull request Jul 17, 2023

Build error with setuptools 0.68.0 #723

Closed

kyleknap mentioned this pull request Jul 17, 2023

Add support for PyYAML 6.0 aws/aws-cli#8037

Merged

GoelJatin approved these changes Jul 17, 2023

View reviewed changes

rmorshea mentioned this pull request Jul 17, 2023

Restrict Cython to <3 for now pyFFTW/pyFFTW#359

Open

mharding-hpe approved these changes Jul 17, 2023

View reviewed changes

addyess approved these changes Jul 17, 2023

View reviewed changes

cgoessling approved these changes Jul 17, 2023

View reviewed changes

lalanametala approved these changes Jul 17, 2023

View reviewed changes

romga approved these changes Jul 17, 2023

View reviewed changes

scottwthompson approved these changes Jul 17, 2023

View reviewed changes

jgigliotti approved these changes Jul 17, 2023

View reviewed changes

shmsr mentioned this pull request Jul 17, 2023

Pin PyYAML version to 5.3.1 to avoid CI errors temporarily elastic/beats#36091

Merged

6 tasks

mharding-hpe mentioned this pull request Jul 17, 2023

CASMCMS-8713: Temporarily modify PyYAML, kubernetes install procedure Cray-HPE/cray-product-catalog#263

Merged

ruhul-opcity approved these changes Jul 17, 2023

View reviewed changes

galgertz mentioned this pull request Jul 17, 2023

Error installing Pyyaml==5.4, Cython_sources #724

Closed

ghislainbourgeois approved these changes Jul 17, 2023

View reviewed changes

maxhniebergall mentioned this pull request Jul 17, 2023

Installing Rally develop mode blocked by transitive dependency elastic/rally#1751

Closed

mharding-hpe mentioned this pull request Jul 17, 2023

CASMCMS-8713: Temporarily modify PyYAML install procedure in Dockerfile Cray-HPE/cf-gitea-import#89

Merged

rwblokzijl approved these changes Jul 17, 2023

View reviewed changes

themowski mentioned this pull request Aug 29, 2023

Unable to install CKAN dependencies in Alpine Linux due to old PyYAML version ckan/ckan#7766

Closed

anxjok mentioned this pull request Sep 5, 2023

Backport #702 to pyyaml 5.4 #751

Closed

nitzmahone mentioned this pull request Sep 6, 2023

Switch to "new_build_ext" broke PyYAML cython/cython#4568

Open

elukey added a commit to elukey/yamlconf that referenced this pull request Sep 15, 2023

Bump pyyaml version to 6.0.1

8c7f779

The new release includes yaml/pyyaml#702

elukey mentioned this pull request Sep 15, 2023

Bump pyyaml version to 6.0.1 halfak/yamlconf#8

Open

This was referenced Sep 18, 2023

Bump pyyaml version to 6.0.1 halfak/yamlconf#9

Closed

bump YAML to 6.0.1 wikimedia/yamlconf#1

Merged

amercader mentioned this pull request Oct 17, 2023

[2.10] Bump pyyaml to 6.0.1 ckan/ckan#7864

Merged

dhruvmanila mentioned this pull request Nov 3, 2023

Upgrade PyYAML to 6.0.1 to avoid build error astral-sh/ruff#8460

Merged

dhruvmanila added a commit to astral-sh/ruff that referenced this pull request Nov 3, 2023

Upgrade PyYAML to 6.0.1 to avoid build error (#8460)

a08c5b7

Refer: yaml/pyyaml#702

Jack-Dane mentioned this pull request Nov 7, 2023

[BUG] "pip install docker-compose" fails on debian 12 docker/compose#11168

Closed

mreso added a commit to pytorch/serve that referenced this pull request Feb 27, 2024

bump up pyyaml version to avoid cython 3 issue yaml/pyyaml#702

ef62412

mcmanustfj mentioned this pull request Mar 26, 2024

PyYAML version 6 can't build, upgrade to 6.0.1 deepdoctection/deepdoctection#304

Closed

licquia mentioned this pull request Apr 5, 2024

CI is failing spdx/spdx-spec#867

Open

rpatureau mentioned this pull request Apr 19, 2024

Compatibility with newer Python versions lbl-srg/BuildingsPy#547

Merged

noahp mentioned this pull request Apr 30, 2024

Support building chip-tool with python3.12 project-chip/connectedhomeip#33242

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pin Cython build constraint to < 3.0 #702

Pin Cython build constraint to < 3.0 #702

nitzmahone commented Mar 3, 2023 •

edited

keith commented Jul 17, 2023

rahul-theorem commented Jul 17, 2023

hssyoo commented Jul 17, 2023

DeanWay commented Jul 17, 2023 •

edited

mharding-hpe commented Jul 17, 2023

addyess left a comment

enmatt commented Jul 17, 2023

cgoessling left a comment

addyess commented Jul 17, 2023

nitzmahone commented Jul 17, 2023

antoniogomezalvarado commented Jul 17, 2023 •

edited

Pin Cython build constraint to < 3.0 #702

Pin Cython build constraint to < 3.0 #702

Conversation

nitzmahone commented Mar 3, 2023 • edited

keith commented Jul 17, 2023

rahul-theorem commented Jul 17, 2023

hssyoo commented Jul 17, 2023

DeanWay commented Jul 17, 2023 • edited

mharding-hpe commented Jul 17, 2023

addyess left a comment

Choose a reason for hiding this comment

enmatt commented Jul 17, 2023

cgoessling left a comment

Choose a reason for hiding this comment

addyess commented Jul 17, 2023

nitzmahone commented Jul 17, 2023

antoniogomezalvarado commented Jul 17, 2023 • edited

nitzmahone commented Mar 3, 2023 •

edited

DeanWay commented Jul 17, 2023 •

edited

antoniogomezalvarado commented Jul 17, 2023 •

edited