Installing package from git fails on cryptograpghy - Cano not find Rust compiler #3661
Comments
Ikszad
added
kind/bug
|
We are getting the same error, it looks like the dependency chain is Somewhere down that chain its pulling down a source version of cryptography which requires Rust to compile, so not entirely sure this is a poetry problem, but its the install of poetry in our build environment that is triggering the chain of events. |
|
In our environment we are seeing this when installing poetry via the cheese shop on Alpine 3.10.
|
|
According to what @gord1anknot said I tried setting the cryptography to <3 and it also worked for me. This is how my package pyproject.toml (gist b from issue post) looks like now: |
|
IMO it kind of is a poetry problem, in that I'm not expecting |
|
Hello, this is a recent change in cryptography. It always did compile on alpine but now also needs a rust compiler. See pyca/cryptography#5771 I would strongly recommend to not use alpine for python but the official debian images; as these are compatible with manylinux. @brettdh I'd say this depends on your pins, no? Depending on how and what you specified, you are giving poetry the freedom to update within certain version ranges. Anyways, this issue can imo be closed because this is the result of a newer cryptography version and can be reproduce with "pip install cryptography" in every alpine container. |
|
Just to clarify about my recommendation to not use alpine images: Using the debian images means you can use the manylinux wheels, which means that pip does not have to compile anything, this will result in faster image build times. The reason for this is that alpine uses musl which is not supported by manylinux. |
|
I don’t depend directly on cryptography, and it doesn’t appear in my lock file. How do I know to pin it until a breaking change like this comes along?
My point is that there are accidental build dependencies that I’m not aware of and that I don’t know aren’t already pinned. That by itself seems not great, and counter to the way that I don’t have to think about individually pinning any other dependency; poetry just takes care of it for me.
It’s also worth noting that, to my best recollection, I didn’t upgrade poetry itself, and yet it pulled in this new version of cryptography during `poetry install` anyway.
…
On Feb 9, 2021 at 3:57 AM, <Florian Apolloner ***@***.***)> wrote:
Hello, this is a recent change in cryptography. It always did compile on alpine but now also needs a rust compiler. See pyca/cryptography#5771 (pyca/cryptography#5771) I would strongly recommend to not use alpine for python but the official debian images; as these are compatible with manylinux.
@brettdh (https://github.com/brettdh) I'd say this depends on your pins, no? Depending on how and what you specified, you are giving poetry the freedom to update within certain version ranges.
Anyways, this issue can imo be closed because this is the result of a newer cryptography version and can be reproduce with "pip install cryptography" in every alpine container.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (#3661 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAAWLHXE2NBHYYQVGEGXTJLS6D2ITANCNFSM4XJHBOKA).
|
You are right sorry; I thought you also included poetry somehow manually. That said, as it currently looks your best chances are switching to a platform that supports manylinux wheels; otherwise that problem will persist (or poetry locks down |
|
it also worked for my by setting crytography<3. Would be great if someone know a better solution |
|
@apollo13 I'm fine with the workarounds above:
If you agree that the implicit, accidental unpinned dependency issue is worth investigating and improving, I would suggest that this issue remain open to track that, using the unexpected cryptography upgrade as an example. Aside: FWIW, my build env is a docker container that extends |
Absolutely, I am no maintainer, so it is not my choice anyways :) This is what I get in a Do you mind pasting the full output that you get when it errors out for you. I'd like to doublecheck that there isn't another issue with the cryptography package. |
|
Yep, I’ll post a minimal repro in a little bit.
…
On Feb 9, 2021 at 11:29 AM, <Florian Apolloner ***@***.***)> wrote:
>
>
> If you agree that the implicit, accidental unpinned dependency issue is worth investigating and improving, I would suggest that this issue remain open to track that, using the unexpected cryptography upgrade as an example.
>
>
Absolutely, I am no maintainer, so it is not my choice anyways :)
This is what I get in a python:3.6 (which is debian based as you say) container:
***@***.***:/# pip install cryptography Collecting cryptography Downloading cryptography-3.4.3-cp36-abi3-manylinux2014_x86_64.whl (3.2 MB) |████████████████████████████████| 3.2 MB 3.8 MB/s Collecting cffi>=1.12 Downloading cffi-1.14.4-cp36-cp36m-manylinux1_x86_64.whl (401 kB) |████████████████████████████████| 401 kB 3.7 MB/s Collecting pycparser Downloading pycparser-2.20-py2.py3-none-any.whl (112 kB) |████████████████████████████████| 112 kB 4.3 MB/s Installing collected packages: pycparser, cffi, cryptography Successfully installed cffi-1.14.4 cryptography-3.4.3 pycparser-2.20
Do you mind pasting the full output that you get when it errors out for you. I'd like to doublecheck that there isn't another issue with the cryptography package.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (#3661 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/AAAWLHWWHUCFSL5L4HOIZ2DS6FPHLANCNFSM4XJHBOKA).
|
|
Thanks for the updates @apollo13, we build a docker using python:3.6-slim which is erroring. But stripping the dockerfile back to: runs fine, so it must be something we have in our build steps that is causing it to stop using |
|
Similar results to @EdgyEdgemond so far for me... I thought this would repro the issue, but it hasn't. https://github.com/brettdh/poetry-unpinning-repro/runs/1865173612 Main differences from my actual repro:
Other than that, I can't see that I've changed much - but the internal build was failing at Gotta put this down for now; I'll take another look later. |
|
Btw probably not a full fix, but can you do the following in your pyproject.toml? This is what poetry nowadays generates. Previously it had something like: This means that to build/install such a package it would require poetry to be installed. Which the new variant it is just poetry-core. |
|
Mhm that would just be a build dependency, as poetry generates a setup.py after all. Maybe I am down a completely wrong road. |
|
I have had a handful of builds fail and then succeed again over the past 24 hours, so I'm inclined to think there's some non-determinism at play here. The main thing that comes to mind is that all the pypi interactions are going through Sonatype Nexus Repository, which itself is doing some amount of caching. So, it's possible that the Nexus pulled in a sdist before the wheels were published (I noticed about a 5-minute difference in their upload timestamps), cached it for a while (during which time builds failed because rustc was absent), and then later builds installed the wheel without a problem. It would also explain why I can't repro the problem anymore with Doing a plain I have some amount of normalcy restored, but now I have another question: is there a straightforward way to pin poetry and all its deps when installing it into a virtualenv? |
|
Hello, it seems to me, that different questions are raised and mixed here together. Let's try to order them: Why you cannot install successfully the cryptography package? poetry decided to download the alpine is minimal distribution with the focus on security. So it often lacks packages which are included in other distribution by default and you have install those packages by yourself. In case of Why poetry chooses the sdist and not a wheel? I don't have Run a
If you publish the Hope I could clarify some things. fin swimmer |
|
@finswimmer Regarding
it is a little bit more nuanced. It is not because poetry is trying to install cryptography, but because pip is trying to install poetry. See the first traceback in the initial comment. This is due to the fact that an old Edit:// One can also have the problem if they include cryptography in their package, but I do not think that this is what the issue indicates (according to the traceback). But either way it is not really something poetry can fix. |
|
|
|
I simply installed the rust compiler and got past this error. I'm on Mac OS, but you could translate it to your package manager of choice anyways: brew install rustupAfter that open a new terminal to install the compiler itself: rustup-init |
The build is breaking on CircleCI due to the cryptography library, one of the solutions is to pin the version (to something less that ~3.3): python-poetry/poetry#3661 Refs: #2460
The build is breaking on CircleCI due to the cryptography library, one of the solutions is to pin the version: python-poetry/poetry#3661 Refs: #2460
|
@finswimmer could you reopen this issue please? |
|
Please double check you pyproject.toml against my comment (you should have poetry-core as buildsystem). If not you will need to fix that. Bit no matter what this is not a poetry bug. Fix your system so you are able to install cryptography
…On Sat, Mar 27, 2021, at 23:17, Conor Sheehan wrote:
@finswimmer <https://github.com/finswimmer> could you reopen this issue
please?
Like many of the other comments, I have this issue even though my
package does not depend on cryptography.
I think @apollo13 <https://github.com/apollo13> could be on to
something about pip installing poetry
#3661 (comment)
<#3661 (comment)>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3661 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5C4CH2NDIKDOSVAURYLTFZKPHANCNFSM4XJHBOKA>.
|
|
@apollo13 switching to poetry-core fixed it, thanks so much! I did see your comment about switching to poetry-core #3661 (comment)
|
|
EDIT: Please, disregard the comment below if not useful. Upgrading to Poetry 1.1.5 did solve the issue here. I'm still trying to wrap my head around Poetry and am experiencing this issue with a module I've just created. I've done some testing, and although I couldn't figure out what's happening, I though the session below might be useful to help debug this. Note: stock Debian Buster: $ mkdir x
$ cd x
$ python3 -m venv /tmp/venv
$ . /tmp/venv/bin/activate
$ (venv) python -V
Python 3.7.3
$ (venv) poetry -V
Poetry version 1.0.10
$ (venv) poetry init
This command will guide you through creating your pyproject.toml config.
Package name [x]:
Version [0.1.0]:
Description []: Test
Author [Flávio Veloso <XXXXX@XXXXXXXXX.com>, n to skip]:
License []:
Compatible Python versions [^3.7]:
Would you like to define your main dependencies interactively? (yes/no) [yes] no
Would you like to define your development dependencies interactively? (yes/no) [yes] no
Generated file
[tool.poetry]
name = "x"
version = "0.1.0"
description = "Test"
authors = ["Flávio Veloso <XXXXX@XXXXXXXXX.com>"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
Do you confirm generation? (yes/no) [yes]
$ (venv) touch x.py
$ (venv) poetry install
Installing dependencies from lock file
No dependencies to install or update
- Installing x (0.1.0)
[EnvCommandError]
Command ['/tmp/venv/bin/pip', 'install', '-e', '/home/flaviovs/x'] errored with the following return code 1, and output:
Obtaining file:///home/flaviovs/x
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /tmp/venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-nqpk2my1 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12:
Collecting poetry>=0.12
(...loads of lines omitted...)
If you did intend to build this package from source, try installing a Rust compiler from your system package manager and ensure it is on the PATH during installation. Alternatively, rustup (available at https://rustup.rs) is the recommended way to download and update the Rust compiler toolchain.
This package requires Rust >=1.41.0.
----------------------------------------
Command "/tmp/venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-install-igr4xmnv/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-12ve9gcj/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-nqpk2my1 --compile --install-headers /tmp/venv/include/site/python3.7/cryptography" failed with error code 1 in /tmp/pip-install-igr4xmnv/cryptography/
----------------------------------------
Command "/tmp/venv/bin/python3 -m pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-nqpk2my1 --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in NoneNotes:
|
|
Edit:// Sorry, the edit was not visible when answering via mail
Please read the comments above. You can make the same fixes that I already outlined three times or so and got confirmed by the respective users
…On Mon, Apr 5, 2021, at 01:17, Flávio Veloso wrote:
I'm still trying to wrap my head around Poetry and am experiencing this
issue with a module I've just created. I've done some testing, and
although I couldn't figure out what's happening, I though the session
below might be useful to help debug this.
Note: stock Debian Buster:
$ mkdir x
$ cd x
$ python3 -m venv /tmp/venv
$ . /tmp/venv/bin/activate
$ (venv) python -V
Python 3.7.3
$ (venv) poetry -V
Poetry version 1.0.10
$ (venv) poetry init
This command will guide you through creating your pyproject.toml config.
Package name [x]:
Version [0.1.0]:
Description []: Test
Author [Flávio Veloso ***@***.***>, n to skip]:
License []:
Compatible Python versions [^3.7]:
Would you like to define your main dependencies interactively? (yes/no)
[yes] no
Would you like to define your development dependencies interactively?
(yes/no) [yes] no
Generated file
[tool.poetry]
name = "x"
version = "0.1.0"
description = "Test"
authors = ["Flávio Veloso ***@***.***>"]
[tool.poetry.dependencies]
python = "^3.7"
[tool.poetry.dev-dependencies]
[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
Do you confirm generation? (yes/no) [yes]
$ (venv) touch x.py
$ (venv) poetry install
Installing dependencies from lock file
No dependencies to install or update
- Installing x (0.1.0)
[EnvCommandError]
Command ['/tmp/venv/bin/pip', 'install', '-e', '/home/flaviovs/x']
errored with the following return code 1, and output:
Obtaining file:///home/flaviovs/x
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
Complete output from command /tmp/venv/bin/python3 -m pip install
--ignore-installed --no-user --prefix /tmp/pip-build-env-nqpk2my1
--no-warn-script-location --no-binary :none: --only-binary :none: -i
https://pypi.org/simple -- poetry>=0.12:
Collecting poetry>=0.12
(...loads of lines omitted...)
If you did intend to build this package from source, try
installing a Rust compiler from your system package manager and ensure
it is on the PATH during installation. Alternatively, rustup (available
at https://rustup.rs) is the recommended way to download and update the
Rust compiler toolchain.
This package requires Rust >=1.41.0.
----------------------------------------
Command "/tmp/venv/bin/python3 -u -c "import setuptools,
tokenize;__file__='/tmp/pip-install-igr4xmnv/cryptography/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-record-12ve9gcj/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-nqpk2my1 --compile --install-headers /tmp/venv/include/site/python3.7/cryptography" failed with error code 1 in /tmp/pip-install-igr4xmnv/cryptography/
----------------------------------------
Command "/tmp/venv/bin/python3 -m pip install --ignore-installed
--no-user --prefix /tmp/pip-build-env-nqpk2my1
--no-warn-script-location --no-binary :none: --only-binary :none: -i
https://pypi.org/simple -- poetry>=0.12" failed with error code 1 in
None
Notes:
1. If I rename `x.py` to anything else, then `poetry install` works
(no *Installing x (0.1.0)* though)
2. If I comment out the following lines from `pyproject.toml`, then
`poetry install` also works, *with* *Installing x (0.1.0)*:
`[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"
`
3. `poetry install --no-root` works like # 2 above
4. `poetry show --tree | grep cryptography` is empty in both failed
and non-failed situations
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#3661 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAT5C3JGTAQT3UDHPZ3SQ3THDXQ3ANCNFSM4XJHBOKA>.
|
|
Thank you -- I've read the comments, and edited my comment already:
upgrading Poetry got rid of the problem. I decided to left the comment
intact because it may help someone in the future. Sorry for any extra
work this might have caused.
|
|
Add dependency fro cryptography for Alpine More details for other distro in official Doc |
|
This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
-vvvoption).a) Application pyproject.toml - https://gist.github.com/Ikszad/b4e8c19665220779952622663f3f603c
b) Our package pyproject.toml -https://gist.github.com/Ikszad/9514a754f4e09c370ab1907e54052291
c) Base dockerfile - https://gist.github.com/Ikszad/0f002345680c95e384e0b879995622f1
d) Base pyproject.toml - https://gist.github.com/Ikszad/a332db7e7e6a542cfa7595fc6a0871e2
e) Child dockerfile - https://gist.github.com/Ikszad/bee76870470660cca7b5a7f0c6c720d4
Issue
Hello!
Here is an issue that has hit us in the production environment out of nowhere.
We have dockerized environment (dockerfiles gist links c and e) in which there are packages installed based on the two pyproject.toml files (firstly base image d and child image a). Besides that, we have a separate package with its pyproject.toml (b).
We have this packages inside the application pyproject.toml (a) as:
hubspot-integration = {git = "https://SOMEPACKAGE.git", branch="master"}Now during the
poetry installin the application build (Dockerfile e using pyproject.toml a) it crashes on the package mentioned above during installation of cryptography. What is interesting installation of cryptography package alone (as we have this dependency defined in the pyproject.toml a) works fine.I tried upgrading pip, setuptools, poetry and neither seem to have any effect.
Thank you for the help
Here is the error message:
The text was updated successfully, but these errors were encountered: