-
Notifications
You must be signed in to change notification settings - Fork 3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
pip wheel
produces a "Hashes are required" when building a wheel from a local sdist
#12942
Comments
Does the constraints file contain |
Yes
…On Wed, Aug 28, 2024, 7:18 PM Damian Shaw ***@***.***> wrote:
Does the constraints file contain --require-hashes?
—
Reply to this email directly, view it on GitHub
<#12942 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBGQDUMASKHYXPZC4RTZTZLDBAVCNFSM6AAAAABNIEIQSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMJWGM4TKNJSGM>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Sorry, to be more precise, the constraints file contains hashes. It doesn't have |
Is it possible to have a self-contained reprod? |
|
In this case the local sdist does not have a hash, so pip’s complaint is not groundless (whether a local sdist needs a hash is another question). I thought in the original issue the local sdist does have a hash but pip fails to recognise it? |
No, there's no hash for the local sdist.
Personally, I find it surprising and unexpected that a hash is required for
a local file.
…On Fri, Aug 30, 2024 at 12:45 AM Tzu-ping Chung ***@***.***> wrote:
pip wheel --require-hashes ./pretend-1.0.9.tar.gz
In this case the local sdist does not have a hash, so pip’s complaint is
not groundless (whether a local sdist needs a hash is another question). I
thought in the original issue the local sdist *does* have a hash but pip
fails to recognise it?
—
Reply to this email directly, view it on GitHub
<#12942 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBE5FIFF7L2LD5ZIZTDZT72GZAVCNFSM6AAAAABNIEIQSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRQGAZTEMZWGI>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
While I can see the argument one way or the other, I have worked in teams where the repository is a network attached storage, and if they were concerned about data integrity and hashed their contents before copying it to the network, they would expect How practical this scenario is I don’t know, but usually every feature of pip is relied on by someone. |
I'm reticent to suggest a flag, since that's just Yet Another Thing for users. But if the existing behavior is desired, then maybe this is a feature request for some way to disable this behavior, and only enforce hashes for PyPI packages. |
To clarify what you intend here, do you mean just PyPI, or any index (specified via I sympathise with the idea that hashes are more important for some sources than for others, but I'm not at all clear where we draw the line - and I don't personally use hashes, so I have to be guided by what our users seem to want, which mostly feels like "hashes enforced everywhere, except for the occasional place that I don't want them to be enforced"1. Footnotes
|
Sorry, I should have said index-provided. I don't know what to do about the general case of local file path. It seems clear (to me at least?) that "the sdist I'm building a wheel out of" is a distinct case than the more general |
To understand your workflow a bit better, why are you using If your goal is to build the sdist and you're not worried about its data integrity, why not just build it without resolving its dependencies? i.e. drop |
Because I want to pin the versions of |
I wasn't aware that If I am reading this correctly, you are building the wheel(s), copying the wheel(s) to another location to be used, and then venv is not used further? Perhaps as a workaround, you could:
I do something very similar for one of my build steps, I don't need build isolation because I can already create a reproducible pinned build environment via the docker steps, and I'm not reusing that environment for anything else. Hope this helps anyway. |
I suppose it's possible I agree that it's possible to work around this by simply not relying on build isolation, but this increases the complexity of the build. It really should be possible to build a wheel from an sdist while exerting precise control over all dependencies to be built. (If there's a better tool than pip for this, I'm happy to hear it, but I'm not aware of another.) |
If you're building a wheel, rather than installing, build might be better for you. |
build has no way to take an sdist as input, AFAICT.
…On Fri, Aug 30, 2024 at 11:22 AM Paul Moore ***@***.***> wrote:
(If there's a better tool than pip for this, I'm happy to hear it, but I'm
not aware of another.)
If you're building a wheel, rather than installing, build
<https://pypi.org/project/build/> might be better for you.
—
Reply to this email directly, view it on GitHub
<#12942 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBFDF3RD4XN3A6SIYALZUCEZVAVCNFSM6AAAAABNIEIQSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRRGU4TKNRUGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
I think it is (though I've not tried it with this sdist example), but it requires constructing a constraints.txt which includes a hash of your sdist and point the env variable |
You probably also need to set the env var for require hashes, I think |
My understanding is constraints pre-date isolated builds, and further pip has no user facing way to find out the required build dependencies of a package, therefore workflows which involve using pip freeze to generate pinned constraints might break if Pip-tools has a way of extracting build dependencies: https://github.com/jazzband/pip-tools?tab=readme-ov-file#maximizing-reproducibility, but relies on the same uv improves the situation by separating out regular constraints and build constraints: https://docs.astral.sh/uv/pip/compatibility/#build-constraints, but it's not clear to me from the docs if the CLI option is applied recursively or you need to use I've heard Bazel supports reproducible fully pinned Python projects, but I don't understand the tool well enough that looking at their documentation tells me if this is true or not. |
To take a step back here: My overall goal is to take a local sdist, build a wheel from it, and do so with any downloaded artifacts pinned to a version and hash verified. The last element is presently an impediment because I have a lack of clarity about whether this is desired behavior by the
Perhaps there's other options too, but I'd be interested in which direction the maintainers prefer. |
I can't speak for the other maintainers, but my personal view is somewhere between (2) and (3). I think that In addition, as I said above, I think that pinning down precisely what the semantics of any potential "disable hash verification for local sdists" are would be both difficult to do, and difficult to document. So even if the consensus was (2), I'm against having an option unless someone can prove me wrong by specifiying the behaviour clearly and unambiguously. Having said all of this, I have little or no experience of acually using hash checking mode, so I'd defer to someone with real world experience if they said otherwise. |
My experience working at a place where hash checking is strongly recommended by security but we also have several local requirements is I eventually wound up adding a flag to pip compile to work around this issue. What I do now is I have, requirements.in file which has a list of dependencies to install of local packages. I use pip compile (now uv pip compile) to convert requirements.in to requirements.txt and I include flag --exclude-package/--unsafe-package (name varies by uv vs pip tools) to exclude local packages from .txt file. Then I do pip install --no-dependencies requirements.txt and pip install --no-dependencies requirements.in (second one installs local packages). A little convoluted, but I think current hash checking mode mostly annoying with local/editable dependencies mixed in and forces some tricks like this. The pip tools issue about --unsafe-package also had other people comment using this kind of trick to work around --require-hashes behavior. So my own preference is 1 would make usage of editable/local easier, but today I've found a workable alternative with multiple install commands/files that deals with this issue. Before I found this solution security recommendation boiled down to we lack a good way to handle this case and only see awkward choices. edit: Glancing at how other team's in my company deal with this kind of issue, it either is multiple install commands/requirement files or not use hashes. Although for latter I'm unsure if it's avoid for this issue or they are unaware of using hashes/workaround paths. edit 2: Also one suggested possible solution is flag like —no-hashes package-name that can be specified multiple times and explicitly specify which packages to not check hashes for. That’s roughly how exclude package way. No special logic for local/editable but allow user/script running install to explicitly mark some as fine without hash. |
Okay, I tried to create a workflow for OP without any workarounds or using other tools, and it's not clear to me it's even possible to pin build requirements in an isolated build environment with hashes? In short:
Example of trying to install an unhashed requirement with a hashed constraint:
Full example of minimal workflowI'm using bash for this example, you'll need to adapt to whatever shell you use:
[build-system]
requires = ["setuptools==74.1.1", "wheel==0.44.0"]
build-backend = "setuptools.build_meta"
[project]
name = "minimal_project"
version = "0.1.0"
Am I missing something? Is it possible to use hashes with pip for build requirements at all? I think if hashes from constraints were accepted, then this workflow would work. |
On a side note for OP, uv just added |
For build requirements you can have separate build_requirements.txt file with hashes install that file with no-dependencies and then afterwards for main install do no-build-isolation as a workaround. How do you even determine build requirements is another problem as pip compile/similar tool only produce resolution of install requirements not build ones although in practice my experience is build requirement list is usually very short so I’ve just manually made it. |
Yes, I gave an example of that workflow earlier (#12942 (comment)), but OP was unhappy with |
This is basically a standards issue at the core. Hashes are a form of locking, and currently we have no package locking standard. In particular:
The proposed lockfile standard, PEP 751, includes a section for specifying locked build requirements. That may help with this workflow, once the PEP gets approved and implemented. |
It's only a standards issue in the sense that pip's existing installer features don't currently work in this scenario, but if pip would allow a constraints file to constrain requirements via hashes this would solve this workflow without the need for a new standard.
No, but pip documents how to: https://pip.pypa.io/en/stable/topics/secure-installs/ to pin a requirement via hashes, and it allows a user to specify them in a constraints file, but then it doesn't functionally let the user constrain to those hashes.
For a lock file this seems a little under specified. Specifically I would expect an actual standard around locking to let you lock build requirements per requirement, I guess I'll have to chime in on that very very long discuss thread 🙁 |
I guess that's true, yes. Given that pip is tending to add features based on standards these days, rather than innovating functionality, I think I'd rather see a standards-based solution for this, though.
As you noted, though, constraints files don't allow hashes. Constraint files have historically been very under-documented and prior to the new resolver implementation, had some odd behaviours. They were streamlined and clarified when we implemented them for the new resolver, to act in the "package finding" phase to limit what files the finder could see. With that design, I'm not sure that including hashes in a constraint file makes sense (hash checking happens much later in the install process, if I recall the details correctly). Also, the details of what configuration is shared between the main pip process and the (recursive1) build environment construction is fairly underspecified, having been based on some quite simplified assumptions and then extended as needed. With all of that in mind, re-working the build environment creation process to correctly pass through and respect hashes is likely to be a complicated design and implementation task, and with pip's limited maintainer base, I'm not sure it's the most important thing for us to tackle. All of which is why I'd prefer it if we had a standards-based solution, so the design is done for us, up front.
I view that section of the PEP as indicative that the intention is to cover this area, but it's not something that has had extensive discussion, so yes, it may be under specified. I'd strongly advise you to point out any issues if you think that's the case, as there's a risk otherwise that it'll get missed with all of people's energy having been used up by questions like portability of lockfiles. Footnotes
|
I think pip should immediately error out when hashes are included in the constraints file then, the current user experience is poor as it appears to the user that constraints are allowed, as the error pip produces is much later than reading the constraints file and is not immediately clear why it produces it.
How would these issues be any different adopting this standard vs. supporting hashes in constraints files 😉? |
I'm surprised that hashes are allowed in constraints files. It's possible I've misremembered and they are somhow used in some places? But if not, then yes, we shouldn't be allowing the user to enter plausible-looking data that we ignore.
I'm hoping that "the community" will thrash out the various design issues, so we don't have to 🙂 In particular, I'm hoping the people participating in the lockfile discussion are more motivated to work through the various questions, particularly around recursive build requirements, than we are here (I'm personally only looking at this as a theoretical problem, I have no need myself for locking build requirements). |
Okay, so summarising this:
Given this, does someone still have a scenario where skipping required hashes for local sdists is required for their workflow? And if so, why providing a hash as part of a requirements file doesn't work for them? e.g. echo "file://$(realpath {sdist}.tar.gz) --hash=sha256:$(sha256sum {sdist}.tar.gz | cut -d' ' -f1)" > sdist-requirements.txt
pip {command} -r sdist-requirements.txt |
It should be noted, since opening this issue, uv has gone from not being able to build wheels to fully supporing @alex workflow (as I beleive he is aware):
So, the answer to "How can I use pinned build requirement hashes in an isolated build environment?", is to use uv 0.4.6+. |
Yes, as a practical matter the combination of: a) discovering the depths of
the design space here for pip (much larger than I thought when I filed
this!), and b) uv implementing the behavior I desired, means that I'm far
less motivated to keep driving this issue forward.
…On Thu, Sep 5, 2024 at 3:28 PM Damian Shaw ***@***.***> wrote:
It should be noted, since opening this issue, uv has gone from not being
able to build wheels to fully supporing @alex <https://github.com/alex>
workflow (as I beleive he is aware):
- Add uv build: astral-sh/uv#6895
<astral-sh/uv#6895>
- Support build from source distribution: astral-sh/uv#6898
<astral-sh/uv#6898>
- uv build support build constraints: astral-sh/uv#7085
<astral-sh/uv#7085>
- Respecting hashes in constraints files: astral-sh/uv#7093
<astral-sh/uv#7093>
So, the answer to "How can I use pinned build requirement hashes in an
isolated build environment?", is to use uv 0.4.6+.
—
Reply to this email directly, view it on GitHub
<#12942 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAGBHKSX2U2AO6VSTWA33ZVCWE5AVCNFSM6AAAAABNIEIQSSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMZSGQ4DCOBYGQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--
All that is necessary for evil to succeed is for good people to do nothing.
|
Is it just me, or is it true that the correct answer to any feature request for pip these days is "wait 15 minutes until (To be clear, I'm not upset by this, if anything I'm just impressed and at the same time painfully aware of how much more it's possible to achieve with proper funding and reduced overhead from backward compatibility concerns). |
Description
An innovation like the following:
pip wheel -c constraints-file-with-hashes.txt local-sdist.tar.gz
produces an error like:Note that the package which is missing the hash is the local sdist.
Expected behavior
pip
should enforce hashes for any downloaded/remote packages, but should not require a hash for the local sdist.pip version
24.2
Python version
3.11.9
OS
All
How to Reproduce
pip download --no-binary --no-deps cryptography
pip wheel -c constraints-file-with-hashes.txt cryptography*.tar.gz
Output
No response
Code of Conduct
The text was updated successfully, but these errors were encountered: