Lock file gets incorrect hashes when a custom index is specified #988
Comments
pfmoore
changed the title
|
@pfmoore Im just taking a random stab at this but you may need to rename the wheel to include version info in order for pip-tools to resolve it properly |
|
@techalchemy the wheel name does include the version - If you meant the name in the |
|
@pfmoore no sorry I was on mobile and didn't click all the way through, you covered exactly what I meant. I'll check, we use pip-tools to resolve the lockfile but I am not sure if we are passing the custom indexes to it. It's pulling hashes from pypi right? What happens if you run |
|
I guess the hashes must come from PyPI, but I don't know how, as I can't find them anywhere on there (PyPI has md5 hashes but not sha256). I tried |
|
Oh pypi has sha256 hashes, you just have to request them specifically when you look. For instance, |
|
Ah, I was looking at https://pypi.python.org/pypi/pycurl/json. My mistake. Unfortunately, if the wheel says it's for Python 3.5 (which that one does) it's not installable on Python 3.6. That's why I need to get the wheel from a different index... |
|
Yeah, we rely on Warehouse rather than the legacy pypi server for hashes @pfmoore. I agree this should work for your use case though. A lot of the hash work that we'd previously done was ripped out and redone by Kenneth recently and I don't know if we have proper test coverage around cases like this. We need to try to resolve the hash back to the third party instance, which we don't currently do. In the event it's not available though, we'll probably have to include no hashes. You'll need to use |
|
Well... we've removed |
|
@nateprewitt thanks for the explanation. Am I right that by including There's no massive rush - I can work around this for now just by manually installing pycurl with pip. But I'll be happy to help test if you need me to. |
|
@nateprewitt wait why did we remove |
|
@pfmoore, I'm still making sure I understand our new implementation. I would say yes, that's how I'd prefer it to work, but it seems we may have the json endpoint check hardcoded. I'm going to see how much work it is to get something more general. @techalchemy, I don't know. It was sometime in the last month or two while I was away. |
|
@nateprewitt @pfmoore seems like if pipenv isn't supplied any hashes when it installs, it will ignore hashes by default |
|
@techalchemy So if I remove the hashes from my index it'll work? I don't think it did (I originally didn't have the hashes in the index IIRC). |
|
@techalchemy 😬 so that's the exact functionality we've (I've?) tried to avoid. We definitely shouldn't be installing packages without a check if the hash field is empty. Pipfile.lock files that are sufficiently complex aren't going to be visually read, and that means no one is going to know whether or not a package is actually being validated. If we're going to implement a "security" feature and claim it does something, it MUST do that thing unambiguously. Silently ignoring missing hashes is not an acceptable default. That's why we eventually moved to an opt-in model rather than opt-out. At least until these most recent changes. @pfmoore I've confirmed we do ignore empty hash lists on at least the main Warehouse instance (https://pypi.org). I haven't been able to set up a Windows environment to test your instance. Also in regards to the PEP503 compliance question earlier, the issue with that is we need ALL possible hashes for a given version when building the lockfile. Otherwise, we can't move the lockfile cross platform because we're going to get a different wheel on linux than windows. |
|
@nateprewitt I tend to agree... but even if we could 'fail silently' I couldn't think of a reason why we would want to prevent users from explicitly ignoring them. |
I'm not positive I know what you're referring to with 'fail silently'. If you're referring to silently ignoring hashes, it appears we do that now, and that's wrong imo. There are two acceptable models here from a security perspective. 1.) We provide an 2.) We claim to provide a hashing opt-in and refuse to install anything (except maybe vcs/local dependencies) without a hash. This has the same strictness but requires the users to choose to use it. Having a "We may check but also maybe we won't ¯_(ツ)_/¯" approach to a feature like this is irresponsible and means the implementation isn't useful. It's got to be all or nothing. |
|
Yeah silently ignoring hashes, seems like failing to me if we are claiming to use hash verification by default. A hashing opt-in would be a breaking change for some people now I'm guessing, although if we find hashes in the As far as hash verification for vcs or local dependencies, I didn't dig too far into it but I've been through a lot of the pip codebase and I'm not sure it would be that hard. @vphilippon might know more |
|
@nateprewitt I'm not sure what you are implying by the comment that we can't move the lockfile cross platform. The problem here is that I cannot use PyPI, because there's no binary for my platform on PyPI, and building from source fails. Ideally, I'd tell Pipfile to use my custom index and PyPI, but I don't know how to do that so I've specified only my custom index, as that's the essential one. So there's a number of separate issues here:
My problem is with (1) and to a lesser extent (2). By saying that it's difficult to solve those because it makes the lock file non-portable, you're tying them to (3). I've no objection to having a solution to (3), but it's not an immediate concern for me. So I'd rather it was handled separately. With regard to PEP 503, that's the standardised definition of what "a package index" is. If pipenv needs extra (e.g. a JSON interface) then it needs to make that a lot clearer in the docs, as that presents a much higher barrier to entry for people who need something beyond what PyPI provides (for instance, I'm not sure if It sounds to me (and don't take this as criticism, it's not) that pipenv is focused very much on the "get everything from PyPI" use case, and hasn't fully considered the implications of handling private indexes. That's fine, and if that is the case ("we're still working out how private indexes fit into our design") then I'm OK with that, but it would be worth being a little clearer about that in the docs. But there's no doubt (for instance) that using a private index affects portability of the Anyway, the design principles of pipenv are peripheral here, so I won't say any more on that matter (beyond pointing out that it would be useful to document them, to help people understand what types of project pipenv is applicable for). |
|
@pfmoore I am not sure what is confusing so I will clarify a couple of points.
So the reality is that actually we have considered private indexes quite a lot, and they are part of the design, obviously. However, this is an open source project and handling the various complexities around resolving dependencies using those private indexes is not as straightforward as you seem to suggest. Also, I do want to point out that if you want to criticize our support for PEP standards or common tools like devpi you should probably try them first. We support devpi. So yeah, this particular case actually has nothing to do with any of these things. Pipenv supports using multiple indexes. It will always pick up all available hashes when it does hash comparisons, but currently it only picks up hashes from PyPI. This has nothing to do with where your package gets installed from. This only matters for hash comparisons when validating lockfiles. That’s what @nateprewitt and I have been talking about. I don’t know why this turned into a whole discussion about whether we intend to support PEP 503 and that we should tell people they can’t use private indexes and PyPI at the same time How about instead we fix the behavior and allow people to use private indexes as before? |
|
Hmm, looks like I did cause offence, which I didn't mean to - I apologise. I've been struggling to understand how pipenv works with my use case, and I suspect I've been further confused by hitting bugs, assuming it was me doing something wrong, and then things getting out of hand as I kept trying to "explain". My apologies for that also. I'll leave you guys to thrash out the implementation details. If there is anything you need clarification on in what I'm trying to do, then feel free to ask - I'll be happy to explain further. But I'll avoid doing so unprompted, as I'm doing a lousy job of helping when I do that :-( I appreciate you spending time helping me with this, and it's definitely not my intent to criticise, or imply that you've done anything wrong. |
erinxocon
added
Type: Question ❔
|
@pfmoore no worries and sorry if I responded harshly, I answered before I had coffee this morning! Generally I'm on IRC as hawkerz in #pipenv on freenode if you want to have more in depth discussion but the tl;dr is that the things you want to do should work. Essentially, and sorry for not being clear about this also, your use case should and will be supported. As a workaround until we can resolve private index hashes, we built in functionality to allow users to skip hash checking. That seems to be auto-toggled now and I'm not sure what we need to do to fix the short term problem, but the long term problem is to properly handle hashes. This hasn't been high on the priority list but may need to be moved up. |
Thanks for the clarification. My frustration was essentially because I really wasn't sure if that was the case. Having established that this is "just" a bug, and not me misunderstanding how I should be using the tool, I'm much less bothered - I appreciate that things take time in open source, and I trust you guys to prioritise the workload appropriately.
No need just for me (I have plenty of alternative workflows that I can use in the immediate term). But you may find more people hitting this if pipenv becomes the PyPA recommended tool. I've no feel for how likely that is, though (my experience of "people hitting weird corner cases" is almost certainly biased as a result of being a pip developer!). |
|
@pfmoore We've encountered a bunch of people using devpi or private indexes but most of them haven't actually loaded SHA256 hashes so pipenv has automatically turned off hash comparisons. I will dig into this a bit if I can. We do need to work on documentation, I think we are very behind on that |
|
@nateprewitt @pfmoore @vphilippon i was looking at this a bit today and I remembered that pip also supports putting pip configuration files inside virtualenvs. Do any of you know whether the pip-tools resolver would resolve hashes on an index specified in a |
|
There's no plans for pip 10 to change the way config settings work - it's all in the docs so it's fully supported and would require a deprecation period if we were to change it. |
|
I think respecting a |
|
I don't know if we should be forcing users to have to create one for every project. It's intended to typically be a global file that you edit in one place. I haven't dug into why we're discarding those settings in our copy of pip, but fixing that seems like the better approach to me. |
|
Yeah you are correct. I suppose that would be more inline with how pip in a virutalenv works. |
|
Technically we don't disregard the settings entirely, I'm pretty sure they should pass through... |
|
My use case is per-project (I don't want the custom index in my global |
|
That’s the precise issue. It seems like the pip-tools hash resolver doesn’t rely on parsing URLs, and the only other way we acquire hashes is from the pypi JSON api directly. Pip-tools appears to have the capability to get a hash for a remote file if it needs to, for any PEP 503 index. But adding an index to In terms of respecting existing options, I’m not too sure. If we are tracking sources in the |
|
@techalchemy Late reply on my part sorry, but: As far as I can tell, |
|
@vphilippon I passed a custom index directly to the |
|
hashes are only properly supported against pypi at this time |
|
So if I am using a non-PyPI index, how do I suppress the hash mismatch error? Will |
I am specifying a custom index in
Pipfileto host a wheel forpycurl(which is also on PyPI, but does not have a wheel for my environment there). My index just hosts the wheel, with a SHA256 hash specified.When I do
pipenv install, I get an error saying that the hash is not found, and showing a list of expected hashes. I don't know where it got that list from, but it's not from my index... The install fails, and doesn't installpycurl.Describe you environment
$ python -V. Python 3.6.2$ pipenv --version. pipenv, version 8.2.7Pipenv file:
Expected result
I expected pipenv to download the wheel (which is valid for my environment, confirmed via
pip install) and install it.Actual result
Output from
pipfile install --verbose:Steps to replicate
pipenv install.The text was updated successfully, but these errors were encountered: