New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Locked info for local wheel should contain hash of the wheel file itself #2394

Closed
uranusjr opened this Issue Jun 20, 2018 · 4 comments

Comments

Projects
4 participants
@uranusjr
Member

uranusjr commented Jun 20, 2018

From #2390. Take googleappenginepipeline for example, here’s the lock file

{
    "_meta": {
        "hash": {
            "sha256": "6929c0ba32c595c656469787bbbe9ae6f7f840b5bc03b9e97dd518c3468f3672"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "googleappenginecloudstorageclient": {
            "hashes": [
                "sha256:b00ad9e7ca5a5524f07b6099b4c1e920cca9badc8e0d57cb6c1e2fb976356c95"
            ],
            "version": "==1.9.22.1"
        },
        "googleappenginepipeline": {
            "hashes": [
                "sha256:cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c"
            ],
            "path": "./GoogleAppEnginePipeline-1.9.22.1-py2-none-any.whl",
            "version": "==1.9.22.1"
        }
    },
    "develop": {}
}

The hash cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c is the source archive’s hash from PyPI, but that does not match the wheel’s, and causes an failure when you run pipenv sync:

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    GoogleAppEnginePipeline==1.9.22.1 from file:///.../GoogleAppEnginePipeline-1.9.22.1-py2-none-any.whl (from -r ... (line 1)):
        Expected sha256 cef7f04172870db661c8ae26fc1417b1bb30bdcd0019b1cdd15c5c4cd3c0ae1c
             Got        13210232486bf683d928aae1a2d9d97edada6d2d83063dc23b71fb6bbcee9874

The solution to this, I think, is to include the wheel file’s hash with those from PyPI. The user should be responsible for providing a legit wheel file during locking, and if (s)he does, this hash can be used to ensure the wheel file used during sync is also legit.

@uranusjr uranusjr added the bug 🐛 label Jun 20, 2018

@uranusjr uranusjr added this to To do in 2018.7.x Release via automation Jul 5, 2018

@uranusjr uranusjr moved this from To do to Potential things to work on in 2018.7.x Release Jul 26, 2018

@uranusjr uranusjr added this to Potential things to work on in Next Jul 30, 2018

@uranusjr uranusjr removed this from Potential things to work on in 2018.7.x Release Jul 30, 2018

@Laski

This comment has been minimized.

Laski commented Oct 14, 2018

Any workaround until this is fixed? Using --skip-lock every time kinda defeats the purpose of pipenv.

@Laski

This comment has been minimized.

Laski commented Oct 14, 2018

Found a workaround: changing the name of the package to a random string (or any string that's not a valid pypi package) prevents pipenv from searching the hash in pypi, but the installation works because a local wheel does not seem to need a correct package name.

This fails:

mysqlclient = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}

But this works:

mysqlclient-win = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}

And this should too (haven't tested it tho):

majhxhxoic = {path = "./dependencies/mysqlclient-1.3.13-cp35-cp35m-win32.whl", os_name = "== 'nt'"}
@andyneff

This comment has been minimized.

andyneff commented Oct 18, 2018

This is also an issue for downloaded wheels that are not from pypi, and thus should not be compared to it. pipenv attempts to compare to hashes on pypi which just doesn't make sense, nor is it security, since pipenv install --keep-outdated doesn't have this issue, and actually works.

Also seen in #3053 #2863 and #2757

TLDR;

If the wheel is local/an external file, not a pypi file, the hash should represent that file or not be there at all, but should not be the hash of pypi

Works

pipenv install https://github.com/uqfoundation/dill/archive/dill-0.2.8.zip
pipenv --rm
pipenv install --keep-outdated

Just showing an example of setting up an environment, pretending I'm "starting over", and then "I want to reinstall what's in the Pipfile/Pipfile.lock". But in this case, the zip file is left alone, in peace, like you would expect. pipenv install also works, as does pipenv lock

Also works as expected

pipenv install https://github.com/uqfoundation/dill/archive/dill-0.2.8.zip#egg=dill
...

Does not work as intended

pipenv install http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl

And I can't even get past that, without using @Laski 's workaround, which I can't figure out from the pipenv cli.

Other incomplete workarounds

#Egg

pipenv install http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl#egg=foobar
pipenv --rm
pipenv install --keep-outdated

Will not actually work. The Pipfile will contain a torch entry (not foobar) and the Pipfile,lock will contain none. So pipenv install --keep-outdated does nothing. pipenv install or pipenv lock just repeats the same hash issue.

Editing Pipfile.lock

You can edit your Pipfile.lock file to look like this

{
    "_meta": {
        "hash": {
            "sha256": "a498ccb6a730b630bed3d7ce40bdb058e9f64cd962f70bc640366b5f9d2d3c38"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "torch": {
            "file": "http://download.pytorch.org/whl/cu92/torch-0.4.1-cp36-cp36m-linux_x86_64.whl"
        }
    },
    "develop": {}
}

And now, pipenv install --keep-outdated works, but any attempt to pipenv lock will lead to the same hash disaster. So this is not a long term solution, as a project using this workaround can no longer breath.

Pytorch background

Pytorch links with a library (called cuda) that can come in a number of incompatible versions, 8.0, 9.0, 9.1, etc... So they decided to put the 9.0 version on pypi and you have to download the wheel for any other version. While on the surface it may seem weird, this is just a use case of "using a wheel not on pypi". And currently, I can't find a work around short of manually editing the Pipfile, to make this work. The current behavior seems to be buggy, and does not add "security", as --keep-outdated makes this work partially.

@ed-alertedh

This comment has been minimized.

ed-alertedh commented Nov 2, 2018

I can confirm @Laski 's workaround is the easiest way to deal with this for now.

techalchemy added a commit that referenced this issue Nov 2, 2018

Include hashes from local artifacts
- Fixes #2394

Signed-off-by: Dan Ryan <dan@danryan.co>

Next automation moved this from Potential things to work on to Done Nov 3, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment