Fresh install has hash errors for python-logstash on Arm based os

[jmurrayufo]

Installing python-logstash and smbus2 from pipenv on a Raspberry Pi results in errors that seem to be related to Pipenv's hashing?

Describe your environment

1. Linux roachhab 4.9.59-v7+ Ensure 'run <command>' arguments are not stolen by 'run' arguments #1047 SMP Sun Oct 29 12:19:23 GMT 2017 armv7l GNU/Linux (Raspberrian Debian)
2. Python version: Python 2.7.13 and Python 3.6.3
3. Pipenv version: pipenv, version 9.0.0

Expected result

Virtualenv install of python-logstash and smbus2

Actual result

Sample run of actual development Pipfile

pipenv update
Updating all dependencies from Pipfile…
Found 11 installed package(s), purging…
Uninstalling aiofiles-0.3.2:
  Successfully uninstalled aiofiles-0.3.2
Uninstalling crc16-0.1.1:
  Successfully uninstalled crc16-0.1.1
Uninstalling crcmod-1.7:
  Successfully uninstalled crcmod-1.7
Uninstalling httptools-0.0.9:
  Successfully uninstalled httptools-0.0.9
Uninstalling logstash-formatter-0.5.16:
  Successfully uninstalled logstash-formatter-0.5.16
Uninstalling python-logstash-0.4.6:
  Successfully uninstalled python-logstash-0.4.6
Uninstalling sanic-0.7.0:
  Successfully uninstalled sanic-0.7.0
Uninstalling Sanic-Auth-0.2.0:
  Successfully uninstalled Sanic-Auth-0.2.0
Uninstalling smbus2-0.2.0:
  Successfully uninstalled smbus2-0.2.0
Uninstalling ujson-1.35:
  Successfully uninstalled ujson-1.35
Uninstalling websockets-4.0.1:
  Successfully uninstalled websockets-4.0.1

Environment now purged and fresh!
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (48f27e)!
Installing dependencies from Pipfile.lock (48f27e)…
An error occurred while installing python-logstash==0.4.6! Will try again.
An error occurred while installing smbus2==0.2.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 12/12 — 00:00:17
Installing initially–failed dependencies…
Collecting python-logstash==0.4.6 ▉▉▉ 0/2 — 00:00:00
  Using cached https://www.piwheels.hostedpi.com/simple/python-logstash/python_l                                                                                                  ogstash-0.4.6-py3-none-any.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated t                                                                                                  he package versions, please update the hashes. Otherwise, examine the package co                                                                                                  ntents carefully; someone may have tampered with them.
    python-logstash==0.4.6 from https://www.piwheels.hostedpi.com/simple/python-                                                                                                  logstash/python_logstash-0.4.6-py3-none-any.whl#sha256=0a184b86c27d17fe328247347                                                                                                  3568da545f8b51b1201697d01576ddef0790e97 (from -r /tmp/pipenv-3qjbcm_e-requiremen                                                                                                  t.txt (line 1)):
        Expected sha256 10943e5df83f592b4d61b63ad1afff855ccc8c9467f78718f0a59809                                                                                                  ba1fe68c
             Got        0a184b86c27d17fe3282473473568da545f8b51b1201697d01576dde                                                                                                  f0790e97


  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 — 

Of note, only these two packages seem to have this error. I am unsure as to why this is (not really knowing enough about the pip system in general to see how they might be different).
Reducing the Pipfile to just smbus2 and python-logstash as follows:
Test Pipfile

[[source]]

url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"


[packages]

"smbus2" = "*"
python-logstash = "*"

[dev-packages]



[requires]

python_version = "3.6"

Resulting Error

pipenv install
Installing dependencies from Pipfile.lock (58d3ae)…
An error occurred while installing python-logstash==0.4.6! Will try again.
An error occurred while installing smbus2==0.2.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:03
Installing initially–failed dependencies…
Collecting python-logstash==0.4.6 ▉▉▉ 0/2 — 00:00:00
  Using cached https://www.piwheels.hostedpi.com/simple/python-logstash/python_logstash-0.4.6-py3-none-any.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    python-logstash==0.4.6 from https://www.piwheels.hostedpi.com/simple/python-logstash/python_logstash-0.4.6-py3-none-any.whl#sha256=0a184b86c27d17fe3282473473568da545f8b51b1201697d01576ddef0790e97 (from -r /tmp/pipenv-ijo7c_vx-requirement.txt (line 1)):
        Expected sha256 10943e5df83f592b4d61b63ad1afff855ccc8c9467f78718f0a59809ba1fe68c
             Got        0a184b86c27d17fe3282473473568da545f8b51b1201697d01576ddef0790e97


  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 —

Results seem the same if using pipenv install on a fresh OS, and git checkout, or when using pipenv update otherwise. Doing a pip3 install of the packages succeeds, but sort of removes the utility of pipenv.

Steps to replicate

1. Fresh install of Raspberrian on a Raspberry Pi 3
2. Fresh compile of Python 3.6.3 (Tested for functionality, works as expected)
3. Install pip3
4. Install pipenv from pip3
5. Create test directory
6. Setup work space with pipenv --python 3.6
7. Run command pipenv install python-logstash smbus2 (This will succeed, and simulates the creation of the git downloaded Pipfile. Note: I am not using git to push the Pipfile.lock at this time.)
8. Run command pipenv update (This will fail with the following)

pipenv update
Updating all dependencies from Pipfile…
Found 2 installed package(s), purging…
Uninstalling python-logstash-0.4.6:
  Successfully uninstalled python-logstash-0.4.6
Uninstalling smbus2-0.2.0:
  Successfully uninstalled smbus2-0.2.0

Environment now purged and fresh!
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (58d3ae)!
Installing dependencies from Pipfile.lock (58d3ae)…
An error occurred while installing python-logstash==0.4.6! Will try again.
An error occurred while installing smbus2==0.2.0! Will try again.
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 2/2 — 00:00:03
Installing initially–failed dependencies…
Collecting python-logstash==0.4.6 ▉▉▉ 0/2 — 00:00:00
  Using cached https://www.piwheels.hostedpi.com/simple/python-logstash/python_logstash-0.4.6-py3-none-any.whl

THESE PACKAGES DO NOT MATCH THE HASHES FROM Pipfile.lock!. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    python-logstash==0.4.6 from https://www.piwheels.hostedpi.com/simple/python-logstash/python_logstash-0.4.6-py3-none-any.whl#sha256=0a184b86c27d17fe3282473473568da545f8b51b1201697d01576ddef0790e97 (from -r /tmp/pipenv-qs2e80iu-requirement.txt (line 1)):
        Expected sha256 10943e5df83f592b4d61b63ad1afff855ccc8c9467f78718f0a59809ba1fe68c
             Got        0a184b86c27d17fe3282473473568da545f8b51b1201697d01576ddef0790e97


  ☤  ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/2 — 

[nateprewitt]

@jmurrayufo would you mind posting the lockfile that your system is generating?

[jmurrayufo]

Pipfile.lock from my 'real world' case

{
    "_meta": {
        "hash": {
            "sha256": "dabfec1aefc7f605462f41cf9fe145e7cd93ebe5d7a8265f8aceb189bd48f27e"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "armv7l",
            "platform_python_implementation": "CPython",
            "platform_release": "4.9.59-v7+",
            "platform_system": "Linux",
            "platform_version": "#1047 SMP Sun Oct 29 12:19:23 GMT 2017",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "linux"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "aiofiles": {
            "hashes": [
                "sha256:25c66ea3872d05d53292a6b3f7fa0f86691512076446d83a505d227b5e76f668",
                "sha256:852a493a877b73e11823bfd4e8e5ef2610d70d12c9eaed961bcd9124d8de8c10"
            ],
            "version": "==0.3.2"
        },
        "crc16": {
            "hashes": [
                "sha256:c1f86aa0390f4baf07d2631b16b979580eae1d9a973a826ce45353a22ee8d396",
                "sha256:eb47b9cec9818b684537d05552ff10404c92d9dcdd455156456b19e7423855bc",
                "sha256:7998bb0cacb0005ba62f963540c0f7cb09e86b51f6b4e64ed5632f31a7266fa1",
                "sha256:1439e3cc0244a4758aa2d40a31b062086c24f5602046ec2fa4356484c4b5a385",
                "sha256:521643768ff000a7758bc1f1c5e1dc41ae64b42fa57c50e133ab083cfaf9b8f9",
                "sha256:63db8577ce0e03b39f30071166c6667659124a65d7fcd38adffe6a34487ce6fb",
                "sha256:1b9f697a93491ae42ed653c1e78ea25a33532afab87b513e6890975450271a01"
            ],
            "version": "==0.1.1"
        },
        "crcmod": {
            "hashes": [
                "sha256:dc7051a0db5f2bd48665a990d3ec1cc305a466a77358ca4492826f41f283601e",
                "sha256:69a2e5c6c36d0f096a7beb4cd34e5f882ec5fd232efb710cdb85d4ff196bd52e",
                "sha256:737fb308fa2ce9aed2e29075f0d5980d4a89bfbec48a368c607c5c63b3efb90e",
                "sha256:50586ab48981f11e5b117523d97bb70864a2a1af246cf6e4f5c4a21ef4611cd1"
            ],
            "version": "==1.7"
        },
        "httptools": {
            "hashes": [
                "sha256:f47f9870f19e3488e8def0898c46f87d36e5d59e90eb77d01453a6747cdab9f8"
            ],
            "version": "==0.0.9"
        },
        "logstash-formatter": {
            "hashes": [
                "sha256:9d9bff5a03a303e8af0a06718c02e4d93c683bb122ab40db712f87e4d0c1defd",
                "sha256:f5495d1bc17face9ee1b5f6e07e20f203e4ebc5535ec3abead6c0b60818cc6ea"
            ],
            "version": "==0.5.16"
        },
        "python-logstash": {
            "hashes": [
                "sha256:10943e5df83f592b4d61b63ad1afff855ccc8c9467f78718f0a59809ba1fe68c"
            ],
            "version": "==0.4.6"
        },
        "sanic": {
            "hashes": [
                "sha256:18a3bd729093ac93a245849c44045c505a11e6d36da5bf231cb986bfb1e3c14c",
                "sha256:22b1a6f1dc55db8a136335cb0961afa95040ca78aa8c78425a40d91e8618e60e"
            ],
            "version": "==0.7.0"
        },
        "sanic-auth": {
            "hashes": [
                "sha256:6431c2dc06995f2da5ecf0c2fc23208bb0afd5ae0048c060537a79e0bb2a255f",
                "sha256:b7cb9e93296c035ada0aa1ebfb33f9f7b62f7774c519e374b7fe703ff73589cb"
            ],
            "version": "==0.2.0"
        },
        "smbus2": {
            "hashes": [
                "sha256:f0b17997e5a0886ca0d5b0d91d0452de68c282f31bdda8aaae1ca80d1531b2f0"
            ],
            "version": "==0.2.0"
        },
        "ujson": {
            "hashes": [
                "sha256:f66073e5506e91d204ab0c614a148d5aa938bdbf104751be66f8ad7a222f5f86"
            ],
            "version": "==1.35"
        },
        "uvloop": {
            "hashes": [
                "sha256:01cf7199728867f406ba5af78cc47c80acd663ccc52cae105e737a997f1b2bca",
                "sha256:e7c871ba3edd5fcf2afb756de88a9a65245070161e24f75abe79c0a241bb8c76",
                "sha256:b057ef2b0d0162c1ef257f43a95f59bfec37ee9a75cc5412d6b7f9ac6d1d69cb",
                "sha256:89c3bfaad77625490c42a6b99af1879234767ab0c31dd193486a909506e5e549",
                "sha256:68574150720a380509a3409bf2941be0199cfdacff144a97502fb29c250ba927",
                "sha256:7fba5f390db607b2f026bc598df6b2a2a9e062bffe82910b5ffe2b88560135e5",
                "sha256:7ee14835a75c72227d3f8a3f370519a3106a6f02e5453f275f16437ebdb92953"
            ],
            "version": "==0.9.1"
        },
        "websockets": {
            "hashes": [
                "sha256:f5192da704535a7cbf76d6e99c1ec4af7e8d1288252bf5a2385d414509ded0cf",
                "sha256:0c31bc832d529dc7583d324eb6c836a4f362032a1902723c112cf57883488d8c",
                "sha256:da7610a017f5343fdf765f4e0eb6fd0dfd08264ca1565212b110836d9367fc9c",
                "sha256:fd81af8cf3e69f9a97f3a6c0623a0527de0f922c2df725f00cd7646d478af632",
                "sha256:3d425ae081fb4ba1eef9ecf30472ffd79f8e868297ccc7a47993c96dbf2a819c",
                "sha256:ebdd4f18fe7e3bea9bd3bf446b0f4117739478caa2c76e4f0fb72cc45b03cbd7",
                "sha256:3859ca16c229ddb0fa21c5090e4efcb037c08ce69b0c1dfed6122c3f98cd0c22",
                "sha256:d1a0572b6edb22c9208e3e5381064e09d287d2a915f90233fef994ee7a14a935",
                "sha256:80188abdadd23edaaea05ce761dc9a2e1df31a74a0533967f0dcd9560c85add0",
                "sha256:fecf51c13195c416c22422353b306dddb9c752e4b80b21e0fa1fccbe38246677",
                "sha256:367ff945bc0950ad9634591e2afe50bf2222bc4fad1088a386c4bb700888026e",
                "sha256:6df87698022aef2596bffdfecc96d656db59c8d719708c8a471daa815ee61656",
                "sha256:341824d8c9ad53fc43cca3fa9407f294125fa258592f7676640396501448e57e",
                "sha256:64896a6b3368c959b8096b655e46f03dfa65b96745249f374bd6a35705cc3489",
                "sha256:1f3e5a52cab6daa3d432c7b0de0a14109be39d2bfaad033ee5de4a3d3e11dcdf",
                "sha256:da4d4fbe059b0453e726d6d993760065d69b823a27efc3040402a6fcfe6a1ed9"
            ],
            "version": "==4.0.1"
        }
    },
    "develop": {}
}

And the limited case (just targeting smbus2 and python-logstash)

{
    "_meta": {
        "hash": {
            "sha256": "d7f46acb7b927885605beef4262ae8571b085b35ae71db0da094e180d358d3ae"
        },
        "host-environment-markers": {
            "implementation_name": "cpython",
            "implementation_version": "3.6.3",
            "os_name": "posix",
            "platform_machine": "armv7l",
            "platform_python_implementation": "CPython",
            "platform_release": "4.9.59-v7+",
            "platform_system": "Linux",
            "platform_version": "#1047 SMP Sun Oct 29 12:19:23 GMT 2017",
            "python_full_version": "3.6.3",
            "python_version": "3.6",
            "sys_platform": "linux"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.6"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "python-logstash": {
            "hashes": [
                "sha256:10943e5df83f592b4d61b63ad1afff855ccc8c9467f78718f0a59809ba1fe68c"
            ],
            "version": "==0.4.6"
        },
        "smbus2": {
            "hashes": [
                "sha256:f0b17997e5a0886ca0d5b0d91d0452de68c282f31bdda8aaae1ca80d1531b2f0"
            ],
            "version": "==0.2.0"
        }
    },
    "develop": {}
}

Fixing the hash in the lockfile does result in correct behavior, so it seems there is a problem generating this hash?

[nateprewitt]

Ok, I didn't catch this before. You're using a wheel distributed by a 3rd party packaging server. I don't know if we have support for that right now because we only have a way to retrieve hashes from PyPI.

Is there something specific about the 3rd party wheel you need for the Raspberry Pi, or would the standard distribution work?

[jmurrayufo]

Forgive my ignorance here, but I am just using what gets defaultly setup? At no point have I attempted to point anything my toolchain anywhere different. Is there a way to ask pipenv to install from the default packaging servers?

[nateprewitt]

No problem, there's a lot of machinery working in the background of both pip and pipenv.

If you look at the original log you provided, you'll see the line:
Using cached https://www.piwheels.hostedpi.com/simple/python-logstash/python_logstash-0.4.6-py3-none-any.whl

This isn't one of the official pypi servers, so perhaps Raspberry Pi is adding this with their python build? Either way, there's a few ways to get around this.

The first is to try clearing the cache with pipenv lock --clear. If that doesn't work then pip seems to be insisting on using the third party server as the primary. I'm not sure why they'd choose to do that but it's probably something we shouldn't touch further than trying to clear.

If the above doesn't work, you can either use pipenv run pip install -i https://pypi.python.org/simple python-logstash to install from the primary pypi repo, or a less attractive method of pipenv install --skip-lock which will bypass the locking portion. This will be somewhat slower than using the lockfile but will bypass the current error when fetching from a 3rd party repo.

[jmurrayufo]

Trying the suggestions now.

pipenv lock --clear doesn't change the behavior. For some reason pipenv still tries to pull from that third party server. I'm curious to know where this server comes from as well, as I have no clue where it is being added from. My python 3.6.3 install was compiled from source on the device (HA! 6 hours to run PGO, that was fun...), which was then used to install pip + pipenv. Maybe my distro does mix it it's own repos for pip, but I'm not sure where to even start looking for that?

Using pipenv install --skip-lock seems to be working. And is actually going faster than normal for me? Even on a fast install, without this option it takes 3-5 minutes or so to finish the install (And pipenv reports 12 installing packages), but with --skip-lock A fresh install happens in ~15 seconds.

For my own purposes, I'm happy with just using this to deploy to my set of Pis. I would love to have a similar flag for pipenv update? Still, it's not the worst thing in the world to run a pipenv uninstall --all;pipenv install --skip-lock

Thank you so much for the support in getting me running again!

[alexsavio]

The raspbian distribution adds a '/etc/pip.conf' file with an extra source, piwheels: piwheels/piwheels#47
For now the only workaround I've found has been the --skip-lock option and removing the PyPI source from the Pipfile.