New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Attempting to install pytest under Python 2.7.15 and pipenv==2018.11.14 fails to complete with a claimed hash mismatch #3289

Closed
kfkitsune opened this Issue Nov 23, 2018 · 4 comments

Comments

Projects
None yet
2 participants
@kfkitsune

kfkitsune commented Nov 23, 2018

Issue description

Attempting to install pytest under Python 2.7.15 and pipenv==2018.11.14 fails to complete with a claimed hash mismatch.

Expected result

pytest should successfully install, as it did under pipenv==2018.10.13 (build log: https://gitlab.com/kfkitsune/pyHIBP/-/jobs/124421196 ).

Actual result

Full verbose output of a minimized test scenario can be found in the following build log, testing only the issue at hand: https://gitlab.com/kfkitsune/pyHIBP/-/jobs/124421197

Exception information:
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/resolve.py", line 109, in resolve
    raise hash_errors
HashErrors: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    scandir==1.9.0 from https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz#sha256=44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064 (from -r /tmp/pipenv-7LZzKV-requirements/pipenv-KmrdXf-requirement.txt (line 1)):
        Expected sha256 f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd
             Got        44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/pipenv/cli/command.py", line 249, in install
    editable_packages=state.installstate.editables,
  File "/usr/lib/python2.7/site-packages/pipenv/core.py", line 1872, in do_install
    keep_outdated=keep_outdated
  File "/usr/lib/python2.7/site-packages/pipenv/core.py", line 1232, in do_init
    pypi_mirror=pypi_mirror,
  File "/usr/lib/python2.7/site-packages/pipenv/core.py", line 841, in do_install_dependencies
    retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
  File "/usr/lib/python2.7/site-packages/pipenv/core.py", line 748, in batch_install
    _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
  File "/usr/lib/python2.7/site-packages/pipenv/core.py", line 676, in _cleanup_procs
    raise exceptions.InstallError(c.dep.name, extra=err_lines)
pipenv.exceptions.InstallError: ERROR: ERROR: Package installation failed...

Steps to replicate

  1. Using Python 2.7.15
  2. Install pipenv==2018.11.14.
  3. Create a Pipfile with the following:
[[source]]

verify_ssl = true
url = "https://pypi.python.org/simple"
name = "pypi"


[packages]

pytest = "*"


[dev-packages]
  1. Execute: pipenv --verbose install --dev

$ pipenv --support

Pipenv version: '2018.11.14'

Pipenv location: '/usr/lib/python2.7/site-packages/pipenv'

Python location: '/usr/bin/python2'

Python installations found:

  • 2.7.15: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.14.48-coreos-r2',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Jun 14 08:23:03 UTC 2018',
 'python_full_version': '2.7.15',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

  • CI_COMMIT_BEFORE_SHA
  • CI_CONFIG_PATH
  • CI_PROJECT_NAMESPACE
  • CI_BUILD_REF_SLUG
  • CI_SERVER_VERSION_PATCH
  • CI_SERVER_NAME
  • CI_PIPELINE_URL
  • CI_PROJECT_PATH_SLUG
  • CI_COMMIT_MESSAGE
  • CI_COMMIT_REF_SLUG
  • CI_PROJECT_VISIBILITY
  • CI_REGISTRY_IMAGE
  • CI_COMMIT_REF_NAME
  • CI_COMMIT_SHA
  • PATH
  • HOME
  • CI_REGISTRY_PASSWORD
  • CI_REPOSITORY_URL
  • GITLAB_USER_EMAIL
  • CI_DISPOSABLE_ENVIRONMENT
  • CI_RUNNER_ID
  • CI_COMMIT_TITLE
  • GITLAB_CI
  • CI_SERVER_TLS_CA_FILE
  • CI_RUNNER_EXECUTABLE_ARCH
  • CI_RUNNER_REVISION
  • SHLVL
  • CI_BUILD_BEFORE_SHA
  • CI_BUILD_REF_NAME
  • GITLAB_USER_NAME
  • CI_BUILD_ID
  • PYTHONFINDER_IGNORE_UNSUPPORTED
  • CI_RUNNER_VERSION
  • GITLAB_USER_LOGIN
  • CI_PROJECT_PATH
  • DOCKER_DRIVER
  • CI_PROJECT_URL
  • CI_JOB_STAGE
  • CI_PROJECT_DIR
  • CI_BUILD_STAGE
  • CI
  • GITLAB_USER_ID
  • CI_BUILD_REF
  • CI_REGISTRY_USER
  • CI_SERVER_VERSION
  • CI_JOB_NAME
  • CI_BUILD_TOKEN
  • CI_PROJECT_NAME
  • CI_RUNNER_TAGS
  • CI_COMMIT_DESCRIPTION
  • PYTHONDONTWRITEBYTECODE
  • CI_REGISTRY
  • PIP_PYTHON_PATH
  • CI_JOB_TOKEN
  • CI_PROJECT_ID
  • CI_PIPELINE_SOURCE
  • CI_SERVER_VERSION_MAJOR
  • CI_PIPELINE_ID
  • CI_SERVER_VERSION_MINOR
  • CI_SERVER_REVISION
  • PIP_DISABLE_PIP_VERSION_CHECK
  • OLDPWD
  • HOSTNAME
  • CI_JOB_URL
  • CI_NODE_TOTAL
  • PIPENV_CACHE_DIR
  • CI_BUILD_NAME
  • PIP_CACHE_DIR
  • PWD
  • CI_JOB_ID
  • PIP_SHIMS_BASE_MODULE
  • CI_SERVER
  • CI_PIPELINE_IID
  • CI_RUNNER_DESCRIPTION
  • GITLAB_FEATURES
  • FF_K8S_USE_ENTRYPOINT_OVER_COMMAND

Pipenv–specific environment variables:

  • PIPENV_CACHE_DIR: /builds/kfkitsune/pyHIBP/.cache

Debug–specific environment variables:

  • PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • PWD: /builds/kfkitsune/pyHIBP

Contents of Pipfile ('/builds/kfkitsune/pyHIBP/Pipfile'):

[[source]]

verify_ssl = true
url = "https://pypi.python.org/simple"
name = "pypi"


[packages]

pytest = "*"


[dev-packages]

Observation

I did take the initiative to compare the SHA-256 value of scandir==1.9.0 as it is on PyPI, and the SHA-256 hash of the file downloaded locally matches that provided by the SHA-256 link on the download page. So I believe it is reasonable to state that the file itself is not corrupt.

However, I did notice that pipenv is claiming that it is expecting f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd as the SHA-256 hash. Looking at the verbose scan log, I do see the following:

    Skipping link https://files.pythonhosted.org/packages/9e/a5/56b4dec02b16bb720cac9872fccd63b61a815b70633ef15bfe3ea5ce4488/scandir-1.9.0-cp27-cp27m-win32.whl#sha256=f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd (from https://pypi.org/simple/scandir/); it is not compatible with this Python
    [... lines snipped ...]
    Found link https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz#sha256=44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064 (from https://pypi.org/simple/scandir/), version: 1.9.0
  Using version 1.9.0 (newest of versions: 1.9.0)
  Created temporary directory: /tmp/pip-unpack-BLHSt5
  Looking up "https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz" in the cache
  Current age based on date: 204440
  Ignoring unknown cache-control directive: immutable
  Freshness lifetime from max-age: 365000000
  The response is "fresh", returning cached response
  365000000 > 204440
  Using cached https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz
  Downloading from URL https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz#sha256=44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064 (from https://pypi.org/simple/scandir/)

So from the above, we can see that the source tarball is selected for download, and that the claimed SHA-256 hash for the tarball is 44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064, which is what pipenv calculated. However, it was expecting f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd, which corresponds to the file scandir-1.9.0-cp27-cp27m-win32.whl. As such, while I cannot speak definitively as I have not dug into the source, what I believe is happening is that pipenv incorrectly selected the hash to compare, while using the correct file for calculation (such as, select the first scandir==1.9.0 file link, and use that SHA hash as the expected hash). Again, I don't know if this is actually what is happening, but based on the way the error presents itself, that is my best speculation.

@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 23, 2018

Is the build running on windows? If the posix style paths are anything to go by, the resolver is correct to skip the win32 wheel. I can't replicate this in the master branch, but I also can't replicate it with the 2018.11.14 version using the command you provided with the pipfile you provided. I'm thinking we have this fixed now due to a number of other issues, is there any way you can try with the master branch?

That would be pip install -e git+https://github.com/pypa/pipenv.git@master#egg=pipenv at the start of your CI build

@kfkitsune

This comment has been minimized.

kfkitsune commented Nov 23, 2018

Negative, the build is running on Docker with the alpine:latest image, so the build environment would be Linux. For the rebuild you requested, specifically docker image sha256:196d12cf6ab19273823e700516e98eb1910b03b17840f9d5509f03858484d321 for alpine:latest, according to the CI log.

For clarity, the only reason I even brought up the win32 wheel link was because I noticed that the expected hash was referred to by the link's sha256 parameter, while what was actually downloaded was the source tarball, which had its own hash in the URL. It seemed a little more than mere coincidence, so I felt it best to mention it, in the event it helped to narrow down what might be occurring.

As per your request, I reran the CI build, substituting for pipenv/master with the command you provided, and the build failed. Full build log: https://gitlab.com/kfkitsune/pyHIBP/-/jobs/124465524

Output from pipenv --support:

$ pipenv --support

Pipenv version: '2018.11.15.dev0'

Pipenv location: '/builds/kfkitsune/pyHIBP/src/pipenv/pipenv'

Python location: '/usr/bin/python2'

Python installations found:

  • 2.7.15: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.14.48-coreos-r2',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP Thu Jun 14 08:23:03 UTC 2018',
 'python_full_version': '2.7.15',
 'python_version': '2.7',
 'sys_platform': 'linux2'}

System environment variables:

  • CI_COMMIT_BEFORE_SHA
  • CI_CONFIG_PATH
  • CI_PROJECT_NAMESPACE
  • CI_BUILD_REF_SLUG
  • CI_SERVER_VERSION_PATCH
  • CI_SERVER_NAME
  • CI_PIPELINE_URL
  • CI_PROJECT_PATH_SLUG
  • CI_COMMIT_MESSAGE
  • CI_COMMIT_REF_SLUG
  • CI_PROJECT_VISIBILITY
  • CI_REGISTRY_IMAGE
  • CI_COMMIT_REF_NAME
  • CI_COMMIT_SHA
  • PATH
  • HOME
  • CI_REGISTRY_PASSWORD
  • CI_REPOSITORY_URL
  • GITLAB_USER_EMAIL
  • CI_DISPOSABLE_ENVIRONMENT
  • CI_RUNNER_ID
  • CI_COMMIT_TITLE
  • GITLAB_CI
  • CI_SERVER_TLS_CA_FILE
  • CI_RUNNER_EXECUTABLE_ARCH
  • CI_RUNNER_REVISION
  • SHLVL
  • CI_BUILD_BEFORE_SHA
  • CI_BUILD_REF_NAME
  • GITLAB_USER_NAME
  • CI_BUILD_ID
  • PYTHONFINDER_IGNORE_UNSUPPORTED
  • CI_RUNNER_VERSION
  • GITLAB_USER_LOGIN
  • CI_PROJECT_PATH
  • DOCKER_DRIVER
  • CI_PROJECT_URL
  • CI_JOB_STAGE
  • CI_PROJECT_DIR
  • CI_BUILD_STAGE
  • CI
  • GITLAB_USER_ID
  • CI_BUILD_REF
  • CI_REGISTRY_USER
  • CI_SERVER_VERSION
  • CI_JOB_NAME
  • CI_BUILD_TOKEN
  • CI_PROJECT_NAME
  • CI_RUNNER_TAGS
  • CI_COMMIT_DESCRIPTION
  • PYTHONDONTWRITEBYTECODE
  • CI_REGISTRY
  • PIP_PYTHON_PATH
  • CI_JOB_TOKEN
  • CI_PROJECT_ID
  • CI_PIPELINE_SOURCE
  • CI_SERVER_VERSION_MAJOR
  • CI_PIPELINE_ID
  • CI_SERVER_VERSION_MINOR
  • CI_SERVER_REVISION
  • PIP_DISABLE_PIP_VERSION_CHECK
  • OLDPWD
  • HOSTNAME
  • CI_JOB_URL
  • CI_NODE_TOTAL
  • PIPENV_CACHE_DIR
  • CI_BUILD_NAME
  • PIP_CACHE_DIR
  • PWD
  • CI_JOB_ID
  • PIP_SHIMS_BASE_MODULE
  • CI_SERVER
  • CI_PIPELINE_IID
  • CI_RUNNER_DESCRIPTION
  • GITLAB_FEATURES
  • FF_K8S_USE_ENTRYPOINT_OVER_COMMAND

Pipenv–specific environment variables:

  • PIPENV_CACHE_DIR: /builds/kfkitsune/pyHIBP/.cache

Debug–specific environment variables:

  • PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
  • PWD: /builds/kfkitsune/pyHIBP

Contents of Pipfile ('/builds/kfkitsune/pyHIBP/Pipfile'):

[[source]]

verify_ssl = true
url = "https://pypi.python.org/simple"
name = "pypi"


[packages]

pytest = "*"


[dev-packages]

The failure mode looks to be the same, with pipenv expecting the f5c71e29... hash, but getting the 44975e20... hash (the full --verbose output is in the build log, of course):

Exception information:
Traceback (most recent call last):
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/cli/base_command.py", line 143, in main
    status = self.run(options, args)
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/commands/install.py", line 318, in run
    resolver.resolve(requirement_set)
  File "/root/.local/share/virtualenvs/pyHIBP-JuzWPqk-/lib/python2.7/site-packages/pip/_internal/resolve.py", line 109, in resolve
    raise hash_errors
HashErrors: THESE PACKAGES DO NOT MATCH THE HASHES FROM THE REQUIREMENTS FILE. If you have updated the package versions, please update the hashes. Otherwise, examine the package contents carefully; someone may have tampered with them.
    scandir==1.9.0 from https://files.pythonhosted.org/packages/16/2a/557af1181e6b4e30254d5a6163b18f5053791ca66e251e77ab08887e8fe3/scandir-1.9.0.tar.gz#sha256=44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064 (from -r /tmp/pipenv-jz3KO_-requirements/pipenv-6DTzfE-requirement.txt (line 1)):
        Expected sha256 f5c71e29b4e2af7ccdc03a020c626ede51da471173b4a6ad1e904f2b2e04b4bd
             Got        44975e209c4827fc18a3486f257154d34ec6eaec0f90fef0cca1caa482db7064
Traceback (most recent call last):
  File "/usr/bin/pipenv", line 11, in <module>
    load_entry_point('pipenv', 'console_scripts', 'pipenv')()
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/vendor/click/decorators.py", line 17, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/cli/command.py", line 250, in install
    editable_packages=state.installstate.editables,
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/core.py", line 1887, in do_install
    keep_outdated=keep_outdated
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/core.py", line 1246, in do_init
    pypi_mirror=pypi_mirror,
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/core.py", line 852, in do_install_dependencies
    retry_list, procs, failed_deps_queue, requirements_dir, **install_kwargs
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/core.py", line 756, in batch_install
    _cleanup_procs(procs, not blocking, failed_deps_queue, retry=retry)
  File "/builds/kfkitsune/pyHIBP/src/pipenv/pipenv/core.py", line 681, in _cleanup_procs
    raise exceptions.InstallError(c.dep.name, extra=err_lines)
pipenv.exceptions.InstallError: ERROR: ERROR: Package installation failed...
Running after script...
$ date
Fri Nov 23 08:20:50 UTC 2018
@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 23, 2018

Wow. This one is a head scratcher. Thanks for the build logs, that saved a lot of pointless back and forth. I’ll just make an alpine container and see what I can do

techalchemy added a commit that referenced this issue Nov 23, 2018

Fix hash caching race condition
- Clean up more unused code
- Fixes #3289

Signed-off-by: Dan Ryan <dan@danryan.co>
@techalchemy

This comment has been minimized.

Member

techalchemy commented Nov 23, 2018

That was a fun one! I think i got it sorted out, sorry for the trouble!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment