Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

Closed
AdrianoKF opened this issue May 28, 2020 · 3 comments · Fixed by #4302
Closed

Exception when installing Git VCS dependency with plus symbol in HTTP basic auth #4271

AdrianoKF opened this issue May 28, 2020 · 3 comments · Fixed by #4302
Labels
Priority: High Type: Bug 🐛 Type: Vendored Dependencies

Comments

@AdrianoKF
Copy link

@AdrianoKF AdrianoKF commented May 28, 2020

Issue description

Running pipenv install with a Git VCS dependency that has a plus symbol in the HTTP basic auth part of the URL produces an InstallationError due to a failed HTTP basic authentication.

It seems the URL is internally rewritten incorrectly, since the output logs the URL with the plus symbol replaced by a space (%20).

The behaviour was not observed in pipenv==2018.11.26, but occurs in latest pipenv==2020.5.28.

Expected result

Pipenv should install the VCS dependency without errors.

Actual result

pipenv install raises an InstallationError. Note the Git clone URL in the error message: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git', which contains a space character instead of the plus sign:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…

⠋
Building requirements...
 Locking...
Resolving dependencies...
�
⠙ Locking...�
⠹ Locking...�
⠸ Locking...�
⠼ Locking...�
⠴ Locking...�
⠦ Locking...�
⠧ Locking...�
⠇ Locking...�
⠏ Locking...�
⠋ Locking...�
⠙ Locking...ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
�
⠹ Locking...�✘ Locking Failed! 
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.
ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.

Steps to replicate

Command: pipenv install -d

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"

$ pipenv --support

Pipenv version: '2020.5.28'

Pipenv location: '/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv'

Python location: '/home/adriano/.virtualenvs/pipenv2020/bin/python'

Python installations found:

  • 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
  • 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
  • 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
  • 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
  • 3.8.2: /usr/bin/python3
  • 3.8.2: /usr/bin/python3.8
  • 3.8.2: /bin/python3
  • 3.8.2: /bin/python3.8
  • 3.7.7: /home/adriano/.pyenv/versions/3.7.7/bin/python3
  • 2.7.18: /usr/bin/python2.7
  • 2.7.18: /usr/bin/python2
  • 2.7.18: /bin/python2.7
  • 2.7.18: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.6.12-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Sun May 10 14:36:43 UTC 2020',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

  • SHELL
  • LSCOLORS
  • WINDOWID
  • PYENV_HOOK_PATH
  • COLORTERM
  • PYENV_SHELL
  • CSF_MDTVTexturesDirectory
  • LESS
  • XDG_SESSION_PATH
  • CSF_DrawPluginDefaults
  • I3SOCK
  • LC_ADDRESS
  • CSF_LANGUAGE
  • SSH_AUTH_SOCK
  • CSF_MIGRATION_TYPES
  • DESKTOP_SESSION
  • LC_MONETARY
  • SSH_AGENT_PID
  • CSF_OCCTResourcePath
  • CSF_STEPDefaults
  • CLOUDSDK_PYTHON_ARGS
  • PYENV_VERSION
  • GTK_MODULES
  • XDG_SEAT
  • PWD
  • XDG_SESSION_DESKTOP
  • LOGNAME
  • XDG_SESSION_TYPE
  • DRAWHOME
  • XAUTHORITY
  • CSF_StandardLiteDefaults
  • XDG_GREETER_DATA_DIR
  • VIRTUALENVWRAPPER_SCRIPT
  • HOME
  • LC_PAPER
  • LANG
  • LS_COLORS
  • XDG_CURRENT_DESKTOP
  • VIRTUAL_ENV
  • VTE_VERSION
  • CLOUDSDK_ROOT_DIR
  • VIRTUALENVWRAPPER_WORKON_CD
  • XDG_SEAT_PATH
  • CSF_ShadersDirectory
  • CSF_EXCEPTION_PROMPT
  • PYENV_DIR
  • CSF_XmlOcafResource
  • CLOUDSDK_PYTHON
  • WORKON_HOME
  • CSF_SHMessage
  • XDG_SESSION_CLASS
  • TERM
  • ZSH
  • GOOGLE_CLOUD_SDK_HOME
  • USER
  • CSF_StandardDefaults
  • CSF_IGESDefaults
  • VIRTUALENVWRAPPER_PROJECT_FILENAME
  • DISPLAY
  • CSF_XCAFDefaults
  • SHLVL
  • PAGER
  • LC_MESSAGES
  • LC_MEASUREMENT
  • XDG_VTNR
  • CSF_PluginDefaults
  • CSF_TObjMessage
  • XDG_SESSION_ID
  • CASROOT
  • XDG_RUNTIME_DIR
  • PYENV_ROOT
  • LC_TIME
  • CSF_XSMessage
  • MMGT_CLEAR
  • PATH
  • CSF_TObjDefaults
  • VIRTUALENVWRAPPER_HOOK_DIR
  • GDMSESSION
  • DBUS_SESSION_BUS_ADDRESS
  • MAIL
  • DRAWDEFAULT
  • OLDPWD
  • PIP_DISABLE_PIP_VERSION_CHECK
  • PYTHONDONTWRITEBYTECODE
  • PIP_SHIMS_BASE_MODULE
  • PIP_PYTHON_PATH
  • PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

  • PATH: /usr/share/pyenv/libexec:/home/adriano/.virtualenvs/pipenv2020/bin:/home/adriano/.yarn/bin:/home/adriano/.yarn/bin:/home/adriano/.config/yarn/global/node_modules/.bin:/home/adriano/.pyenv/shims:/home/adriano/.pyenv/bin:/home/adriano/.local/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/adriano/.local/bin
  • SHELL: /usr/bin/zsh
  • LANG: en_US.UTF-8
  • PWD: /home/adriano/playground/pipenv-2020-url-sanitation
  • VIRTUAL_ENV: /home/adriano/.virtualenvs/pipenv2020

Contents of Pipfile ('/home/adriano/playground/pipenv-2020-url-sanitation/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"
@zpsjs
Copy link

@zpsjs zpsjs commented May 28, 2020

Got the same issue with the actual version:

ERROR:pip.subprocessor:Command errored out with exit status 128: command: git clone -q 'https://****@github.com/...' /tmp/reqlib-srcfjz5vsf1/... cwd: None Complete output (1 lines): fatal: could not read Password for 'https://${ACCESS_TOKEN}@github.com': No such device or address

Pipenv is asking for password although I'm using a token.

@AdrianoKF
Copy link
Author

@AdrianoKF AdrianoKF commented May 28, 2020

After some digging, it seems that pipenv.vendor.requirementslib.models.url.URI doesn't handle this edge case correctly and replaces + with a space in the URL when converting the parsed URI to a str (note, that username is correctly populated in the returned URI instance but broken in the returned string):

>>> from pipenv.vendor.requirementslib.models.url import URI
>>> url = 'git+https://foo+bar:baz@example.org'
>>> URI.parse(url)
URI(host='example.org', scheme='git+https', port=None, path=None, query='', fragment=None, subdirectory=None, ref='', username='foo+bar', query_dict=omdict([('', '')]), name='', extras=(), is_direct_url=False, is_implicit_ssh=False, _fragment_dict={}, _username_is_quoted=True, _password_is_quoted=False)
>>> print(URI.parse(url))
git+https://foo bar:----@example.org

RFC 3986 specifies that the userinfo subcomponent of a URI may contain the sub-delim tokens, so it is okay to include a literal plus sign (not percent-encoded) there, I believe (https://tools.ietf.org/html/rfc3986#section-3.2.1). requirementslib seems to incorrectly assume that the field is always percent-encoded, which is why the + is replaced with a space in URI.get_username().

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 28, 2020

Thanks for the report, and for the debugging, and sorry for the issue!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Priority: High Type: Bug 🐛 Type: Vendored Dependencies
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants