Exception when installing Git VCS dependency with plus symbol in HTTP basic auth

[AdrianoKF]

Issue description

Running pipenv install with a Git VCS dependency that has a plus symbol in the HTTP basic auth part of the URL produces an InstallationError due to a failed HTTP basic authentication.

It seems the URL is internally rewritten incorrectly, since the output logs the URL with the plus symbol replaced by a space (%20).

The behaviour was not observed in pipenv==2018.11.26, but occurs in latest pipenv==2020.5.28.

Expected result

Pipenv should install the VCS dependency without errors.

Actual result

pipenv install raises an InstallationError. Note the Git clone URL in the error message: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git', which contains a space character instead of the plus sign:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…

⠋
Building requirements...
 Locking...
Resolving dependencies...
�
⠙ Locking...�
⠹ Locking...�
⠸ Locking...�
⠼ Locking...�
⠴ Locking...�
⠦ Locking...�
⠧ Locking...�
⠇ Locking...�
⠏ Locking...�
⠋ Locking...�
⠙ Locking...ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
�
⠹ Locking...�✘ Locking Failed! 
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.
ERROR:pip.subprocessor:Command errored out with exit status 128:
 command: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms
     cwd: None
Complete output (2 lines):
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab.aai.lab/tms/tms-core.git/'
----------------------------------------
Traceback (most recent call last):
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 807, in <module>
    main()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 802, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 785, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 746, in resolve_packages
    results, resolver = resolve(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/resolver.py", line 728, in resolve
    return resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1378, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1090, in actually_resolve_deps
    resolver = Resolver.create(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 647, in create
    constraints, skipped, index_lookup, markers_lookup = cls.get_metadata(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 462, in get_metadata
    constraint_update, lockfile_update = cls.get_deps_from_req(
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 540, in get_deps_from_req
    req_list, lockfile = get_vcs_deps(reqs=[req])
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 1874, in get_vcs_deps
    with temp_path(), locked_repository(requirement) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/utils.py", line 2031, in locked_repository
    with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2194, in locked_vcs_repo
    vcsrepo = self.get_vcs_repo(src_dir=src_dir)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 2153, in get_vcs_repo
    vcsrepo.obtain()
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 77, in obtain
    self.repo_backend.obtain(self.checkout_directory, self.parsed_url)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 527, in obtain
    self.fetch_new(dest, url, rev_options)
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/git.py", line 225, in fetch_new
    self.run_command(make_command('clone', '-q', url, dest))
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/vcs/versioncontrol.py", line 660, in run_command
    return call_subprocess(cmd, show_stdout, cwd,
  File "/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 128: git clone -q 'https://gitlab%20deploy-token-3:****@gitlab.aai.lab/tms/tms-core.git' /tmp/pipenv-cgcmg1hk-src/tms Check the logs for full command output.

Steps to replicate

Command: pipenv install -d

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"

---

$ pipenv --support

Pipenv version: '2020.5.28'

Pipenv location: '/home/adriano/.virtualenvs/pipenv2020/lib/python3.8/site-packages/pipenv'

Python location: '/home/adriano/.virtualenvs/pipenv2020/bin/python'

Python installations found:

• 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
• 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
• 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3
• 3.8.2: /home/adriano/.virtualenvs/pipenv2020/bin/python3.8
• 3.8.2: /usr/bin/python3
• 3.8.2: /usr/bin/python3.8
• 3.8.2: /bin/python3
• 3.8.2: /bin/python3.8
• 3.7.7: /home/adriano/.pyenv/versions/3.7.7/bin/python3
• 2.7.18: /usr/bin/python2.7
• 2.7.18: /usr/bin/python2
• 2.7.18: /bin/python2.7
• 2.7.18: /bin/python2

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.8.2',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '5.6.12-1-MANJARO',
 'platform_system': 'Linux',
 'platform_version': '#1 SMP PREEMPT Sun May 10 14:36:43 UTC 2020',
 'python_full_version': '3.8.2',
 'python_version': '3.8',
 'sys_platform': 'linux'}

System environment variables:

• SHELL
• LSCOLORS
• WINDOWID
• PYENV_HOOK_PATH
• COLORTERM
• PYENV_SHELL
• CSF_MDTVTexturesDirectory
• LESS
• XDG_SESSION_PATH
• CSF_DrawPluginDefaults
• I3SOCK
• LC_ADDRESS
• CSF_LANGUAGE
• SSH_AUTH_SOCK
• CSF_MIGRATION_TYPES
• DESKTOP_SESSION
• LC_MONETARY
• SSH_AGENT_PID
• CSF_OCCTResourcePath
• CSF_STEPDefaults
• CLOUDSDK_PYTHON_ARGS
• PYENV_VERSION
• GTK_MODULES
• XDG_SEAT
• PWD
• XDG_SESSION_DESKTOP
• LOGNAME
• XDG_SESSION_TYPE
• DRAWHOME
• XAUTHORITY
• CSF_StandardLiteDefaults
• XDG_GREETER_DATA_DIR
• VIRTUALENVWRAPPER_SCRIPT
• HOME
• LC_PAPER
• LANG
• LS_COLORS
• XDG_CURRENT_DESKTOP
• VIRTUAL_ENV
• VTE_VERSION
• CLOUDSDK_ROOT_DIR
• VIRTUALENVWRAPPER_WORKON_CD
• XDG_SEAT_PATH
• CSF_ShadersDirectory
• CSF_EXCEPTION_PROMPT
• PYENV_DIR
• CSF_XmlOcafResource
• CLOUDSDK_PYTHON
• WORKON_HOME
• CSF_SHMessage
• XDG_SESSION_CLASS
• TERM
• ZSH
• GOOGLE_CLOUD_SDK_HOME
• USER
• CSF_StandardDefaults
• CSF_IGESDefaults
• VIRTUALENVWRAPPER_PROJECT_FILENAME
• DISPLAY
• CSF_XCAFDefaults
• SHLVL
• PAGER
• LC_MESSAGES
• LC_MEASUREMENT
• XDG_VTNR
• CSF_PluginDefaults
• CSF_TObjMessage
• XDG_SESSION_ID
• CASROOT
• XDG_RUNTIME_DIR
• PYENV_ROOT
• LC_TIME
• CSF_XSMessage
• MMGT_CLEAR
• PATH
• CSF_TObjDefaults
• VIRTUALENVWRAPPER_HOOK_DIR
• GDMSESSION
• DBUS_SESSION_BUS_ADDRESS
• MAIL
• DRAWDEFAULT
• OLDPWD
• PIP_DISABLE_PIP_VERSION_CHECK
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH
• PYTHONFINDER_IGNORE_UNSUPPORTED

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /usr/share/pyenv/libexec:/home/adriano/.virtualenvs/pipenv2020/bin:/home/adriano/.yarn/bin:/home/adriano/.yarn/bin:/home/adriano/.config/yarn/global/node_modules/.bin:/home/adriano/.pyenv/shims:/home/adriano/.pyenv/bin:/home/adriano/.local/bin:/opt/google-cloud-sdk/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl:/home/adriano/.local/bin
• SHELL: /usr/bin/zsh
• LANG: en_US.UTF-8
• PWD: /home/adriano/playground/pipenv-2020-url-sanitation
• VIRTUAL_ENV: /home/adriano/.virtualenvs/pipenv2020

---

Contents of Pipfile ('/home/adriano/playground/pipenv-2020-url-sanitation/Pipfile'):

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
tms = {ref = "develop",git = "https://gitlab+deploy-token-3:nHLsRjx5eCpU6zxmj5sE@gitlab.aai.lab/tms/tms-core.git"}

[packages]

[requires]
python_version = "3.8"

[zpsjs]

Got the same issue with the actual version:

ERROR:pip.subprocessor:Command errored out with exit status 128: command: git clone -q 'https://****@github.com/...' /tmp/reqlib-srcfjz5vsf1/... cwd: None Complete output (1 lines): fatal: could not read Password for 'https://${ACCESS_TOKEN}@github.com': No such device or address

Pipenv is asking for password although I'm using a token.

[AdrianoKF]

After some digging, it seems that pipenv.vendor.requirementslib.models.url.URI doesn't handle this edge case correctly and replaces + with a space in the URL when converting the parsed URI to a str (note, that username is correctly populated in the returned URI instance but broken in the returned string):

>>> from pipenv.vendor.requirementslib.models.url import URI
>>> url = 'git+https://foo+bar:baz@example.org'
>>> URI.parse(url)
URI(host='example.org', scheme='git+https', port=None, path=None, query='', fragment=None, subdirectory=None, ref='', username='foo+bar', query_dict=omdict([('', '')]), name='', extras=(), is_direct_url=False, is_implicit_ssh=False, _fragment_dict={}, _username_is_quoted=True, _password_is_quoted=False)
>>> print(URI.parse(url))
git+https://foo bar:----@example.org

RFC 3986 specifies that the userinfo subcomponent of a URI may contain the sub-delim tokens, so it is okay to include a literal plus sign (not percent-encoded) there, I believe (https://tools.ietf.org/html/rfc3986#section-3.2.1). requirementslib seems to incorrectly assume that the field is always percent-encoded, which is why the + is replaced with a space in URI.get_username().

[techalchemy]

Thanks for the report, and for the debugging, and sorry for the issue!