Pipenv fails to install from hg+https at a given revision

[eolo999]

Issue description

Trying to install a package from hg+https fails with AttributeError: 'Mercurial' object has no attribute 'get_revision_sha'

Steps to replicate

cd /tmp
mkdir pipenv_hg_fail
cd pipenv_hg_fail
pipenv install -e "hg+https://www.mercurial-scm.org/repo/hg/@stable#egg=hg"

results in the following stacktrace:

Traceback (most recent call last):
File "/home/xxx/.local/bin/pipenv", line 11, in
sys.exit(cli())
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 764, in call
return self.main(*args, **kwargs)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 64, in new_func
return ctx.invoke(f, obj, *args, **kwargs)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/cli/command.py", line 249, in install
editable_packages=state.installstate.editables,
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/core.py", line 1976, in do_install
skip_lock=skip_lock,
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/core.py", line 1274, in do_init
pypi_mirror=pypi_mirror,
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/core.py", line 1058, in do_lock
dev=settings["dev"],
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1154, in get_vcs_deps
with locked_repository(requirement) as repo:
File "/home/xxx/.pyenv/versions/3.6.4/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/utils.py", line 1288, in locked_repository
with requirement.req.locked_vcs_repo(src_dir=src_dir) as repo:
File "/home/xxx/.pyenv/versions/3.6.4/lib/python3.6/contextlib.py", line 81, in enter
return next(self.gen)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 621, in locked_vcs_repo
vcsrepo = self.get_vcs_repo(src_dir=src_dir)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/requirements.py", line 599, in get_vcs_repo
vcsrepo.obtain()
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 37, in obtain
self.checkout_ref(self.ref)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 45, in checkout_ref
self.checkout_directory, self.get_commit_hash(ref)
File "/home/xxx/.local/lib/python3.6/site-packages/pipenv/vendor/requirementslib/models/vcs.py", line 63, in get_commit_hash
return self.repo_instance.get_revision_sha(self.checkout_directory, target_ref.arg_rev)
AttributeError: 'Mercurial' object has no attribute 'get_revision_sha'

---

$ pipenv --support

Pipenv version: '2018.10.13'

Pipenv location: '/home/xxx/.local/lib/python3.6/site-packages/pipenv'

Python location: '/home/xxx/.pyenv/versions/3.6.4/bin/python3.6'

Python installations found:

• 3.6.6: /usr/bin/python3.6
• 3.6.6: /usr/bin/python3.6m
• 3.6.2: /usr/local/bin/python3.6
• 3.6.2: /usr/local/bin/python3.6m
• 2.7.15rc1: /usr/bin/python2.7

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '3.6.4',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '4.15.0-36-generic',
 'platform_system': 'Linux',
 'platform_version': '#39-Ubuntu SMP Mon Sep 24 16:19:09 UTC 2018',
 'python_full_version': '3.6.4',
 'python_version': '3.6',
 'sys_platform': 'linux'}

System environment variables:

• XDG_CONFIG_DIRS
• LC_TELEPHONE
• LANG
• DISPLAY
• SHLVL
• LOGNAME
• XDG_VTNR
• MANDATORY_PATH
• LC_NAME
• PWD
• XAUTHORITY
• QT_QPA_PLATFORMTHEME
• GTK_IM_MODULE
• COLORTERM
• XDG_SESSION_ID
• GTK2_MODULES
• GNOME_TERMINAL_SCREEN
• DESKTOP_SESSION
• XDG_SESSION_DESKTOP
• GDMSESSION
• USERNAME
• TEXTDOMAINDIR
• WINDOWPATH
• LC_MEASUREMENT
• LC_PAPER
• LC_NUMERIC
• LC_MONETARY
• IM_CONFIG_PHASE
• DEFAULTS_PATH
• DBUS_SESSION_BUS_ADDRESS
• VTE_VERSION
• CLUTTER_IM_MODULE
• LC_IDENTIFICATION
• TEXTDOMAIN
• QT4_IM_MODULE
• GNOME_TERMINAL_SERVICE
• XDG_DATA_DIRS
• S_COLORS
• SHELL
• XDG_SESSION_TYPE
• QT_IM_MODULE
• LC_TIME
• TERM
• SSH_AGENT_PID
• GTK_MODULES
• XDG_CURRENT_DESKTOP
• PATH
• SSH_AUTH_SOCK
• LIBVIRT_DEFAULT_URI
• HOME
• XDG_SEAT
• XMODIFIERS
• XDG_RUNTIME_DIR
• LC_ADDRESS
• GPG_AGENT_INFO
• USER
• QT_ACCESSIBILITY
• OLDPWD
• LS_COLORS
• PYTHONSTARTUP
• DEBFULLNAME
• DEBEMAIL
• LESS
• MANPAGER
• EDITOR
• GIT_EDITOR
• NOSE_REDNOSE
• N_PREFIX
• GOROOT
• GOPATH
• AUTOJUMP_SOURCED
• AUTOJUMP_ERROR_PATH
• _
• PYTHONDONTWRITEBYTECODE
• PIP_SHIMS_BASE_MODULE
• PIP_PYTHON_PATH

Pipenv–specific environment variables:

Debug–specific environment variables:

• PATH: /home/xxx/code/bin:/home/xxx/code/n/n/versions/node/6.10.3/bin:/home/xxx/.local/bin:/home/xxx/bin:/home/xxx/.cargo/bin:/home/xxx/xiki-master/bin:/home/xxx/code/bin:/home/xxx/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/xxx/code/go/bin
• SHELL: /usr/bin/zsh
• EDITOR: /home/xxx/code/bin/vim
• LANG: en_US.UTF-8
• PWD: /tmp/test_pipenv_hg

---

Contents of Pipfile ('/tmp/test_pipenv_hg/Pipfile'):

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
hg = {editable = true, ref = "stable", hg = "https://www.mercurial-scm.org/repo/hg/"}

[dev-packages]

[requires]
python_version = "3.6"

[kr1]

I have the same issue.
It appears to be caused by vcs in requirementslib calling get_revision_sha whereas mercurial in patched/notpip/_internal/vcs has get_revision_hash, one possible workaround is here:

kr1@518176b#diff-e8a9d126ae6283d8364c7fe4853401bdL81

Should this be addressed in requirementslib?

[techalchemy]

Yes most likely, although it’s also possible we are using this wrong somewhere (some of the git API didn’t behave quite how I expected so I wound up just experimenting until o got the results I wanted).

Possibly @cjerdonek can speak to whether this is a change that should be reflected back up to pip or whether we can just fix the call in requirementslib.

[cjerdonek]

Currently, get_revision() is the only method along those lines meant to be implemented in all the VCS-specific subclasses (which is why it's in the VersionControl base class). There are a couple related methods in certain subclasses that are for use only within those subclasses (e.g. get_revision_sha in the Git class and get_revision_hash for Mercurial). So pipenv should probably be using get_revision(). It would be good to know if get_revision() isn't doing what pipenv wants for some VCS class though. Perhaps the method could be adjusted to be more useful.

[techalchemy]

Hmmm my implementation provides for two possibilities but I only seem to use one of them myself in most cases, so by default I actually do just call get_revision(checkout_directory) which may or may not be correct. I have an implementation which relies on get_revision_sha but only if you ask explicitly for a ref lookup--

This lives in requirementslib.models.requirements.VCSRequirement

    def get_vcs_repo(self, src_dir=None):
        from .vcs import VCSRepository
        checkout_dir = self.get_checkout_dir(src_dir=src_dir)
        url = "{0}#egg={1}".format(self.vcs_uri, self.name)
        vcsrepo = VCSRepository(
            url=url,
            name=self.name,
            ref=self.ref if self.ref else None,
            checkout_directory=checkout_dir,
            vcs_type=self.vcs
        )
        if not self.is_local:
            vcsrepo.obtain()
        return vcsrepo

    @contextmanager
    def locked_vcs_repo(self, src_dir=None):
        vcsrepo = self.get_vcs_repo(src_dir=src_dir)
        if self.ref and not self.is_local:
            vcsrepo.checkout_ref(self.ref)
        self.ref = self.get_commit_hash()
        self.req.revision = self.ref

and it creates a new requirementslib.models.VCSRepository instance which will obtain a copy of the repository which is secretly a wrapper around pip's implementation:

VCS_SUPPORT = VcsSupport()


@attr.s
class VCSRepository(object):
...
    @repo_instance.default
    def get_repo_instance(self):
        backend = VCS_SUPPORT._registry.get(self.vcs_type)
        return backend(url=self.url)

    def obtain(self):
        if not os.path.exists(self.checkout_directory):
            self.repo_instance.obtain(self.checkout_directory)
        if self.ref:
            self.checkout_ref(self.ref)
            self.commit_sha = self.get_commit_hash(self.ref)
        else:
            if not self.commit_sha:
                self.commit_sha = self.get_commit_hash()

    def checkout_ref(self, ref):
        if not self.repo_instance.is_commit_id_equal(
            self.checkout_directory, self.get_commit_hash(ref)
        ) and not self.repo_instance.is_commit_id_equal(self.checkout_directory, ref):
            if not self.is_local:
                self.update(ref)

    def get_commit_hash(self, ref=None):
        if ref:
            target_ref = self.repo_instance.make_rev_options(ref)
            return self.repo_instance.get_revision_sha(self.checkout_directory, target_ref.arg_rev)
        return self.repo_instance.get_revision(self.checkout_directory)

[techalchemy]

ah but in update I do use it:

    def update(self, ref):
        target_ref = self.repo_instance.make_rev_options(ref)
        sha = self.repo_instance.get_revision_sha(self.checkout_directory, target_ref.arg_rev)
        if parse_version(pip_version) > parse_version("18.0"):
            self.repo_instance.update(self.checkout_directory, self.url, target_ref)
        else:
            self.repo_instance.update(self.checkout_directory, target_ref)
        self.commit_hash = self.get_commit_hash(ref)

[cjerdonek]

By the way, one observation: in the case of not os.path.exists(self.checkout_directory), you shouldn't ever need to call update() after obtain() because the first obtain() should be putting you in the right state after a fresh clone. (If something's not right about that state, I'd like to know.)

[techalchemy]

Oh perfect. That simplified things considerably