$ pipenv update <git dep> broken

[randycoulman]

With pipenv 11.1.5, trying to update a git dependency is failing with the following error:

> pipenv install --selective-upgrade hats
Installing hats{'git': 'git@git-host:hats', 'editable': True}...
WARNING: Invalid requirement, parse error at ""{'git': ""
ABORTING INSTALL: You will have to reinstall any packages that failed to install
You may have to manually run pipenv lock when you are finished

The dependency in my Pipfile looks like this:

hats = {git = "git@git-host:hats", editable = true}

and was initially created by pipenv install -e git+git@git-host:hats#egg=hats.

pipenv update hats fails with the same error.

pipenv lock and pipenv install both work fine.

I'm pretty sure I remember this working in pipenv 9.1.0; I'm not sure when it broke between then and 11.1.5.

Describe your environment

1. OS Type: Windows 7
2. Python version: Python 3.6.4
3. Pipenv version: pipenv, version 11.1.5

Expected result

The hats dependency should have be successfully updated to the latest version in the git repository.

Actual result

See error above.

Steps to replicate

Because my test case is a private package in a private repo, I've recreated the bug using a public package/repo:

mkdir pipenv_bug
cd pipenv_bug
pipenv --three
pipenv install -e git@https://github.com/pypa/sampleproject#egg=sampleproject
pipenv update sampleproject

[randycoulman]

Update: I tested in several versions of pipenv; this breakage happens in every version I tried back through 10.1.0.

I was unable to test version 10.0.0, because I couldn't find a way to do an update in that version. The update command was gone and the install --selective-upgrade option hadn't been added yet.

Version 9.1.0 worked as I reported above.

[techalchemy]

@randycoulman I believe this has to do with how we handle update -- in the past, update was an aggressive command, as in in wiped your environment and reinstalled from the pipfile no matter what. Now, however, with --selective-upgrade, we have granularity. Because hats is just what you have chosen to name the package, we haven't built the functionality to simply tell pipenv to update hats, you'd still have to pass the whole string I would think.

If you pipenv update git+git@whatever#egg=hats, does that work?

[randycoulman]

@techalchemy No, it doesn't quite work.

First:

> pipenv update git+git@git-host:hats#egg=hats
Installing git+git@dev-git02:hats#egg=hats…
Warning: You installed a VCS dependency in non–editable mode. This will work fine, but sub-dependencies will not be resolved by $ pipenv lock.
  To enable this sub–dependency functionality, specify that this dependency is editable.

Error:  An error occurred while installing git+git@git-host:hats#egg=hats!
Invalid requirement: 'git+git@git-host:hats#egg=hats'
= is not a valid operator. Did you mean == ?

I tried adding a -e flag, but update doesn't take one of those. So I switched over to install --selective-upgrade instead:

> pipenv install -e --selective-upgrade git+git@git-host:hats#egg=hats
Installing -e git+git@git-host:hats#egg=hats…
Obtaining hats from git+git@git-host:hats#egg=hats
  Updating c:\users\rcoulman\.virtualenvs\py_spike-crfvfphy\src\hats clone
Installing collected packages: hats
  Found existing installation: hats 0.3.0
    Uninstalling hats-0.3.0:
      Successfully uninstalled hats-0.3.0
  Running setup.py develop for hats
Successfully installed hats

Adding -e git+git@git-host:hats#egg=hats to Pipfile's [packages]…
Installing dependencies from Pipfile.lock (b3d8cd)…
An error occurred while installing -e git+git@git-host:hats@682cfd84c2b1d92962d0f23d8ba39dbedfae5101#egg=hats! Will try
 again.
  ================================ 5/5 - 00:00:02
Installing initially–failed dependencies…
Obtaining hats from git+git@git-host:hats@682cfd84c2b1d92962d0f23d8ba39dbedfae5101#egg=hats
  Cloning git@git-host:hats@682cfd84c2b1d92962d0f23d8ba39dbedfae5101 to c:\users\rcoulman\.virtualenvs\py_spike-crfvfphy\src\hats
fatal: 'hats@682cfd84c2b1d92962d0f23d8ba39dbedfae5101' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

Command "git clone -q git@git-host:hats@682cfd84c2b1d92962d0f23d8ba39dbedfae5101 C:\Users\rcoulman\.virtualenvs\py_spike-crFVfpHY\src\hats" failed with error code 128 in None

Somewhere along the way a ref= entry got added to my Pipfile.lock. Once that's there, nothing works because it's trying to add the ref to the end of the git URL with an @ sign, which git doesn't understand. If I take out the ref= entry, everything starts working again.

It seems like I shouldn't have to re-specify the repo location for the dependency once it's in my Pipfile. I expected update (or install --selective-upgrade) to be able to figure out where to re-fetch from.

[kennethreitz]

Can you give me an example I can reproduce myself?

[randycoulman]

@kennethreitz My original report has "steps to reproduce" using a public repo on GitHub that you should be able to follow.

Unless you're asking about the latter report where a ref = entry got added to Pipfile.lock. I'm still not sure when that happened or what caused it, so I don't know how to reproduce it. I suspect you could add one manually to the example generated by my "steps to reproduce" to see the git behavior.

[kennethreitz]

ah, i see now, thanks!

[kennethreitz]

'update' is not to be used with git projects, basically.

[kennethreitz]

i think there's a legitimate bug here, but it'd be really hard to fix

[techalchemy]

@randycoulman the ref gets added to the lockfile because essentially it has to, that's the only way to pin a git repository as far as I know (a reproducible lockfile needs a ref to pin a git repo)... update should just update the ref ideally, we are just constructing the urls incorrectly somehow

[randycoulman]

@techalchemy I think it requires a two-step process. First, clone the repo, then checkout the ref. From the error messages, it looks like we're trying to do it all in one step.

[kennethreitz]

fixed :)

[ghost]

This doesn't give an error message anymore but I'm not sure it actually upgrades the package. I think it might reinstall the same version just uninstalled. I'm not sure how to make a shareable example here, but I just tried it a few times and got this behavior: pipenv update my_github_package uninstalls and reinstalls the same version; pipenv uninstall my_github_package followed by pipenv install git+https://github.com/me/mypackage.git#egg=mypackage installs the new version.

[techalchemy]

We have another issue opened on this. It doesn’t update anything yet

[techalchemy]

(Fix will land in 11.11.0)

[ghost]

@techalchemy Linking so interested folks can track: This one, right? #1690

[techalchemy]

@apnewberry that’s the one, thanks