Unable to install package from private repository

[clayk]

I am attempting to install from a private repository. The output indicates the package and dependencies were installed but when pipenv is "Locking [packages] dependencies…"

I receive an error indicating: pip.exceptions.InstallationError: Could not detect requirement name, please specify one with #egg=

Describe your environment

1. macOS 10.13.2
2. Python version: Python 3.6.3
3. Pipenv version: pipenv, version 9.0.3

Expected result

I am expecting for the package to install and the lockfile to be written.

Actual result

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Traceback (most recent call last):
  File "/Users/clayk/Library/Python/3.6/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/cli.py", line 1934, in install
    do_lock(system=system, pre=pre)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/cli.py", line 1102, in do_lock
    pre=pre
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/utils.py", line 545, in resolve_deps
    resolved_tree = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/utils.py", line 470, in actually_resolve_reps
    constraint = pip.req.InstallRequirement.from_editable(dep[len('-e '):])
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/patched/pip/req/req_install.py", line 143, in from_editable
    editable_req, default_vcs)
  File "/Users/clayk/Library/Python/3.6/lib/python/site-packages/pipenv/patched/pip/req/req_install.py", line 1196, in parse_editable
    "Could not detect requirement name, please specify one with #egg="
pip.exceptions.InstallationError: Could not detect requirement name, please specify one with #egg=

Steps to replicate

execute pipenv install -e git+ssh://path/to/private/Redacted.git@v1.1.8#egg=privatepackagename

[guaguanco]

I just started seeing this problem as well, but when I update the package rather than when I'm installing it.

In the Pipfile I have an entry like:

mypkg = {git = "file:///some/path/to/working/dir", editable = true, subdirectory = "inside/workdir"}

when I run pipenv update mypkg the package gets uninstalled and reinstalled correctly, but at the end pipenv tries to add a new entry using just part of the original one, and in the Pipfile I get:

mypkg = {git = "file:///some/path/to/working/dir", editable = true, subdirectory = "inside/workdir"}
"a65c66e" = {path = "git+file:///some/path/to/working/dir", editable = true}

As you can see there's no #egg because the mypkg name is lost, and no subdirectory.

I'm sure it used to work.

Now I have to manually remove that additional line to keep my Pipfile clean, fortunately the update works so I can keep on until this is fixed

[hcoura]

I am also seeing similar problems.

When installing without the -e flag the package is installed, but if I pipenv install afterwards I get:

Installing dependencies from Pipfile.lock (5fd147)…
An error occurred while installing git+git@github.com/my/package/! Will try again.
Installing initially–failed dependencies…
Invalid requirement: 'git+git@github.com/my/package/'
It looks like a path. Does it exist ?

If I install the same package with the -e flag I get:

Locking [dev-packages] dependencies…
Locking [packages] dependencies…
fatal: repository 'git@github.com/my/package.git' does not exist
Traceback (most recent call last):
  File "/home/henrique/miniconda3/bin/pipenv", line 11, in <module>
    sys.exit(cli())
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/vendor/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/cli.py", line 213, in install
    deploy=deploy
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/core.py", line 1795, in do_install
    do_lock(system=system, pre=pre)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/core.py", line 1113, in do_lock
    allow_global=system
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/utils.py", line 340, in resolve_deps
    resolved_tree, resolver = actually_resolve_reps(deps, index_lookup, markers_lookup, project, sources, verbose, clear, pre)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/utils.py", line 301, in actually_resolve_reps
    resolved_tree.update(resolver.resolve(max_rounds=PIPENV_MAX_ROUNDS))
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 200, in _resolve_one_round
    for dep in self._iter_dependencies(best_match):
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/piptools/resolver.py", line 274, in _iter_dependencies
    for dependency in self.repository.get_dependencies(ireq):
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 153, in get_dependencies
    result = reqset._prepare_file(self.finder, ireq)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/req/req_set.py", line 521, in _prepare_file
    req_to_install.update_editable(not self.is_download)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/req/req_install.py", line 587, in update_editable
    vcs_backend.export(self.source_dir)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/vcs/git.py", line 69, in export
    self.unpack(temp_dir)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/vcs/__init__.py", line 286, in unpack
    self.obtain(location)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/vcs/git.py", line 142, in obtain
    self.run_command(['clone', '-q', url, dest])
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/vcs/__init__.py", line 325, in run_command
    spinner)
  File "/home/henrique/miniconda3/lib/python3.6/site-packages/pipenv/patched/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))
pip.exceptions.InstallationError: Command "git clone -q git@github.com/my/package.git /tmp/pip-53zad75t-export" failed with error code 128 in None

[bokhi]

@techalchemy
To install a private github repository from pip the syntax is
pip install git+ssh://git@github.com/username/repo.git@branch#egg=blah

This worked properly with pipenv up to version 9
pipenv install git+ssh://git@github.com/username/repo.git@branch#egg=blah

but with pipenv 11
pipenv install git+ssh://git@github.com/username/repo.git@branch#egg=blah
returns with the an error for
git clone -q git@github.com/username/repo.git /tmp/temp
it's because the correct syntax should be git clone -q git@github.com:username/repo.git /tmp/temp with : and not /

[Moritz90]

@bokhi
You're close to the actual issue, but your proposed fix is incorrect. The problem is not the slash, but the fact that the leading ssh:// is omitted, causing pipenv to prompt for the ssh password for user git in my case.

I'm trying to install a package from a private Bitbucket repository from which you can only clone via SSH, not HTTPS. Pipenv tries to execute git clone -q git@host:7999/project/repo.git instead of git clone -q ssh://git@host:7999/project/repo.git.

[Moritz90]

For some reason, the stack trace is cut off. Here's what I get:

.py", line 162, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 199, in get_legacy_dependencies
    result = reqset._prepare_file(self.finder, ireq)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/req/req_set.py", line 521, in _prepare_file
    req_to_install.update_editable(not self.is_download)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/req/req_install.py", line 587, in update_editable
    vcs_backend.export(self.source_dir)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/vcs/git.py", line 69, in export
    self.unpack(temp_dir)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/vcs/__init__.py", line 286, in unpack
    self.obtain(location)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/vcs/git.py", line 142, in obtain
    self.run_command(['clone', '-q', url, dest])
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/vcs/__init__.py", line 325, in run_command
    spinner)
  File "/home/moritz/.local/lib/python2.7/site-packages/pipenv/patched/pip/utils/__init__.py", line 707, in call_subprocess
    % (command_desc, proc.returncode, cwd))

[techalchemy]

That’s not where the problem is but it is actually really helpful. If you use the git+git@ syntax it should work until I get a fix in place

[Moritz90]

@techalchemy How do you even specify the port with the syntax git+git@host:project/repo.git? I've tried a few variants and each of them results in a similar prompt for git's password (which should not happen). The only difference is that I don't get a stacktrace after entering something incorrect.

[techalchemy]

From https://pip.pypa.io/en/stable/reference/pip_install/#git

Here are the supported forms:
...
-e git+git@git.myproject.org:MyProject#egg=MyProject

Naturally if you use an ssh based approach you will need to have a password or a key pair for authentication. You can also use an http based approach which is not going to be impacted by this bug: git+https://git.myhost.org/myrepo.git#egg=myrepo

Please read the pip documentation if you want to know how to interact with vcs repositories.

[Moritz90]

I read the pip documentation (the exact page you posted) and it doesn't cover how to specify the port when using this type of URI, which you've suggested as a workaround. That's why I asked.

As mentioned above, HTTP access is disabled in our Bitbucket instance. My public key is correctly registered with Bitbucket and normal interaction via git and installation of the package via pip itself work perfectly fine.

But no matter what I try, I cannot get pipenv to install the package. The fact that it's asking for a password for git makes me suspect that it's always connecting to the regular SSH server on port 22, not the Bitbucket SSH server on port 7999.

Sorry if what I wrote before wasn't clear enough.

[techalchemy]

Does simply adding the port like you would to a normal url not work?

[Moritz90]

I've tried the following variations:

• git+git@host:port/project/repo.git#egg=...
• git+git@host:port:project/repo.git#egg=...

Both don't work, since in this URL scheme, everything behind the first colon seems to be interpreted as the path.

[Moritz90]

A bit of additional info:

After the initial installation attempt (pipenv install -e git+ssh://...), the Pipfile contains the following line:

package = {git = "git@host:7999/project/repo.git", editable = true}

The step Locking [packages] dependencies… then immediately fails (i.e. asks for the password).

I tried to manually change the line to the following:

package = {git = "ssh://git@host:7999/project/repo.git", editable = true}

Running pipenv install after that change results in a partial success:

Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Updated Pipfile.lock (28b5f3)!
Installing dependencies from Pipfile.lock (28b5f3)…

This time, it at least installs all dependencies of the package correctly before failing to install the package itself (asking for the password). pipenv run pip freeze shows that the dependencies were indeed installed correctly.

Hope that helps track down the problem.

Edit: Manually adding "ssh://" in Pipfile.lock as well makes pipenv install succeed. I guess that's my workaround for now, then.

[techalchemy]

@Moritz90 thanks for the thorough debug, I'll get a fix together for this which will be much easier because of this info