New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Git+SSH dependencies have subtle (yet critical) differences from git clone #2124

Open
grahamc opened this Issue Nov 13, 2014 · 25 comments

Comments

Projects
None yet
@grahamc

grahamc commented Nov 13, 2014

Reproduction steps:

  1. Have a git repository which is private, and is a pip package.
  2. Have a deploy key which can read the repository.
  3. Confirm you can run git clone git@github.com:account/private-pip-package.git
  4. Google around and discover that the instructions to use a git repo as a pip dependency boil down to very nearly "prefixing the repo with git+ssh:// and suffix it with #egg=private-pip-package."
  5. End up with a pip dependency of git+ssh://git@github.com:account/private-pip-package.git#egg=private-pip
  6. Try installing the package, but get a fatal: Could not read from remote repository. instead.
  7. Run the gamut of tests (ssh git@github.com works, but ssh ssh://git@github.com fails ...)
  8. Realize (after much hair-pulling) that instead of git+ssh://git@github.com:account/private-pip-package.git#egg=private-pip it should be git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip (if you didn't catch that, its github.com/account instead of github.com:account)

I would propose that a simple note in the failure message, or checking on input, or what have you, about the : vs. / would be quite helpful.

@Ivoz

This comment has been minimized.

Member

Ivoz commented Dec 9, 2014

After checking back on the docs, I noticed you can use that form of uri by git+git@github.com:account/repo (the last example documented).

@grahamc

This comment has been minimized.

grahamc commented Dec 9, 2014

Maybe that only works for public repos? This is for a private repository.

@mgasner

This comment has been minimized.

mgasner commented Feb 12, 2015

While this works for pip install git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip, it doesn't help much when trying to use setup.py.

If you run setup(install_requires=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], ...), you'll get an error like error in package setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers.

If you instead run setup(dependency_links=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Perhaps you thought you'd be clever and do something like install_reqs=parse_requirements( 'requirements.txt')! Too bad, because #2244 (comment).

Is there really no lighter-weight, supported solution for requiring private repos than running your own PyPI?

@remohammadi

This comment has been minimized.

remohammadi commented Jul 9, 2015

If you instead run setup(dependency_links=['git+ssh://git@github.com/account/private-pip-package.git#egg=private-pip'], install_reqs=['private-pip-package'], ...), you'll get an error like Could not find any downloads that satisfy the requirement private-pip-package (from package==1.2.3).

Pip wants to know the version before fetching (git clone) the dependency. If we specify the version after the project name in the egg string, in this hacky form: egg=<project name>-<version>, then there will be an item in _find_all_versions()'s returned list, and everything goes fine. So if we want to dependent our package on the latest commit in the branch STABLE-X of a VCS repository, we can do this:

install_requires=[
    'private-pip-packag',
],
dependency_links=[
    'git+ssh://git@github.com/account/private-pip-package.git@STABLE-X#egg=private-pip-9999999',
],

But, it's too hacky! I think these days it's so reasonable to have a project dependant on the latest version of a branch of some VCS repository, and there should be no need on specifying a dummy version number as a workaround for this purpose.

@ajmazurie

This comment has been minimized.

ajmazurie commented Jun 24, 2016

Hi guys,
I am having the very same problem of not being able to install a specific commit of a Github-hosted project using setuptools' dependency_links option, and was wondering if this was still considered a bug or a misunderstanding of setuptools.

Say, for example, that I want to install a specific commit of the project paramiko-expect; for example, https://github.com/fgimian/paramiko-expect/tree/943630ac499284e6441854a9c4ae1e04301bfdd9

Here is what works and what doesn't, using pip 8.1.2 and setuptools 23.1.0:

  • 🆗 pip install "git+ssh://git@github.com/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2"
  • 🚫 (in dependency_links in setup.py) git+ssh://git@github.com/fgimian/paramiko-expect.git@943630a#egg=paramiko-expect-0.2
  • 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/archive/943630ac499284e6441854a9c4ae1e04301bfdd9.zip#egg=paramiko-expect-0.2
  • 🚫 (in dependency_links in setup.py) https://github.com/fgimian/paramiko-expect/zipball/943630a#egg=paramiko-expect-0.2

Note that the setup.py file then has paramiko-expect==0.2 in its install_requires field, and that I tried all possible variants of paramiko-expect versus paramiko_expect. The setup.py file is then used by typing pip install ..

Is there something I'm missing here?
Best, Aurélien

@BertrandBordage

This comment has been minimized.

BertrandBordage commented Jul 11, 2016

@mgasner @ajmazurie Same issue here… I almost got crazy by spending 3 hours testing all the possibly imaginable combinations for this, and couldn’t find a single one working.

Pip is always trying to find the private module on PyPI, which of course doesn’t work. Instead, it should simply install all dependency_links before checking that the correct version of each install_requires package is installed, and otherwise try to download it on PyPI.

Of course, installing what I put in dependency_links using pip install works.
And obviously, dependency_links items are ignored when they don’t match an install_requires entry, so we can’t even ask for the setup script to always reinstall the same module without checking it was installed.

@xavfernandez

This comment has been minimized.

Contributor

xavfernandez commented Oct 15, 2016

If you want pip to take dependenc_links into account, you should use the --process-dependency-links.

@ChillarAnand

This comment has been minimized.

ChillarAnand commented Dec 27, 2016

@xavfernandez --process-dependency-links is deprecated?

@xavfernandez

This comment has been minimized.

Contributor

xavfernandez commented Dec 27, 2016

@ChillarAnand it is, but without any deprecation timeline and available replacement so this is still the way to go.

@ava-dylang

This comment has been minimized.

ava-dylang commented Jan 5, 2017

soo, there is no way to install a git+https://{url/to/repo}@{branch}#egg={package}-{version} url in setup.py? my understanding was that your module's dependencies should go into setup.py and your pinned-for-release dependencies should go in requirements.txt. But If I cant load first party dependencies in setup.py, this work flow cant be done.. what?

Edit, I suppose this workflow is also complicated by the fact that pip doesn't preserve the source uri of its package, so pip freeze doesn't actually produce an install-ready file, since the "dependency links" equivalent names have to be manually inserted into the file. :(

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

Any update on this issue? having the same problem

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

@wobeng What problem are you having? The solutions above worked for me

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

I have a private package A on github and it requires a private package B on github. When I try to install A, it tries to installs B but stop short with an error saying Could not find a version that satisfies the requirement package B No matching distribution found for package B

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

What's the exact situation? What's the string in setup.py you're using?

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

@MaximilianR are you on slack?

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

No

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

setup(
    name='some-private library',

    // Omitted some less important stuff here...

    install_requires=[
        'some-git-dependency',
        'another-git-dependency',
    ],
    dependency_links=[
        'git+ssh://git@github.com/my-organization/some-git-dependency.git#egg=some-git-dependency',
        'git+ssh://git@github.com/my-organization/another-git-dependency.git#egg=another-git-dependency',
    ],
)

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

Try finishing egg with a number (I know it seems weird!), i.e. #egg=some-git-dependency-0

I've had success with http, but looks like others have with ssh. This is what works for me: 'git+https://{github_token}@github.com/company/{package}.git/@{version}#egg={package}-0'

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

That means I will have to hard code token in the setup.py?

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

No, you can do github_token = os.environ['GITHUB_TOKEN']

@wobeng

This comment has been minimized.

wobeng commented Feb 21, 2017

ok what about when you want to include a private package in requirements.txt? can you use os.environ?

@max-sixty

This comment has been minimized.

max-sixty commented Feb 21, 2017

I'm not sure about that actually, I'd be interested to know what works best there

@ssaumitra

This comment has been minimized.

ssaumitra commented Mar 17, 2017

I am seeing this exact problem #2124 (comment) While @MaximilianR 's HTTPS solution might work, could someone make it work with SSH?

@TimZaman

This comment has been minimized.

TimZaman commented Jul 13, 2017

Still can't get this working properly.

@hnykda

This comment has been minimized.

hnykda commented Jan 10, 2018

We were able to make it work only by using this solution: #3610 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment