Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to install packages with a vcs dependency in their setup.py #3396

Closed
wreed4 opened this issue Dec 18, 2018 · 11 comments · Fixed by #4241
Closed

How to install packages with a vcs dependency in their setup.py #3396

wreed4 opened this issue Dec 18, 2018 · 11 comments · Fixed by #4241
Labels
Category: Dependency Resolution Category: VCS

Comments

@wreed4
Copy link

@wreed4 wreed4 commented Dec 18, 2018

I'm working on a project at work for which one of the dependencies cannot be published to pypi, so I have to install it with the git url. Unfortunately I can't paste the setup.py here.

What I am trying to accomplish is to use the Pipfile/Pipfile.lock as a development environment for the tool and then deploy the tool with pip install. So I'm trying to specify all my dependencies in the setup.py and then just have the -e . line in my Pipfile. My package installs perfectly fine with pip install . and pipenv install --skip-lock. However, pipenv lock fails with the error (obviously redacted):

pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches packagename@ git+https://GITHUB_TOKEN@github.com/user/repo.git@v0.1.0#egg=packagename from git+https://GITHUB_TOKEN@github.com/user/repo.git@v0.1.0#egg=packagename                          
No versions found    

I know it's going to be difficult to give specific advice without more information, but I'm hoping you can give me the format and conditions that must be true (does the repo need to be tagged, etc) for pipenv to resolve this dependency the way pip does. Thanks

@frostming frostming added Category: Dependency Resolution Category: VCS labels Jan 4, 2019
@wreed4
Copy link
Author

@wreed4 wreed4 commented Jan 10, 2019

Any thoughts here?

@frostming
Copy link
Contributor

@frostming frostming commented Jan 11, 2019

Can you provide the content of setup.py and how you specify your dependency? You can, of course, hide the sensitive part.

@wreed4
Copy link
Author

@wreed4 wreed4 commented Jan 18, 2019

@frostming here's the setup.py with sensitive things changed

Any suggestions on the what the Pipfile should look like? My goal is now to use pipenv as the development environment and to install the package in "production" (other developer's machines and jenkins) using the requirements.txt file generated by pipenv lock -r. I've currently achieved that by installing everything BUT -e . in the pipfile directly and then requiring devs to install -e . inside the pipenv environment themselves... Is this the suggested workflow?

"""
Setup script for mypackage package
"""
import os
import sys

from subprocess import run, PIPE
from setuptools import setup, find_packages

def get_version():
    """
    Dynamically get the version for this package based on git
    """
    tag = run(['git', 'describe', '--tags', '--abbrev=0'], stdout=PIPE, universal_newlines=True)
    tag = tag.stdout[1:].strip()  # get rid of leading v for pip

    branch = run(['git', 'describe', '--all'], stdout=PIPE, universal_newlines=True)
    branch: str = branch.stdout.split('/')[-1]  # strip of 'heads/'
    if branch.startswith('prefix-'):
        branch = branch[3:].replace('-', '.')  # replace subsequent dashes with dots
        branch = ''.join(filter(lambda c: c.isdigit() or c == '.', branch))  # take all the digits and dots
        branch = branch.strip('.')  # strip unnecessary dots
        return f'{tag}.{branch}'

    return tag

try:
    TOKEN = os.environ["GITHUB_TOKEN"]
except KeyError:
    print("ERROR: Installation requires a github token be present in the environment.  Please `export GITHUB_TOKEN=<TOKEN>`.")
    sys.exit(1)

setup(
    name='mypackage',
    version=get_version(),
    packages=find_packages(),
    package_data={
        'mypackage': ['resources/*'],
    },
    include_package_data=True,
    install_requires=[
        'pip_package_dependency1',
        'pip_package_dependency2',
        f'vcs_private_dependency @ git+https://{TOKEN}@github.com/user/repo.get',

        #  These are only required for tests. setuptools doesn't like tests_require anymore for some reason
        'pytest-runner',
        'pytest-pylint',
        'pylint',
        'pytest',
    ],
    entry_points='''
        [console_scripts]
        mypackage=mypackage:cli
    ''',
)

@itsayellow
Copy link

@itsayellow itsayellow commented May 28, 2019

I have a non-pypi but public github example of this issue, with the same type of setup.py trying to use a git repo as an install_requires dependency.

I attempt to install my python library using:
pipenv install -e git+https://github.com/itsayellow/mlutils.git#egg=mlutils

This has a dependency to another one of my git repo python libraries, tictoc.

pip can install mlutils fine, but pipenv fails:

[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches tictoc@ git+https://github.com/itsayellow/tictoc@master from git+https://github.com/itsayellow/tictoc@master
[pipenv.exceptions.ResolutionFailure]:       No versions found

However, pipenv will succeed if I install tictoc first using:
pipenv install -e git+https://github.com/itsayellow/tictoc.git#egg=tictoc

I can then successfully use pipenv to install mlutils that depends on tictoc:
pipenv install -e git+https://github.com/itsayellow/mlutils.git#egg=mlutils

So this depends on the order. If I try to install mlutils first with pipenv, it fails. But if I install its dependency tictoc using pipenv first, then a later install of mlutils will succeed.

https://github.com/itsayellow/mlutils
https://github.com/itsayellow/tictoc

krzentner added a commit to rlworkgroup/garage that referenced this issue Aug 1, 2019
Because dm_control needs to be installed from vcs or tarball, garage
cannot be directly installed in pipenv (see [this
issue](pypa/pipenv#3396)).

I've copied the example that used our dm_control wrapper, modified
the original example to use gym, and modified the copy to clarify the
extra dependency.

I've also modified the `garage.env.dm_control` module to report a
helpful error message if dm_control isn't found.

`pip install garage[all]` will still install dm_control.
krzentner added a commit to rlworkgroup/garage that referenced this issue Aug 19, 2019
Because dm_control needs to be installed from vcs or tarball, garage
cannot be directly installed in pipenv (see [this
issue](pypa/pipenv#3396)).

I've copied the example that used our dm_control wrapper, modified
the original example to use gym, and modified the copy to clarify the
extra dependency.

I've also modified the `garage.env.dm_control` module to report a
helpful error message if dm_control isn't found.

`pip install garage[all]` will still install dm_control.
@paddie
Copy link

@paddie paddie commented Apr 8, 2020

@itsayellow I seem to be having precisely the same issue. Could it be that it is not attempting to perform a -e installation? If I attempt to install the repo using pipenv but without the -e argument it fails for me in the same way as you are reporting. So, is there some way to tell pipenv to install some VCS dependency using the -e flag?

@coltonbh
Copy link

@coltonbh coltonbh commented Apr 20, 2020

@frostming I'm having the exact same issue outlined here--specifically that when I try to install a package using pipenv install -e git+ssh://git@github.com/my_org/my_pacakge.git@0.3.3#egg=my_package if that package contains a VSC reference in it's setup.py in install_requires, e.g., "my_other_package @ git+ssh://git@github.com/my_org/my_other_package.git@1.8.0", then I get the error:

[pipenv.exceptions.ResolutionFailure]: pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches my_other_package@ git+ssh://git@github.com/my_org/my_other_package.git@1.8.0 from git+ssh://git@github.com/my_org/my_other_package.git@1.8.0.

I can install the main package using pip just fine. This command will work:

pipenv run pip install -e git+ssh://git@github.com/my_org/my_pacakge.git@0.3.3#egg=my_package

However this command will fail:
pipenv install -e git+ssh://git@github.com/my_org/my_pacakge.git@0.3.3#egg=my_package

I can solve the problem by installing the package referenced in the setup.py file of my_package first:
pipenv install -e git+ssh://git@github.com/my_org/my_other_pacakge.git@1.8.0"#egg=my_other_package

Then running:
pipenv install -e git+ssh://git@github.com/my_org/my_pacakge.git@0.3.3#egg=my_package.

What's going on here? Seems like some kind of bug.

Here's the full log output using the actual packages I'm trying to install. They are private so you can't view them, but it should give you an idea of what's going on.

Setup.py file:

from setuptools import find_packages, setup

import biovault_etl

setup(
    name="biovault_etl",
    version=biovault_etl.__version__,
    description="Biovault ETL Code",
    author="BioVault",
    author_email="colton.hicks@q.bio",
    packages=find_packages(exclude=["tests*", "lambda*"]),
    install_requires=[
        "boto3>=1.12.37",
        "click>=7.1.1",
        "hl7>=0.3.4",
        "pdftotext>=2.1.4",
        "psycopg2-binary>=2.8.5",
        "pydantic>=1.4",
        "sqlalchemy>=1.3.16",
        "ids @ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1",
    ],
    dependency_links=[
        "git+ssh://git@github.com/qbio/health-gorilla-client.git@0.1.2#egg=health-gorilla",
        "git+ssh://git@github.com/qbio/immutable-data-store.git@2.3.0#egg=ids",
    ],
    entry_points="""
        [console_scripts]
        etl=biovault_etl.cli.cli:cli
    """,
)

Failed Installation:

❯ pipenv install -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault_etl --verbose                                                                                                                                                         ─╯
Creating a virtualenv for this project…
Pipfile: /Users/cbh/dev/hacking/bvetl/Pipfile
Using /usr/local/Cellar/pipenv/2018.11.26_4/libexec/bin/python3.8 (3.8.2) to create virtualenv…
⠇ Creating virtual environment...created virtual environment CPython3.8.2.final.0-64 in 488ms
  creator CPython3Posix(dest=/Users/cbh/.local/share/virtualenvs/bvetl-i-Jw8lEd, clear=False, global=False)
  seeder FromAppData(download=False, pip=latest, setuptools=latest, wheel=latest, via=copy, app_data_dir=/Users/cbh/Library/Application Support/virtualenv/seed-app-data/v1)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator,XonshActivator

✔ Successfully created virtual environment! 
Virtualenv location: /Users/cbh/.local/share/virtualenvs/bvetl-i-Jw8lEd
Creating a Pipfile for this project…
Installing -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault_etl…
⠋ Installing...Installing 'biovault_etl'
$ ['/Users/cbh/.local/share/virtualenvs/bvetl-i-Jw8lEd/bin/pip', 'install', '--src', '/Users/cbh/.local/share/virtualenvs/bvetl-i-Jw8lEd/src', '--verbose', '--upgrade', '-e', 'git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault_etl', '-i', 'https://pypi.org/simple']
Adding biovault_etl to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
⠋ Pinning VCS Packages...INFO:pipenv.patched.notpip._internal.vcs.git:Cloning ssh://git@github.com/qbio/biovault-etl.git (to revision release/0.3.3) to /var/folders/w2/rkn1bm3d3ndclys0f81clbfw0000gn/T/requirementslibd0yxmo_5/biovault-etl
⠹ Pinning VCS Packages...Branch 'release/0.3.3' set up to track remote branch 'release/0.3.3' from 'origin'.
Switched to a new branch 'release/0.3.3'
✘ Locking Failed! 
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  biovault-etl from git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (from -r /var/folders/w2/rkn1bm3d3ndclys0f81clbfw0000gn/T/pipenvvwwqha7lrequirements/pipenv-xtt7hh1q-constraints.txt (line 2))

Finding the best candidates:
  found candidate -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (constraint was <any>)

Finding secondary dependencies:
Switched to a new branch 'release/0.3.3'
Branch 'release/0.3.3' set up to track remote branch 'release/0.3.3' from 'origin'.

New dependencies found in this round:
  adding ['boto3', '>=1.12.37', '[]']
  adding ['click', '>=7.1.1', '[]']
  adding ['hl7', '>=0.3.4', '[]']
  adding ['ids', '', '[]']
  adding ['pdftotext', '>=2.1.4', '[]']
  adding ['psycopg2-binary', '>=2.8.5', '[]']
  adding ['pydantic', '>=1.4', '[]']
  adding ['sqlalchemy', '>=1.3.16', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  biovault-etl from git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (from -r /var/folders/w2/rkn1bm3d3ndclys0f81clbfw0000gn/T/pipenvvwwqha7lrequirements/pipenv-xtt7hh1q-constraints.txt (line 2))
  boto3>=1.12.37
  click>=7.1.1
  hl7>=0.3.4
  ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
  pdftotext>=2.1.4
  psycopg2-binary>=2.8.5
  pydantic>=1.4
  sqlalchemy>=1.3.16

Finding the best candidates:
  found candidate -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (constraint was <any>)
  found candidate boto3==1.12.42 (constraint was >=1.12.37)
  found candidate click==7.1.1 (constraint was >=7.1.1)
  found candidate hl7==0.3.4 (constraint was >=0.3.4)
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 198, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 198, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 263, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 175, in find_best_match
    raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
No versions found
Was https://pypi.org/simple reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 118, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system,
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 78, in _main
    results = resolve(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 61, in resolve
    return resolve_deps(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 718, in resolve_deps
    resolved_tree, hashes, markers_lookup, resolver = actually_resolve_deps(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 395, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
No versions found
Was https://pypi.org/simple reachable?
Using pip: -i https://pypi.org/simple
Using pip: -i https://pypi.org/simple

                          ROUND 1                           
Current constraints:
  biovault-etl from git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (from -r /var/folders/w2/rkn1bm3d3ndclys0f81clbfw0000gn/T/pipenvvwwqha7lrequirements/pipenv-xtt7hh1q-constraints.txt (line 2))

Finding the best candidates:
  found candidate -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (constraint was <any>)

Finding secondary dependencies:
Switched to a new branch 'release/0.3.3'
Branch 'release/0.3.3' set up to track remote branch 'release/0.3.3' from 'origin'.

New dependencies found in this round:
  adding ['boto3', '>=1.12.37', '[]']
  adding ['click', '>=7.1.1', '[]']
  adding ['hl7', '>=0.3.4', '[]']
  adding ['ids', '', '[]']
  adding ['pdftotext', '>=2.1.4', '[]']
  adding ['psycopg2-binary', '>=2.8.5', '[]']
  adding ['pydantic', '>=1.4', '[]']
  adding ['sqlalchemy', '>=1.3.16', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2                           
Current constraints:
  biovault-etl from git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (from -r /var/folders/w2/rkn1bm3d3ndclys0f81clbfw0000gn/T/pipenvvwwqha7lrequirements/pipenv-xtt7hh1q-constraints.txt (line 2))
  boto3>=1.12.37
  click>=7.1.1
  hl7>=0.3.4
  ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
  pdftotext>=2.1.4
  psycopg2-binary>=2.8.5
  pydantic>=1.4
  sqlalchemy>=1.3.16

Finding the best candidates:
  found candidate -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault-etl (constraint was <any>)
  found candidate boto3==1.12.42 (constraint was >=1.12.37)
  found candidate click==7.1.1 (constraint was >=7.1.1)
  found candidate hl7==0.3.4 (constraint was >=0.3.4)
Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 385, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 102, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 198, in _resolve_one_round
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 198, in <setcomp>
    best_matches = {self.get_best_match(ireq) for ireq in constraints}
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/resolver.py", line 263, in get_best_match
    best_match = self.repository.find_best_match(ireq, prereleases=self.prereleases)
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 175, in find_best_match
    raise NoCandidateFound(ireq, all_candidates, self.finder)
pipenv.patched.piptools.exceptions.NoCandidateFound: Could not find a version that matches ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
No versions found
Was https://pypi.org/simple reachable?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 126, in <module>
    main()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 118, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system,
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 78, in _main
    results = resolve(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/resolver.py", line 61, in resolve
    return resolve_deps(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 718, in resolve_deps
    resolved_tree, hashes, markers_lookup, resolver = actually_resolve_deps(
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
    resolved_tree = resolver.resolve()
  File "/usr/local/Cellar/pipenv/2018.11.26_4/libexec/lib/python3.8/site-packages/pipenv/utils.py", line 395, in resolve
    raise ResolutionFailure(message=str(e))
pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches ids@ git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1 from git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1
No versions found
Was https://pypi.org/simple reachable?

But if I run:
pipenv install -e git+ssh://git@github.com/qbio/immutable-data-store.git@release/2.3.1#egg=ids
then:
pipenv install -e git+ssh://git@github.com/qbio/biovault-etl.git@release/0.3.3#egg=biovault_etl
the installation succeeds. Here is the successful log:

Any insights?

@techalchemy
Copy link
Member

@techalchemy techalchemy commented Apr 29, 2020

Sorry for the issues everyone -- dependency links are deprecated, you will need to use the direct url syntax in the format of install_requires=["name@ git+ssh://github.com/org/repo.git@ref"] to include dependencies from version control. This should work as of the current prerelease of pipenv which is available via pip install --upgrade --pre pipenv.

For example, using this take on the setup.py posted above:

from setuptools import find_packages, setup

setup(
    name="biovault_etl",
    version="0.0.1",
    description="Biovault ETL Code",
    author="BioVault",
    author_email="colton.hicks@q.bio",
    packages=find_packages(exclude=["tests*", "lambda*"]),
    install_requires=["vistir@ git+https://github.com/sarugaku/vistir.git@master"],
)

I was able to install:

 /t/test  pipenv install -e .
Installing -e .…
Adding biovault_etl to Pipfile's [packages]…
✔ Installation Succeeded 
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
Building requirements...
Resolving dependencies...
✔ Success! 
Updated Pipfile.lock (c081cf)!
Installing dependencies from Pipfile.lock (c081cf)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 4/4 — 00:00:06
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

I believe this should be resolved with the prerelease so I will close this issue for now, the release should be live in about a week. Please feel free to open a new issue if you encounter any problems.

Thanks for your patience!

@coltonbh
Copy link

@coltonbh coltonbh commented Apr 29, 2020

Thanks, @techalchemy for the response, and for all the work on the new release! The scenario you demonstrated above also worked on the v2018.11.26 version of pipenv too :) The problem arises when vistir package ALSO has a direct url link it its setup.py install_requires section. This is when pipenv creates the error I noted above. Is this a separate issue I should open? Or perhaps maintain the thread here?

Thanks again for all the work on the new release--I know pipenv has gone through a lot of transitions over the past few years and it's great to see you carrying the project forward :)

@techalchemy
Copy link
Member

@techalchemy techalchemy commented Apr 30, 2020

@coltonbh so it's an issue of nesting? to put it visually:

(A): package w/ direct dependency in setup.py
(B): Random normal package
(C): Standard package
(D): another package w/ direct dependency in setup.py

This scenario would install ok:
(A)
| - (B)<-(A)

But this scenario would not:
(A)
| - (D)<-(A)

What about this one?
(C)
| - (D)<-C

i.e. A normal package as the root dependency, which depends on (D) which has a direct URL in its setup.py?

Are you saying that the last case is also broken?

@coltonbh
Copy link

@coltonbh coltonbh commented Apr 30, 2020

Correct! The last two scenarios would fail. It appears to me, based on my understanding of the exception message, is that pipenv is literally trying to find version @ git+ssh://... of the package, which it can't find. (I think...)

Assuming we want to install D from other packages (reference D in install_requires of other packages), you'd need to place my_direct_dependency>=0.3.4 in the install_requires of D instead of the direct url. Then, since I could no longer install package D directly because my_direct_dependency doesn't exist on PyPi and installation would fail I'd need to install my_direct_dependency>=v0.3.4 "manually" so my Pipfile lists my_direct_dependency as being installed from the GitHub. Then I'd install the package I am working on that includes a reference to D>=vWhatever with pipenv install -e .. Pipenv would then correctly install my package that includes a reference to D and notice that locally my_direct_dependency, which D depends on, is already satisfied and installed from Github.

While this is somewhat annoying that we can't install packages that refer to other packages with direct dependencies, there's a case to be made that this is correct behavior based on the notion of abstract/concrete dependences. Pipenv documentation. Good blog post.

After research all this, I think the absolute purist would maintain that there should never be direct URLs in setup.py. However, if the python community feels this should be a supported feature, getting pipenv to make this work would be nice :)

@techalchemy
Copy link
Member

@techalchemy techalchemy commented May 1, 2020

Yeah the pre-release should work with this use case. Does it work if you set PIPENV_RESOLVE_VCS=1?

@techalchemy techalchemy reopened this May 1, 2020
techalchemy added a commit that referenced this issue May 5, 2020
- Recursively resolve PEP 508 URLs in subdependencies
  - Fix `piptools` cache storage of VCS dependencies as results
- Avoid saving VCS dependency versions to `Pipfile.lock`
- Fixes #3396
- Fixes #4217

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
techalchemy added a commit that referenced this issue May 5, 2020
- Recursively resolve PEP 508 URLs in subdependencies
  - Fix `piptools` cache storage of VCS dependencies as results
- Avoid saving VCS dependency versions to `Pipfile.lock`
- Fixes #3396
- Fixes #4217

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
techalchemy added a commit that referenced this issue May 6, 2020
- Recursively resolve PEP 508 URLs in subdependencies
  - Fix `piptools` cache storage of VCS dependencies as results
- Avoid saving VCS dependency versions to `Pipfile.lock`
- Fixes #3396
- Fixes #4217

Signed-off-by: Dan Ryan <dan.ryan@canonical.com>
fwojciak pushed a commit to fwojciak/pipenv that referenced this issue May 29, 2020
2020.5.28 (2020-05-28)
======================

Features & Improvements
-----------------------

-   `pipenv install` and `pipenv sync` will no longer attempt to install satisfied dependencies during installation. pypa#3057, pypa#3506
-   Added support for resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Added full support for resolution of all dependency types including direct URLs, zip archives, tarballs, etc.
    -   Improved error handling and formatting.
    -   Introduced improved cross platform stream wrappers for better `stdout` and `stderr` consistency. pypa#3298
-   For consistency with other commands and the `--dev` option description, `pipenv lock --requirements --dev` now emits both default and development dependencies. The new `--dev-only` option requests the previous behaviour (e.g. to generate a `dev-requirements.txt` file). pypa#3316
-   Pipenv will now successfully recursively lock VCS sub-dependencies. pypa#3328
-   Added support for `--verbose` output to `pipenv run`. pypa#3348
-   Pipenv will now discover and resolve the intrinsic dependencies of **all** VCS dependencies, whether they are editable or not, to prevent resolution conflicts. pypa#3368
-   Added a new environment variable, `PIPENV_RESOLVE_VCS`, to toggle dependency resolution off for non-editable VCS, file, and URL based dependencies. pypa#3577
-   Added the ability for Windows users to enable emojis by setting `PIPENV_HIDE_EMOJIS=0`. pypa#3595
-   Allow overriding `PIPENV_INSTALL_TIMEOUT` environment variable (in seconds). pypa#3652
-   Allow overriding `PIP_EXISTS_ACTION` evironment variable (value is passed to pip install). Possible values here: <https://pip.pypa.io/en/stable/reference/pip/#exists-action-option> Useful when you need to `PIP\_EXISTS\_ACTION=i` (ignore existing packages) - great for CI environments, where you need really fast setup. pypa#3738
-   Pipenv will no longer forcibly override `PIP_NO_DEPS` on all vcs and file dependencies as resolution happens on these in a pre-lock step. pypa#3763
-   Improved verbose logging output during `pipenv lock` will now stream output to the console while maintaining a spinner. pypa#3810
-   Added support for automatic python installs via `asdf` and associated `PIPENV_DONT_USE_ASDF` environment variable. pypa#4018
-   Pyenv/asdf can now be used whether or not they are available on PATH. Setting `PYENV_ROOT`/`ASDF_DIR` in a `.env` file allows Pipenv to install an interpreter without any shell customizations, so long as pyenv/asdf is installed. pypa#4245
-   Added `--key` command line parameter for including personal PyUp.io API tokens when running `pipenv check`. pypa#4257

Behavior Changes
----------------

-   Make conservative checks of known exceptions when subprocess returns output, so user won\'t see the whole traceback - just the error. pypa#2553
-   Do not touch Pipfile early and rely on it so that one can do `pipenv sync` without a Pipfile. pypa#3386
-   Re-enable `--help` option for `pipenv run` command. pypa#3844
-   Make sure `pipenv lock -r --pypi-mirror {MIRROR_URL}` will respect the pypi-mirror in requirements output. pypa#4199

Bug Fixes
---------

-   Raise `PipenvUsageError` when \[\[source\]\] does not contain url field. pypa#2373
-   Fixed a bug which caused editable package resolution to sometimes fail with an unhelpful setuptools-related error message. pypa#2722
-   Fixed an issue which caused errors due to reliance on the system utilities `which` and `where` which may not always exist on some
    systems.
-   Fixed a bug which caused periodic failures in python discovery when executables named `python` were not present on the target `$PATH`. pypa#2783
-   Dependency resolution now writes hashes for local and remote files to the lockfile. pypa#3053
-   Fixed a bug which prevented `pipenv graph` from correctly showing all dependencies when running from within `pipenv shell`. pypa#3071
-   Fixed resolution of direct-url dependencies in `setup.py` files to respect `PEP-508` style URL dependencies. pypa#3148
-   Fixed a bug which caused failures in warning reporting when running pipenv inside a virtualenv under some circumstances.
-   Fixed a bug with package discovery when running `pipenv clean`. pypa#3298
-   Quote command arguments with carets (`^`) on Windows to work around unintended shell escapes. pypa#3307
-   Handle alternate names for UTF-8 encoding. pypa#3313
-   Abort pipenv before adding the non-exist package to Pipfile. pypa#3318
-   Don\'t normalize the package name user passes in. pypa#3324
-   Fix a bug where custom virtualenv can not be activated with pipenv shell pypa#3339
-   Fix a bug that `--site-packages` flag is not recognized. pypa#3351
-   Fix a bug where `pipenv --clear` is not working pypa#3353
-   Fix unhashable type error during `$ pipenv install --selective-upgrade` pypa#3384
-   Dependencies with direct `PEP508` compliant VCS URLs specified in their `install_requires` will now be successfully locked during the resolution process. pypa#3396
-   Fixed a keyerror which could occur when locking VCS dependencies in
    some cases. pypa#3404
-   Fixed a bug that `ValidationError` is thrown when some fields are missing in source section. pypa#3427
-   Updated the index names in lock file when source name in Pipfile is changed. pypa#3449
-   Fixed an issue which caused `pipenv install --help` to show duplicate entries for `--pre`. pypa#3479
-   Fix bug causing `[SSL: CERTIFICATE_VERIFY_FAILED]` when Pipfile `[[source]]` has `verify_ssl=false` and url with custom port. pypa#3502
-   Fix `sync --sequential` ignoring `pip install` errors and logs. pypa#3537
-   Fix the issue that lock file can\'t be created when `PIPENV_PIPFILE` is not under working directory. pypa#3584
-   Pipenv will no longer inadvertently set `editable=True` on all vcs dependencies. pypa#3647
-   The `--keep-outdated` argument to `pipenv install` and `pipenv lock` will now drop specifier constraints when encountering editable dependencies.
    -   In addition, `--keep-outdated` will retain specifiers that would otherwise be dropped from any entries that have not been updated. pypa#3656
-   Fixed a bug which sometimes caused pipenv to fail to respect the `--site-packages` flag when passed with `pipenv install`. pypa#3718
-   Normalize the package names to lowercase when comparing used and in-Pipfile packages. pypa#3745
-   `pipenv update --outdated` will now correctly handle comparisons between pre/post-releases and normal releases. pypa#3766
-   Fixed a `KeyError` which could occur when pinning outdated VCS dependencies via `pipenv lock --keep-outdated`. pypa#3768
-   Resolved an issue which caused resolution to fail when encountering poorly formatted `python_version` markers in `setup.py` and `setup.cfg` files. pypa#3786
-   Fix a bug that installation errors are displayed as a list. pypa#3794
-   Update `pythonfinder` to fix a problem that `python.exe` will be mistakenly chosen for virtualenv creation under WSL. pypa#3807
-   Fixed several bugs which could prevent editable VCS dependencies from being installed into target environments, even when reporting
    successful installation. pypa#3809
-   `pipenv check --system` should find the correct Python interpreter when `python` does not exist on the system. pypa#3819
-   Resolve the symlinks when the path is absolute. pypa#3842
-   Pass `--pre` and `--clear` options to `pipenv update --outdated`. pypa#3879
-   Fixed a bug which prevented resolution of direct URL dependencies which have PEP508 style direct url VCS sub-dependencies with
    subdirectories. pypa#3976
-   Honor `PIPENV_SPINNER` environment variable pypa#4045
-   Fixed an issue with `pipenv check` failing due to an invalid API key from `pyup.io`. pypa#4188
-   Fixed a bug which caused versions from VCS dependencies to be included in `Pipfile.lock` inadvertently. pypa#4217
-   Fixed a bug which caused pipenv to search non-existent virtual environments for `pip` when installing using `--system`. pypa#4220
-   `Requires-Python` values specifying constraint versions of python starting from `1.x` will now be parsed successfully. pypa#4226
-   Fix a bug of `pipenv update --outdated` that can\'t print output correctly. pypa#4229
-   Fixed a bug which caused pipenv to prefer source distributions over wheels from `PyPI` during the dependency resolution phase. Fixed an issue which prevented proper build isolation using `pep517` based builders during dependency resolution. pypa#4231
-   Don\'t fallback to system Python when no matching Python version is found. pypa#4232

Vendored Libraries
------------------

- Updated `pip_shims` to support `--outdated` with new pip versions. pypa#3766
- Update vendored dependencies and invocations
  - Update vendored and patched dependencies
  - Update patches on `piptools`, `pip`, `pip-shims`, `tomlkit`
  - Fix invocations of dependencies
  - Fix custom `InstallCommand` instantiation
  - Update `PackageFinder` usage
  - Fix `Bool` stringify attempts from `tomlkit`
  - Updated vendored dependencies:
    -   **attrs**: `18.2.0 => `19.1.0`
    -   **certifi**: `2018.10.15 => `2019.3.9`
    -   **cached\_property**: `1.4.3 => `1.5.1`
    -   **cerberus**: `1.2.0 => `1.3.1`
    -   **click**: `7.0.0 => `7.1.1`
    -   **click-completion**: `0.5.0 => `0.5.1`
    -   **colorama**: `0.3.9 => `0.4.3`
    -   **contextlib2**: `(new) => `0.6.0.post1`
    -   **distlib**: `0.2.8 => `0.2.9`
    -   **funcsigs**: `(new) => `1.0.2`
    -   **importlib\_metadata** `1.3.0 => `1.5.1`
    -   **importlib-resources**: `(new) => `1.4.0`
    -   **idna**: `2.7 => `2.9`
    -   **jinja2**: `2.10.0 => `2.11.1`
    -   **markupsafe**: `1.0 => `1.1.1`
    -   **more-itertools**: `(new) => `5.0.0`
    -   **orderedmultidict**: `(new) => `1.0`
    -   **packaging**: `18.0 => `19.0`
    -   **parse**: `1.9.0 => `1.15.0`
    -   **pathlib2**: `2.3.2 => `2.3.3`
    -   **pep517**: `(new) => `0.5.0`
    -   **pexpect**: `4.6.0 => `4.8.0`
    -   **pip-shims**: `0.2.0 => `0.5.1`
    -   **pipdeptree**: `0.13.0 => `0.13.2`
    -   **pyparsing**: `2.2.2 => `2.4.6`
    -   **python-dotenv**: `0.9.1 => `0.10.2`
    -   **pythonfinder**: `1.1.10 => `1.2.2`
    -   **pytoml**: `(new) => `0.1.20`
    -   **requests**: `2.20.1 => `2.23.0`
    -   **requirementslib**: `1.3.3 => `1.5.4`
    -   **scandir**: `1.9.0 => `1.10.0`
    -   **shellingham**: `1.2.7 => `1.3.2`
    -   **six**: `1.11.0 => `1.14.0`
    -   **tomlkit**: `0.5.2 => `0.5.11`
    -   **urllib3**: `1.24 => `1.25.8`
    -   **vistir**: `0.3.0 => `0.5.0`
    -   **yaspin**: `0.14.0 => `0.14.3`
    -   **zipp**: `0.6.0`
    - Removed vendored dependency **cursor**. pypa#4169

-   Add and update vendored dependencies to accommodate `safety` vendoring:
    -   **safety** `(none)` => `1.8.7`
    -   **dparse** `(none)` => `0.5.0`
    -   **pyyaml** `(none)` => `5.3.1`
    -   **urllib3** `1.25.8` => `1.25.9`
    -   **certifi** `2019.11.28` => `2020.4.5.1`
    -   **pyparsing** `2.4.6` => `2.4.7`
    -   **resolvelib** `0.2.2` => `0.3.0`
    -   **importlib-metadata** `1.5.1` => `1.6.0`
    -   **pip-shims** `0.5.1` => `0.5.2`
    -   **requirementslib** `1.5.5` => `1.5.6` pypa#4188

-   Updated vendored `pip` => `20.0.2` and `pip-tools` => `5.0.0`. pypa#4215
-   Updated vendored dependencies to latest versions for security and bug fixes:
    -   **requirementslib** `1.5.8` => `1.5.9`
    -   **vistir** `0.5.0` => `0.5.1`
    -   **jinja2** `2.11.1` => `2.11.2`
    -   **click** `7.1.1` => `7.1.2`
    -   **dateutil** `(none)` => `2.8.1`
    -   **backports.functools\_lru\_cache** `1.5.0` => `1.6.1`
    -   **enum34** `1.1.6` => `1.1.10`
    -   **toml** `0.10.0` => `0.10.1`
    -   **importlib\_resources** `1.4.0` => `1.5.0` pypa#4226
-   Changed attrs import path in vendored dependencies to always import from `pipenv.vendor`. pypa#4267

Improved Documentation
----------------------

-   Added documenation about variable expansion in `Pipfile` entries. pypa#2317
-   Consolidate all contributing docs in the rst file pypa#3120
-   Update the out-dated manual page. pypa#3246
-   Move CLI docs to its own page. pypa#3346
-   Replace (non-existant) video on docs index.rst with equivalent gif. pypa#3499
-   Clarify wording in Basic Usage example on using double quotes to escape shell redirection pypa#3522
-   Ensure docs show navigation on small-screen devices pypa#3527
-   Added a link to the TOML Spec under General Recommendations & Version Control to clarify how Pipfiles should be written. pypa#3629
-   Updated the documentation with the new `pytest` entrypoint. pypa#3759
-   Fix link to GIF in README.md demonstrating Pipenv\'s usage, and add descriptive alt text. pypa#3911
-   Added a line describing potential issues in fancy extension. pypa#3912
-   Documental description of how Pipfile works and association with Pipenv. pypa#3913
-   Clarify the proper value of `python_version` and `python_full_version`. pypa#3914
-   Write description for `--deploy` extension and few extensions differences. pypa#3915
-   More documentation for `.env` files pypa#4100
-   Updated documentation to point to working links. pypa#4137
-   Replace docs.pipenv.org with pipenv.pypa.io pypa#4167
-   Added functionality to check spelling in documentation and cleaned up existing typographical issues. pypa#4209
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Category: Dependency Resolution Category: VCS
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants