Skip to content

Allow "double requirements" if versions don't conflict #56

Closed
vbabiy opened this Issue Mar 15, 2011 · 15 comments
@vbabiy
vbabiy commented Mar 15, 2011

I'd like to be able to compose a requirements file that uses the -r option to install several other packages by their own requirements files.

But in this use scenario, it's possible - even likely - that these separate packages may have overlapping requirements.

This works fine when installing each of them with a separate pip command, as long as you don't hit a version conflict; but when using one master requirements file that includes the others, you'd get the "Double requirement given" error.

I'd like to propose that it should not be an error unless there are version conflicts.

That may be tricky, as it means computing the union of the allowed versions, and pkg_resources confuses me so I'm not sure if there's a right way to do it.


@slinkp
slinkp commented Aug 4, 2011

Related: #174

@muelli
muelli commented Jul 18, 2012

My usecase is to have hierarchical requirement files, i.e. "running.txt" and "testing.txt" where "running" would only contain the bare minimum to make it run, i.e. "django>=1.3" and "testing" would need "django>=1.4" because some parts in the testing process use a newer API which is only available in the more recent version.
I'd like to be able to put "django>1.3" in my "running.txt" and then "-r running.txt django>=1.4" in my testing.txt

@jacinda
jacinda commented Aug 21, 2013

I have the same use case as muelli.

@guettli
guettli commented Sep 19, 2013

would be nice to have this feature.

@jgumbley
jgumbley commented Nov 7, 2013

+1

@DXist
DXist commented Dec 26, 2013

+1

@integricho

+1

@carlosviol

+1

@tboggs
tboggs commented Mar 6, 2014

+1

@Gr1N
Gr1N commented Mar 14, 2014

+1

@GheRivero

+2, Approved :)

@qwcode qwcode self-assigned this Mar 27, 2014
@qwcode
Python Packaging Authority member
qwcode commented Mar 27, 2014

btw, I've started on this. will try to get a PR posted in a few days. involves some refactor that's needed to make things clearer about our dependency resolution.

@qwcode
Python Packaging Authority member
qwcode commented Apr 6, 2014

I admit to not reading this issue correctly before. I'm closing this as a dupe to #988, which is our main issue for adding a proper resolver to pip. I just updated the description to cover resolving top-level requirements.

if anyone adding +1's here are actually concerned about literal duplicates across multiple requirement files, see #993. that is actually the problem I mentioned above that I was starting on.

@qwcode qwcode closed this Apr 6, 2014
@piotr-dobrogost

See issue #1795
@qwcode
I don't think it's a duplicate of issue #988 as here the aim is to just "merge" all conditions of the same top-level package and it seems it's orthogonal to general problem of dependency management which is the subject of issue #988. If you agree this should be reopened and taken into consideration when discussing Requirements 2.0 in issue #1795

@qwcode
Python Packaging Authority member
qwcode commented May 24, 2014

@piotr-dobrogost : I only closed this as a dupe after I updated the description of #988 to cover the problem of consolidating multiple top-level requirements for a project. so in a very literal sense, it is a dupe now. consolidating multiple top-level requirements seems similar to consolidating multiple in the dependency tree. I want #988 ticket to hold all the big fundamental changes to our resolver logic, which will likely get fixed at one time together.

@openstack-gerrit openstack-gerrit pushed a commit to openstack/oslo.db that referenced this issue Jul 6, 2014
@zzzeek zzzeek Test for distinct SQLAlchemy major releases
This change presents one way we might include test support
for oslo.db against specific SQLAlchemy major releases, currently
including the 0.7, 0.8, and 0.9 series.  As we will want to
begin including features within oslo.db that target advanced
and in some cases semi-public APIs within SQLAlchemy, it will
be important that we test these features against each major release,
as there may be variances between major revs as well as
version-specific approaches within oslo.

To accomplish this, I was not able to override "deps" alone,
as the SQLAlchemy revision within requirements.txt conflicts
with a hand-entered requirement, and due to pip's lack of
a dependency resolver (see pypa/pip#988
and pypa/pip#56) I instead overrode
"commands".  I don't know that this is the best approach, nor
do I know how the tox.ini file is accommodated by CI servers,
if these CI servers would need their tox invocation altered or
how that works.

This patch may or may not be the way to go, but in any case
I'd like to get input on how we can ensure that more SQLAlchemy-specific
oslo.db features can be tested against multiple SQLAlchemy versions.
Note that even with this change, running the "sqla_07" environment
does in fact produce test failures, see http://paste.openstack.org/show/85263/;
so already oslo.db expects behaviors that are not present in
all SQLAlchemy versions listed in the common requirements.txt.

Change-Id: I4128272ce15b9e576d7b97b1adab4d5027108c7c
a1fd49f
@openstack-gerrit openstack-gerrit added a commit to openstack/openstack that referenced this issue Jul 6, 2014
@zzzeek zzzeek Updated openstack/openstack
Project: openstack/oslo.db  a1fd49fd9b726017de02856ab0e0dfe3751e2394

Test for distinct SQLAlchemy major releases

This change presents one way we might include test support
for oslo.db against specific SQLAlchemy major releases, currently
including the 0.7, 0.8, and 0.9 series.  As we will want to
begin including features within oslo.db that target advanced
and in some cases semi-public APIs within SQLAlchemy, it will
be important that we test these features against each major release,
as there may be variances between major revs as well as
version-specific approaches within oslo.

To accomplish this, I was not able to override "deps" alone,
as the SQLAlchemy revision within requirements.txt conflicts
with a hand-entered requirement, and due to pip's lack of
a dependency resolver (see pypa/pip#988
and pypa/pip#56) I instead overrode
"commands".  I don't know that this is the best approach, nor
do I know how the tox.ini file is accommodated by CI servers,
if these CI servers would need their tox invocation altered or
how that works.

This patch may or may not be the way to go, but in any case
I'd like to get input on how we can ensure that more SQLAlchemy-specific
oslo.db features can be tested against multiple SQLAlchemy versions.
Note that even with this change, running the "sqla_07" environment
does in fact produce test failures, see http://paste.openstack.org/show/85263/;
so already oslo.db expects behaviors that are not present in
all SQLAlchemy versions listed in the common requirements.txt.

Change-Id: I4128272ce15b9e576d7b97b1adab4d5027108c7c
f04960a
@cazino cazino added a commit to scality/ScalitySproxydSwift that referenced this issue Jul 10, 2015
@cazino cazino Can't have mock dependency both in test-requirements.txt and tox.ini,
otherwise pip crashes with 'Double requirement given' error.
That might be fixed in a future version of pip :
see pypa/pip#56
and pypa/pip#988
d7fd934
@cazino cazino added a commit to scality/ScalitySproxydSwift that referenced this issue Jul 10, 2015
@cazino cazino Packages update and install build-essential on Ubuntu
Add swit-2.3.0 target

Avoid to install anything except flake8 which is the only thing needed in the pep8 env
(this is how Swift does it : https://github.com/openstack/swift/blob/49e7c25876cf4d3bc9412443f881e8472e88e827/tox.ini)

Mock 1.1 (latest version as the time of writing) dropped py26 compatibility.
Can't have mock dependency both in test-requirements.txt and tox.ini,
otherwise pip crashes with 'Double requirement given' error.
That might be fixed in a future version of pip :
see pypa/pip#56
and pypa/pip#988
b8f4027
@cazino cazino referenced this issue in scality/ScalitySproxydSwift Jul 10, 2015
Merged

Fix tox job #145

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.