Allow "double requirements" if versions don't conflict #56

Closed
vbabiy opened this Issue Mar 15, 2011 · 15 comments

Comments

Projects
None yet
@vbabiy
Contributor

vbabiy commented Mar 15, 2011

I'd like to be able to compose a requirements file that uses the -r option to install several other packages by their own requirements files.

But in this use scenario, it's possible - even likely - that these separate packages may have overlapping requirements.

This works fine when installing each of them with a separate pip command, as long as you don't hit a version conflict; but when using one master requirements file that includes the others, you'd get the "Double requirement given" error.

I'd like to propose that it should not be an error unless there are version conflicts.

That may be tricky, as it means computing the union of the allowed versions, and pkg_resources confuses me so I'm not sure if there's a right way to do it.


@slinkp

This comment has been minimized.

Show comment
Hide comment

slinkp commented Aug 4, 2011

Related: #174

@muelli

This comment has been minimized.

Show comment
Hide comment
@muelli

muelli Jul 18, 2012

My usecase is to have hierarchical requirement files, i.e. "running.txt" and "testing.txt" where "running" would only contain the bare minimum to make it run, i.e. "django>=1.3" and "testing" would need "django>=1.4" because some parts in the testing process use a newer API which is only available in the more recent version.
I'd like to be able to put "django>1.3" in my "running.txt" and then "-r running.txt django>=1.4" in my testing.txt

muelli commented Jul 18, 2012

My usecase is to have hierarchical requirement files, i.e. "running.txt" and "testing.txt" where "running" would only contain the bare minimum to make it run, i.e. "django>=1.3" and "testing" would need "django>=1.4" because some parts in the testing process use a newer API which is only available in the more recent version.
I'd like to be able to put "django>1.3" in my "running.txt" and then "-r running.txt django>=1.4" in my testing.txt

@jacinda

This comment has been minimized.

Show comment
Hide comment
@jacinda

jacinda Aug 21, 2013

I have the same use case as muelli.

jacinda commented Aug 21, 2013

I have the same use case as muelli.

@guettli

This comment has been minimized.

Show comment
Hide comment
@guettli

guettli Sep 19, 2013

would be nice to have this feature.

guettli commented Sep 19, 2013

would be nice to have this feature.

@jgumbley

This comment has been minimized.

Show comment
Hide comment

jgumbley commented Nov 7, 2013

+1

@DXist

This comment has been minimized.

Show comment
Hide comment

DXist commented Dec 26, 2013

+1

@integricho

This comment has been minimized.

Show comment
Hide comment

+1

@carlosviol

This comment has been minimized.

Show comment
Hide comment

+1

@tboggs

This comment has been minimized.

Show comment
Hide comment

tboggs commented Mar 6, 2014

+1

@Gr1N

This comment has been minimized.

Show comment
Hide comment

Gr1N commented Mar 14, 2014

+1

@GheRivero

This comment has been minimized.

Show comment
Hide comment
@GheRivero

GheRivero Mar 22, 2014

+2, Approved :)

+2, Approved :)

@qwcode qwcode self-assigned this Mar 27, 2014

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Mar 27, 2014

Contributor

btw, I've started on this. will try to get a PR posted in a few days. involves some refactor that's needed to make things clearer about our dependency resolution.

Contributor

qwcode commented Mar 27, 2014

btw, I've started on this. will try to get a PR posted in a few days. involves some refactor that's needed to make things clearer about our dependency resolution.

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode Apr 6, 2014

Contributor

I admit to not reading this issue correctly before. I'm closing this as a dupe to #988, which is our main issue for adding a proper resolver to pip. I just updated the description to cover resolving top-level requirements.

if anyone adding +1's here are actually concerned about literal duplicates across multiple requirement files, see #993. that is actually the problem I mentioned above that I was starting on.

Contributor

qwcode commented Apr 6, 2014

I admit to not reading this issue correctly before. I'm closing this as a dupe to #988, which is our main issue for adding a proper resolver to pip. I just updated the description to cover resolving top-level requirements.

if anyone adding +1's here are actually concerned about literal duplicates across multiple requirement files, see #993. that is actually the problem I mentioned above that I was starting on.

@piotr-dobrogost

This comment has been minimized.

Show comment
Hide comment
@piotr-dobrogost

piotr-dobrogost May 21, 2014

See issue #1795
@qwcode
I don't think it's a duplicate of issue #988 as here the aim is to just "merge" all conditions of the same top-level package and it seems it's orthogonal to general problem of dependency management which is the subject of issue #988. If you agree this should be reopened and taken into consideration when discussing Requirements 2.0 in issue #1795

See issue #1795
@qwcode
I don't think it's a duplicate of issue #988 as here the aim is to just "merge" all conditions of the same top-level package and it seems it's orthogonal to general problem of dependency management which is the subject of issue #988. If you agree this should be reopened and taken into consideration when discussing Requirements 2.0 in issue #1795

@qwcode

This comment has been minimized.

Show comment
Hide comment
@qwcode

qwcode May 24, 2014

Contributor

@piotr-dobrogost : I only closed this as a dupe after I updated the description of #988 to cover the problem of consolidating multiple top-level requirements for a project. so in a very literal sense, it is a dupe now. consolidating multiple top-level requirements seems similar to consolidating multiple in the dependency tree. I want #988 ticket to hold all the big fundamental changes to our resolver logic, which will likely get fixed at one time together.

Contributor

qwcode commented May 24, 2014

@piotr-dobrogost : I only closed this as a dupe after I updated the description of #988 to cover the problem of consolidating multiple top-level requirements for a project. so in a very literal sense, it is a dupe now. consolidating multiple top-level requirements seems similar to consolidating multiple in the dependency tree. I want #988 ticket to hold all the big fundamental changes to our resolver logic, which will likely get fixed at one time together.

openstack-gerrit pushed a commit to openstack/oslo.db that referenced this issue Jul 6, 2014

Test for distinct SQLAlchemy major releases
This change presents one way we might include test support
for oslo.db against specific SQLAlchemy major releases, currently
including the 0.7, 0.8, and 0.9 series.  As we will want to
begin including features within oslo.db that target advanced
and in some cases semi-public APIs within SQLAlchemy, it will
be important that we test these features against each major release,
as there may be variances between major revs as well as
version-specific approaches within oslo.

To accomplish this, I was not able to override "deps" alone,
as the SQLAlchemy revision within requirements.txt conflicts
with a hand-entered requirement, and due to pip's lack of
a dependency resolver (see pypa/pip#988
and pypa/pip#56) I instead overrode
"commands".  I don't know that this is the best approach, nor
do I know how the tox.ini file is accommodated by CI servers,
if these CI servers would need their tox invocation altered or
how that works.

This patch may or may not be the way to go, but in any case
I'd like to get input on how we can ensure that more SQLAlchemy-specific
oslo.db features can be tested against multiple SQLAlchemy versions.
Note that even with this change, running the "sqla_07" environment
does in fact produce test failures, see http://paste.openstack.org/show/85263/;
so already oslo.db expects behaviors that are not present in
all SQLAlchemy versions listed in the common requirements.txt.

Change-Id: I4128272ce15b9e576d7b97b1adab4d5027108c7c

openstack-gerrit added a commit to openstack/openstack that referenced this issue Jul 6, 2014

Updated openstack/openstack
Project: openstack/oslo.db  a1fd49fd9b726017de02856ab0e0dfe3751e2394

Test for distinct SQLAlchemy major releases

This change presents one way we might include test support
for oslo.db against specific SQLAlchemy major releases, currently
including the 0.7, 0.8, and 0.9 series.  As we will want to
begin including features within oslo.db that target advanced
and in some cases semi-public APIs within SQLAlchemy, it will
be important that we test these features against each major release,
as there may be variances between major revs as well as
version-specific approaches within oslo.

To accomplish this, I was not able to override "deps" alone,
as the SQLAlchemy revision within requirements.txt conflicts
with a hand-entered requirement, and due to pip's lack of
a dependency resolver (see pypa/pip#988
and pypa/pip#56) I instead overrode
"commands".  I don't know that this is the best approach, nor
do I know how the tox.ini file is accommodated by CI servers,
if these CI servers would need their tox invocation altered or
how that works.

This patch may or may not be the way to go, but in any case
I'd like to get input on how we can ensure that more SQLAlchemy-specific
oslo.db features can be tested against multiple SQLAlchemy versions.
Note that even with this change, running the "sqla_07" environment
does in fact produce test failures, see http://paste.openstack.org/show/85263/;
so already oslo.db expects behaviors that are not present in
all SQLAlchemy versions listed in the common requirements.txt.

Change-Id: I4128272ce15b9e576d7b97b1adab4d5027108c7c

cazino added a commit to scality/ScalitySproxydSwift that referenced this issue Jul 10, 2015

Can't have mock dependency both in test-requirements.txt and tox.ini,
otherwise pip crashes with 'Double requirement given' error.
That might be fixed in a future version of pip :
see pypa/pip#56
and pypa/pip#988

cazino added a commit to scality/ScalitySproxydSwift that referenced this issue Jul 10, 2015

Packages update and install build-essential on Ubuntu
Add swit-2.3.0 target

Avoid to install anything except flake8 which is the only thing needed in the pep8 env
(this is how Swift does it : https://github.com/openstack/swift/blob/49e7c25876cf4d3bc9412443f881e8472e88e827/tox.ini)

Mock 1.1 (latest version as the time of writing) dropped py26 compatibility.
Can't have mock dependency both in test-requirements.txt and tox.ini,
otherwise pip crashes with 'Double requirement given' error.
That might be fixed in a future version of pip :
see pypa/pip#56
and pypa/pip#988

@cazino cazino referenced this issue in scality/ScalitySproxydSwift Jul 10, 2015

Merged

Fix tox job #145

openstack-gerrit pushed a commit to openstack/deb-python-oslo.db that referenced this issue Aug 17, 2016

Test for distinct SQLAlchemy major releases
This change presents one way we might include test support
for oslo.db against specific SQLAlchemy major releases, currently
including the 0.7, 0.8, and 0.9 series.  As we will want to
begin including features within oslo.db that target advanced
and in some cases semi-public APIs within SQLAlchemy, it will
be important that we test these features against each major release,
as there may be variances between major revs as well as
version-specific approaches within oslo.

To accomplish this, I was not able to override "deps" alone,
as the SQLAlchemy revision within requirements.txt conflicts
with a hand-entered requirement, and due to pip's lack of
a dependency resolver (see pypa/pip#988
and pypa/pip#56) I instead overrode
"commands".  I don't know that this is the best approach, nor
do I know how the tox.ini file is accommodated by CI servers,
if these CI servers would need their tox invocation altered or
how that works.

This patch may or may not be the way to go, but in any case
I'd like to get input on how we can ensure that more SQLAlchemy-specific
oslo.db features can be tested against multiple SQLAlchemy versions.
Note that even with this change, running the "sqla_07" environment
does in fact produce test failures, see http://paste.openstack.org/show/85263/;
so already oslo.db expects behaviors that are not present in
all SQLAlchemy versions listed in the common requirements.txt.

Change-Id: I4128272ce15b9e576d7b97b1adab4d5027108c7c

@encbladexp encbladexp referenced this issue in smarthomeNG/smarthome Aug 24, 2017

Closed

Improved and simplified requirements.txt handling. #226

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment