Forcing order of installation

[themanifold]

Hello,

I realise that this issue might be similar to #3299 but I think it's different enough to talk about.

At the moment, I have a setup.py that contains something that looks like:

# ...typical setuptool import...
setup(
    name='somemod',
    dependency_links=['svn+svn://some.svn.path#egg=somemod'],
    install_requires=['somemod'],
)       

If I now try and install somemod with pip:
pip install . -vvv --process-dependency-links

I see that pip shows something like:

Collecting somemod
    2 location(s) to search for versions of somemod:
    * https://pypi.python.org/simple/somemod/
    * svn+svn://some.svn.path#egg=somemod

The question is this:

Setting aside opinions on dependency_links, is it possible to set the order in which pip searches the locations? At the moment, it appears that it always searches pypi.python.org first. I would like it to search the svn path first (for security reasons).

Many thanks.

[themanifold]

I've found the bit of code that intentionally forces the order of this list, but I don't see any way to changing this apart from forking the code in private: https://github.com/pypa/pip/blob/develop/pip/index.py#L423

[themanifold]

This is certainly a use case that I would like. What are people's feelings on myself trying to submit a pull request for it?

[pfmoore]

How are "security reasons" relevant here? If all you do is force the order, then pip will search PyPI as well, so how did you gain? If you want to disallow PyPI searches, you can use --no-index.

If the version in svn is later than any on PyPI, it will be picked. if not, it won't. If the latest version on PyPI is the same as the version number in svn, then either can be picked, but they should be identical (because that's what having the same version number means) and so why would you care?

If you want to pick the version in svn in preference to those on PyPI, give it a higher version number.

[themanifold]

When I refer to "security reasons", I'm referring to the fact that if I have some internal code base I will end up exposing its namespace to the PyPI web server when pip looks it up.

For my particular case, I know that any svn module I have will never be on PyPI, so I don't want to look for those modules on PyPI. But any other modules will be on PyPI and so for those, I want to look at PyPI first.

[pfmoore]

When I refer to "security reasons", I'm referring to the fact that if I have some internal code base I will end up exposing its namespace to the PyPI web server when pip looks it up.

??? Do you mean that your concern is that pip will look up the name of your internal module on PyPI? That's not something you can stop (except by using pip on your internal modules in a separate invocation using --no-index and manually managing any dependencies.

Even if you altered the order pip searches the indexes, it will always scan all possible locations in the process of looking for the newest version. That's just how pip works, sorry. (I guess there could be an option to say to pip "assume that requirement XXX will never be on the following indexes" but that doesn't exist at the moment, and I suspect that the use case would be too rare to justify the extra complexity it would add to the code - but you're welcome to try implementing it if you think it would help)

[themanifold]

Do you mean that your concern is that pip will look up the name of your internal module on PyPI?

Yes, that's right.

Even if you altered the order pip searches the indexes, it will always scan all possible locations in the process of looking for the newest version. That's just how pip works, sorry.

Thanks for clarifying.

there could be an option to say to pip "assume that requirement XXX will never be on the following indexes" but that doesn't exist at the moment, and I suspect that the use case would be too rare to justify the extra complexity it would add to the code - but you're welcome to try implementing it if you think it would help

I'm up for trying to implement this. I wanted to get someone's opinion on whether the idea would be taken seriously before I invested my time. From your response, I'm assuming that it would?

[themanifold]

For the time being anyway, I'm reduced to monkey patching /usr/local/bin/pip so that the pip._vendor.requests.Session.get method will induce a pip._vendor.requests.exceptions.HTTPError on regular expression match to my namespace.

[xavfernandez]

I suspect that the use case would be too rare to justify the extra complexity it would add to the code

So do I 😐

[pfmoore]

I'm up for trying to implement this. I wanted to get someone's opinion on whether the idea would be taken seriously before I invested my time. From your response, I'm assuming that it would?

Personally, I'd be reluctant to merge such a change. My reasons:

1. It seems like an incredibly obscure use case, so pretty much any cost in maintainability would outweigh the benefits. If other people chime in to say they have the same issue, that would affect my view here.
2. There's a distinct possibility of obscure user errors (if you told pip to ignore PyPI for the wrong things, you could easily get weird results). Good documentation might fix this, but I suspect it would be hard to document it well without making the explanation disproportionately long given the likelihood that anyone would need the option.
3. I have no idea how we'd ensure going forward that changes to pip didn't break this functionality. Good tests for the feature would help here.
4. I don't have the environment to test this, so I'd have to accept it based purely on desk-checking the code.
5. As a security feature, it's likely crucial that it's implemented correctly (if a bug in the implementation exposed your package namespace, I guess this would be considered serious) so it probably needs a reasonably thorough review. So to merge, it should get approval from a few of the pip developers, which may be hard to get given the limited use case).

[offero]

I have the exact same use case. I want to be able to look only in my local network repository for my packages that will not be on pypi and let pip look at pypi for all other packages. It is very slow when querying pypi for packages that won't be there.

[pfmoore]

Why is it slow querying PyPI? It shouldn't be. One lookup of the simple index (which will have already been looked up for the things that are on PyPI and so will be from the local cache) and you're done.

[offero]

Because it will look for packages that are not there, and it is actually pretty slow in doing so.

[pfmoore]

Then we should probably speed up looking for packages that aren't there. As I said, I can't see why it should be slow. That's a much more generally useful solution than providing a (fragile) option that will let people manually optimise something that shouldn't need optimising in the first place.

[Ivoz]

Returning an error from a non-existent package is one of the faster pip networked operations, for me.

 1 tmp-16e121d239d971e ivo·ivosung ~/.virtualenvs/tmp-16e121d239d971e » time pip install nonexistentpackage
Collecting nonexistentpackage
  Could not find a version that satisfies the requirement nonexistentpackage (from versions: )
No matching distribution found for nonexistentpackage

real   0m2.092s
user   0m0.363s
sys    0m0.030s

From Australia

If you really really want things to be speedy you can separate things out into two requirements files, one using --no-index and one not...

[offero]

Doesn't work for packages with dependencies on packages in PyPI.