Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Capture difference that pip doesn't allow for install config details … #233

Merged
merged 2 commits into from
May 2, 2017
Merged

Capture difference that pip doesn't allow for install config details … #233

merged 2 commits into from
May 2, 2017

Conversation

jaraco
Copy link
Member

@jaraco jaraco commented May 10, 2016

…such as the package index to be supplied per project as it is for easy_install.

Am I right in that for someone to use pip to install dependencies for a package, one must specify some parameters to get the package index? For example, with easy_install, one can invoke:

easy_install .

in a project dir, and if that project specifies easy_install.index-url = https://devpi.mygroup.org/ in setup.cfg, the installer will use that index. But in pip, to get equivalent behavior, one must supply the index URL in an environment variable or on the command line or in a system-wide config or in a virtualenv, but it's not possible to derive the preferred, relevant index URL from the context of the package (sdist or checkout)?

…such as the package index to be supplied per project as it is for easy_install.
@dstufft
Copy link
Member

dstufft commented May 10, 2016

That is correct. A random project from PyPI should not be able to force you download packages from someplace else. The pip way of doing that is to have it at the top level (perhaps in a requirements file).

@dstufft
Copy link
Member

dstufft commented May 10, 2016

So it's not really "per project index", it's more like "Package Provided Additional Index" or something.

@jaraco
Copy link
Member Author

jaraco commented May 10, 2016

The case I'm up against right now is I have a company-internal project, and it won't ever be able to resolve its dependencies against PyPI, so I'd like some way to say "unless indicated otherwise, use this index to resolve dependencies." I'm not seeking for the project to force downloading from someplace else, but rather provide a default when "this package" is the thing being installed. It's not an additional index, but a preferred index for this package.

The recommended usage in pip is unpredictable and involves a lot more typing. That is, you could put the index in a requirements.txt file, but then the user has to know to type pip install -r requirements.txt. Maybe instead of using requirements.txt, one could use index.txt and then have various requirements.txt files or command invocations refer to it via -r index.txt. Still, it seems hard to give guidance on how this is done, whereas the easy_install mechanism is clear and predictable.

Additionally, that file may serve other purposes as well, maybe supplying requirements for a heroku install. Should it contain -e . or . or a copy of the dependencies declared in the package metadata?

I sort of feel like pip should support pip.ini or similar at the project level, but I'm not certain of that. I'm mostly seeking here to capture this difference that affects our environment.

@dstufft
Copy link
Member

dstufft commented May 10, 2016

Right, I'm just suggesting that the name "project" is ambiguous, if project == package then it makes sense, but that isn't always the case.

While the easy_install mechanism doesn't have much subtley, it's also a cause for concern when it's not a company-internal project. In this easy_install is too liberal and pip is too restrictive. Longer term the hope I have is we'll use "direct dependencies" from PEP 440 and PyPI will be taught to hard reject any direct dependencies, which will I think give a middle ground where company-internal things can point directly at stuff, but that doesn't end up adding the same capability to PyPI.

@dstufft
Copy link
Member

dstufft commented May 10, 2016

Extra note, I've previously written about the distinction I have between abstract and concrete dependencies which influences how I view all of this :)

@ncoghlan
Copy link
Member

Would a constraints file be able to address Jason's use case here?: https://pip.pypa.io/en/stable/user_guide/#constraints-files

Currently that section just refers off to the Requirements File Format definition, but doesn't state how things like direct package references or VCS links are handled when supplied as constraints rather than as requirements.

Alternatively, if this is just for documentation purposes, then I think the official pip-based answer for "How do I use a mixture of public & private packages for my private projects?" is to run your own internal index, mirror any upstream dependencies you need, and then configure your build and deployment environments to only pull from your private index.

In terms of available options for doing that, devpi is a relevant Python specific repo manager, Pulp's Python plugin provides integration with a self-hosted general purpose repository manager, pinrepo is an S3 based one, while Artifactory is a high-quality-but-proprietary repo manager from JFrog.

@jaraco
Copy link
Member Author

jaraco commented May 11, 2016

Longer term the hope I have is we'll use "direct dependencies" from PEP 440 and PyPI will be taught to hard reject any direct dependencies, which will I think give a middle ground where company-internal things can point directly at stuff, but that doesn't end up adding the same capability to PyPI.

I think this goal is specifically contrary to what I'm trying to achieve and of something like devpi. We use devpi internally to host hundreds of packages, and by being the canonical index but (optionally) mirroring PyPI, the repositories there provide a flexible and manageable mechanism for providing non-public packages. If by direct dependencies, you mean that each dependency required by the project need to reference a specific repository, that would be grossly inefficient and difficult to manage compared to what we currently expect from devpi and easy_install (or pip install -i).

I read your blog on concrete vs. abstract dependencies, and it makes a good point. What I'm seeking to do here is to maintain abstract dependencies in the install_requires file, but for the purposes of developing or running or installing a particular library, make those dependencies concrete by resolving them via the internal index, which is the known best place to resolve dependencies for this package (because we know they won't resolve in PyPI).

Would a constraints file be able to address Jason's use case here?: https://pip.pypa.io/en/stable/user_guide/#constraints-files

I don't see how constraints help at all. The user would still be expected to supply these parameters, so it has the same limitations of the requirements.txt file described above.

While the easy_install mechanism doesn't have much subtley, it's also a cause for concern when it's not a company-internal project.

I'm not sure I follow. The concept of a separate package index for non-standard releases isn't unique to corporate development. For example, the merlinux group, which maintains devpi, keeps their own public-facing index and have the ability to distribute packages via select repositories in that index for purposes other than company-internal operations (though company-internal is probably the most meaningful example).

So if I'm creating a new project, and it depends on a suite of packages that are not released in PyPI (or maybe never will be) and only published to an alternate repository, I'd like a way to declare that for this new project (a repo with a setup.py), pip should grab its dependencies from this alternate repository.

For pip, it's possible to accomplish this goal at the virtualenv scope, and at the system scope, or with environment variables, but not at the project scope.

I think I do now see why pip can't simply implement a mechanism to supply the index at the package level, because that would potentially alter the index url used to install that package's dependencies when the package were installed via another mechanism (such as from a dependency in another package). I agree you don't want a package uploaded to PyPI that when installed causes pip to pull its dependencies from another repository.

All I'm seeking is a mechanism by which pip can infer an alternate index for installing packages (and possibly other configuration) when invoked from this project/package/directory. I would expect pip to ignore this setting when it's building an sdist for installation (and rely on whatever index/config was present when the command was invoked). This mechanism should be something that can stored in a file system or SCM repository (which is why environment variables or virtualenv-local files are inadequate) with the project.

Is there a reason pip doesn't respect a pip.conf in the cwd (for invocations of pip from that directory)?

@jaraco
Copy link
Member Author

jaraco commented May 11, 2016

Alternatively, if this is just for documentation purposes, then I think the official pip-based answer for "How do I use a mixture of public & private packages for my private projects?" is to run your own internal index, mirror any upstream dependencies you need, and then configure your build and deployment environments to only pull from your private index.

What I'd like to do is be able to "configure your build and deployment environments" in the SCM, such that when I'm working on internal-project-a, it pulls dependencies from the internal repo, but when I change directories to public-project-b, it pulls dependencies from PyPI, and when I change directories to partner-project-c, it pulls dependencies from the partner's index... without having to configure separate build environments for each.

@dstufft
Copy link
Member

dstufft commented May 11, 2016

Is there a reason pip doesn't respect a pip.conf in the cwd (for invocations of pip from that directory)?

Off the top of my head, I think the answer is just that nobody has actually written the code to do that. A quick search brings up pypa/pip#232 which Marcus agreed with the concept and I don't see any reason why such a thing couldn't exist as long as it wasn't respected when installing from something like PyPI but only when you do something like pip install path/to/directory/ or invoke pip with it in ./pip.conf or so. IOW, totally fine when the file is part of some directory you have on your computer, but not when it comes from a repository itself.

@dstufft
Copy link
Member

dstufft commented May 11, 2016

And because "repository" is ambiguous, I mean a remote location from which you're discovering and downloading packages, e.g. PyPI.

@theacodes
Copy link
Member

I'd be okay with merging this if:

  1. We define what a "per project index" is in the documentation (perhaps in the glossary)
  2. We make sure the pip column is accurate.

@jaraco
Copy link
Member Author

jaraco commented May 2, 2017

I've added a glossary entry for "per project index". Perhaps it could be called something else like "Private Index" or "Local Index". I'm not particular about the terminology.

We make sure the pip column is accurate.

I assert it is accurate and I have yet to see a demonstration otherwise. What more can be done?

@theacodes
Copy link
Member

LGTM, merging. Any concerns from others can be addressed post-hoc.

@theacodes theacodes merged commit c18e1ee into pypa:master May 2, 2017
ncoghlan pushed a commit to ncoghlan/packaging.python.org that referenced this pull request Jun 24, 2017
pypa#233)

* Capture difference that pip doesn't allow for install config details such as the package index to be supplied per project as it is for easy_install.

* Add definition for 'per project index'.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants