-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Capture difference that pip doesn't allow for install config details … #233
Conversation
…such as the package index to be supplied per project as it is for easy_install.
That is correct. A random project from PyPI should not be able to force you download packages from someplace else. The pip way of doing that is to have it at the top level (perhaps in a requirements file). |
So it's not really "per project index", it's more like "Package Provided Additional Index" or something. |
The case I'm up against right now is I have a company-internal project, and it won't ever be able to resolve its dependencies against PyPI, so I'd like some way to say "unless indicated otherwise, use this index to resolve dependencies." I'm not seeking for the project to force downloading from someplace else, but rather provide a default when "this package" is the thing being installed. It's not an additional index, but a preferred index for this package. The recommended usage in pip is unpredictable and involves a lot more typing. That is, you could put the index in a Additionally, that file may serve other purposes as well, maybe supplying requirements for a heroku install. Should it contain I sort of feel like pip should support pip.ini or similar at the project level, but I'm not certain of that. I'm mostly seeking here to capture this difference that affects our environment. |
Right, I'm just suggesting that the name "project" is ambiguous, if project == package then it makes sense, but that isn't always the case. While the |
Extra note, I've previously written about the distinction I have between abstract and concrete dependencies which influences how I view all of this :) |
Would a constraints file be able to address Jason's use case here?: https://pip.pypa.io/en/stable/user_guide/#constraints-files Currently that section just refers off to the Requirements File Format definition, but doesn't state how things like direct package references or VCS links are handled when supplied as constraints rather than as requirements. Alternatively, if this is just for documentation purposes, then I think the official pip-based answer for "How do I use a mixture of public & private packages for my private projects?" is to run your own internal index, mirror any upstream dependencies you need, and then configure your build and deployment environments to only pull from your private index. In terms of available options for doing that, devpi is a relevant Python specific repo manager, Pulp's Python plugin provides integration with a self-hosted general purpose repository manager, pinrepo is an S3 based one, while Artifactory is a high-quality-but-proprietary repo manager from JFrog. |
I think this goal is specifically contrary to what I'm trying to achieve and of something like devpi. We use devpi internally to host hundreds of packages, and by being the canonical index but (optionally) mirroring PyPI, the repositories there provide a flexible and manageable mechanism for providing non-public packages. If by direct dependencies, you mean that each dependency required by the project need to reference a specific repository, that would be grossly inefficient and difficult to manage compared to what we currently expect from devpi and easy_install (or pip install -i). I read your blog on concrete vs. abstract dependencies, and it makes a good point. What I'm seeking to do here is to maintain abstract dependencies in the install_requires file, but for the purposes of developing or running or installing a particular library, make those dependencies concrete by resolving them via the internal index, which is the known best place to resolve dependencies for this package (because we know they won't resolve in PyPI).
I don't see how constraints help at all. The user would still be expected to supply these parameters, so it has the same limitations of the requirements.txt file described above.
I'm not sure I follow. The concept of a separate package index for non-standard releases isn't unique to corporate development. For example, the merlinux group, which maintains devpi, keeps their own public-facing index and have the ability to distribute packages via select repositories in that index for purposes other than company-internal operations (though company-internal is probably the most meaningful example). So if I'm creating a new project, and it depends on a suite of packages that are not released in PyPI (or maybe never will be) and only published to an alternate repository, I'd like a way to declare that for this new project (a repo with a setup.py), pip should grab its dependencies from this alternate repository. For pip, it's possible to accomplish this goal at the virtualenv scope, and at the system scope, or with environment variables, but not at the project scope. I think I do now see why pip can't simply implement a mechanism to supply the index at the package level, because that would potentially alter the index url used to install that package's dependencies when the package were installed via another mechanism (such as from a dependency in another package). I agree you don't want a package uploaded to PyPI that when installed causes pip to pull its dependencies from another repository. All I'm seeking is a mechanism by which pip can infer an alternate index for installing packages (and possibly other configuration) when invoked from this project/package/directory. I would expect pip to ignore this setting when it's building an sdist for installation (and rely on whatever index/config was present when the command was invoked). This mechanism should be something that can stored in a file system or SCM repository (which is why environment variables or virtualenv-local files are inadequate) with the project. Is there a reason pip doesn't respect a pip.conf in the cwd (for invocations of pip from that directory)? |
What I'd like to do is be able to "configure your build and deployment environments" in the SCM, such that when I'm working on internal-project-a, it pulls dependencies from the internal repo, but when I change directories to public-project-b, it pulls dependencies from PyPI, and when I change directories to partner-project-c, it pulls dependencies from the partner's index... without having to configure separate build environments for each. |
Off the top of my head, I think the answer is just that nobody has actually written the code to do that. A quick search brings up pypa/pip#232 which Marcus agreed with the concept and I don't see any reason why such a thing couldn't exist as long as it wasn't respected when installing from something like PyPI but only when you do something like |
And because "repository" is ambiguous, I mean a remote location from which you're discovering and downloading packages, e.g. PyPI. |
I'd be okay with merging this if:
|
I've added a glossary entry for "per project index". Perhaps it could be called something else like "Private Index" or "Local Index". I'm not particular about the terminology.
I assert it is accurate and I have yet to see a demonstration otherwise. What more can be done? |
LGTM, merging. Any concerns from others can be addressed post-hoc. |
pypa#233) * Capture difference that pip doesn't allow for install config details such as the package index to be supplied per project as it is for easy_install. * Add definition for 'per project index'.
…such as the package index to be supplied per project as it is for easy_install.
Am I right in that for someone to use pip to install dependencies for a package, one must specify some parameters to get the package index? For example, with
easy_install
, one can invoke:in a project dir, and if that project specifies
easy_install.index-url = https://devpi.mygroup.org/
in setup.cfg, the installer will use that index. But in pip, to get equivalent behavior, one must supply the index URL in an environment variable or on the command line or in a system-wide config or in a virtualenv, but it's not possible to derive the preferred, relevant index URL from the context of the package (sdist or checkout)?