Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

refactor pip's use of "#egg=" #1289

Open
qwcode opened this issue Nov 4, 2013 · 30 comments
Open

refactor pip's use of "#egg=" #1289

qwcode opened this issue Nov 4, 2013 · 30 comments
Labels
C: dependency resolution About choosing which dependencies to install C: direct url Direct URL references (PEP 440, PEP 508, PEP 610) C: unnamed reqs Handling and processing unnamed requirements state: awaiting PR Feature discussed, PR is needed type: enhancement Improvements to functionality

Comments

@qwcode
Copy link
Contributor

qwcode commented Nov 4, 2013

I suspect pip doesn't really need the"egg=" value (assuming we make some modifications)

setuptools used it in two cases:

  • installing single py files (which pip doesn't support I guess?)
  • installing from svn urls

pip's support is like so:

  1. officially documented for vcs urls (it determines the src or build dir, depending on whether it's editable, and also determine's it's identification in the dependency resolution process.)
  2. although not documented, I've seen users (including myself) tack "egg=" onto archive urls (that currently don't get their sdist filenames parsed, as you would expect (see The --build-dir options doesn't work for local archive installs #804), so using "egg=" for them makes them have a set build dir)

3 changes are needed, to make pip not require "egg=":

  1. do away with the --no-install/--no-download "workflow" (see refactor --no-install/--no-download/--download (and use mkdtemp build dirs) #906), which assumes fixed build dirs, and then drop fixed build dirs all together.
  2. make it a tested feature that non-specifier requirements (i.e. vcs urls, archive urls and paths) get processed first (i.e. unpacked and have their name/version determined), so they can properly participate in the dependency resolution process. this is currently true but not very prominent in the code and not tested afaict.
  3. when doing editable intalls from vcs urls, use some other approach to determine the "src" directory. Maybe just unpack it to /tmp, then determine the name, then move it back into src.

even if it turns out #egg=is not used in any way by pip, it may be important to document a convention as visual aid (possibly as #pkg=, see #1265), when the vcs url would otherwise contain no clear indication as to what it is. I know for example that various CM modules for chef and puppet make use of this indicator

@Ivoz
Copy link
Contributor

Ivoz commented Mar 19, 2014

@qwcode could you link to an example puppet / chef module that makes use of that currently?

@qwcode
Copy link
Contributor Author

qwcode commented Mar 20, 2014

@qwcode
Copy link
Contributor Author

qwcode commented Mar 20, 2014

cc @coderanger

@coderanger
Copy link

To be fair, the reason I've not merged that PR is because I don't want to be parsing pip-specific URL syntax.

@qwcode
Copy link
Contributor Author

qwcode commented Mar 20, 2014

the module we use at work uses a python helper module (that uses pip to handle the parsing), so no rewriting of pip in ruby.

but regardless, the issue is whether the vcs requirement form should require or at least allow an identifier tag like #egg= (or #dist=).

if you're only thinking about what pip needs to do an install, then you could argue it's not necessary (and worse actually misleads pip). you could just say that pip should always be cloning and using the real name and version from setup.py to determine how it gets processed relative to the other requirements.

but for a CM system that might want to support the vcs url form as a valid way to declare a resource (i.e. not just package specifiers 'pkg==version'), then having an identifier like #egg==name-version is pretty handy to quickly validate whether the resource is currently satisfied on the system.

@Ivoz
Copy link
Contributor

Ivoz commented Mar 21, 2014

So, I've realised that although pip might deprecate making use of that field, there's no-one saying we should disallow the fragment being in the url, right?

@qwcode
Copy link
Contributor Author

qwcode commented Mar 21, 2014

but I'm thinking it's more than just allowing arbitrary fragments.
we should document the convention.

moreover, pip might start using the conventions itself for fast determination of fulfillment:

  • if a '@' ref is present that's a literal version/commit, check that before updating the src clone.
  • or, if #[egg|dist]=<name>-<version> is present, check that name and version before re-updating the clone.

@qwcode qwcode changed the title deprecate pip's "egg=" syntax refactor pip's use of "#egg=" Mar 21, 2014
@dstufft
Copy link
Member

dstufft commented Mar 22, 2017

I think that this is still an issue, I'm not sure it makes sense to get rid of it, but at the very least we should rename it since we have deprecated eggs and the terminology is confusing to people.

@xavfernandez
Copy link
Member

I think it should (very) slowly be deprecated in favor of the PEP-508 blessed pkg_name @ url.
(or am I missing something ?)

@dstufft
Copy link
Member

dstufft commented Mar 22, 2017

Yea that's likely a reasonable outcome.

@pradyunsg
Copy link
Member

So... the actionable part of this would be to deprecate #egg= in favour of the PEP 508 syntax?

Would this be a documentation only change?

@pradyunsg
Copy link
Member

Ping?

@xavfernandez
Copy link
Member

Maybe a warning to suggest the new PEP 508 syntax ? And a very long deprecation period...

@pradyunsg
Copy link
Member

Cool. Let's do this then.

@pradyunsg pradyunsg added the good first issue A good item for first time contributors to work on label Dec 30, 2018
@pradyunsg
Copy link
Member

This issue is a good starting point for anyone who wants to help out with pip's development -- it's simple and the process of fixing this should be a good introduction to pip's development workflow. See the discussion above to understand what the desired fix is.

Feel free to give me a mention "@pradyunsg" if you have any doubts or questions.

@pradyunsg pradyunsg removed the good first issue A good item for first time contributors to work on label Apr 19, 2019
@sbidoul sbidoul added C: direct url Direct URL references (PEP 440, PEP 508, PEP 610) and removed C: vcs pip's interaction with version control systems like git, svn and bzr labels Apr 13, 2020
@gutsytechster
Copy link
Contributor

So, the solution to it is to deprecate the use of #egg= format with a deprecation cycle? May I know how do we perform a deprecation cycle?

@uranusjr
Copy link
Member

Merging in #9429, which is basically another reason why #egg= should be removed (internally) and discouraged.

Proposed solution from #9429 (comment):

  • If a requirement’s does not contain an egg fragment, use it as it.
  • If a requirement contains an egg fragment, and is not in PEP 508 format, convert it into PEP 508 with a warning.
  • If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing an identical package name, from the egg fragment with a warning.
  • If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing a different package name, fail with an error (requirement invalid).

@uranusjr uranusjr added the state: awaiting PR Feature discussed, PR is needed label Jan 10, 2021
@sbidoul
Copy link
Member

sbidoul commented Jan 10, 2021

I've given some more thought to this, and I think the first thing to do is to support -e "name @ vcs url". Indeed, currently the #egg= fragment is mandatory for -e vcs-url, because pip needs a name to create the checkout directory under src. When that is done we should have alternatives to egg fragments for all use cases.

@uranusjr
Copy link
Member

I opened #9471 for -e "NAME @ URL" support.

@piotr-dobrogost
Copy link

If a PEP 508 direct URL requirement’s URL part contains an egg fragment containing a different package name, fail with an error (requirement invalid).

Wouldn't ignoring fragment with a warning be more sensible here?
Just treat PEP 508 direct URL as overriding old approach with an egg fragment and keep going.

@uranusjr
Copy link
Member

uranusjr commented Jun 22, 2021

Cross-linking #10002 (comment) for my thoughts about warnings (tl;dr I don’t think we should do this for now).

@DiddiLeija
Copy link
Member

Hello. It seems like this issue has been fragmented into many issues that were already resolved. Can we close this one?

@pradyunsg
Copy link
Member

This is still definitely relevant. I've poked at pypa/packaging#264 again, to get the ball rolling again for #9471.

@woodruffw
Copy link
Member

Cross-referencing: there's a discussion here on removing egg= fragments entirely: https://discuss.python.org/t/killing-off-the-egg-fragment-once-and-for-all/21660

@lrq3000
Copy link

lrq3000 commented Apr 14, 2023

It was suggested in the forum's discussion to remove the egg= fragment entirely on the premise that the @ syntax has equivalent functionalities, but this is not the case. I would like to suggest to not deprecate egg= fragments entirely until the @ syntax is really equivalent, which it currently is not because it lacks support for PEP 508 URLs as editables. In particular, this means that if the egg= fragment is removed in the current state, it will be impossible to specify extras on an editable PEP508 URL, eg, the following is currently impossible:

pip install --upgrade --editable "mypkg[dist] @ git+https://github.com/username/mypkg" --verbose

The only way it can work currently is as follows:

pip install --upgrade --editable "git+https://github.com/username/mypkg#egg=mypkg[dist]" --verbose

To my knowledge, there is no other way than using an egg fragment to specify an extra, so if this gets removed before an equivalent implementation in @, this would be a net loss in functionality.

@uranusjr
Copy link
Member

The PEP 508 syntax should provide equivalent functionalities, the fact that --editable does not support the syntax is an implementation issue that should be fixed before/when the egg fragment syntax is dropped, not a design issue.

@lrq3000
Copy link

lrq3000 commented May 10, 2023

Cross-referencing #11617 since that's the one that is linked to in the deprecation warning for the issue I mentioned above.

@lrq3000
Copy link

lrq3000 commented May 10, 2023

Ah, it seems that it's not possible to add cross-references on locked issues/PRs. Then I would suggest to change the deprecation warning:

Using pip 23.1.2 from /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip (python 3.11)
DEPRECATION: myproject.git#egg=myproject[test] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at `https://github.com/pypa/pip/issues/1161`

To:

Using pip 23.1.2 from /Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/pip (python 3.11)
DEPRECATION: myproject.git#egg=myproject[test] contains an egg fragment with a non-PEP 508 name pip 25.0 will enforce this behaviour change. A possible replacement is to use the req @ url syntax, and remove the egg fragment. Discussion can be found at `https://github.com/pypa/pip/issues/1161` and `https://github.com/pypa/pip/issues/1289`

Ie, adding a link to the current issue, since #1161 has no relevant information for the case of --editable + egg fragment.

@mr-c
Copy link

mr-c commented May 3, 2024

Hello from 2024, will installing an editable from a VCS URI really be removed in pip 25?
Many of the projects I maintain have a release testing script where we want to have an editable install of a VCS (git) checkout, including the specification of [extras].

https://github.com/common-workflow-language/cwl-utils/blob/67dbfa590c733b6ad5bf9139866e42cb5cd02aa2/release-test.sh#L64
https://github.com/common-workflow-language/schema_salad/blob/b9f2d76b4accfb633aab5b6bdb42b3fe93d5f39e/release-test.sh#L64
https://github.com/common-workflow-language/cwltool/blob/5e41bda52c8b24643aa4b4de86172365bd1c8ff7/release-test.sh#L69

Thanks

@pfmoore
Copy link
Member

pfmoore commented May 3, 2024

To my knowledge, there is no other way than using an egg fragment to specify an extra

Just to clarify, extras are allowed in (URL specifiers)[https://packaging.python.org/en/latest/specifications/dependency-specifiers/]:

url_req       = name wsp* extras? wsp* urlspec wsp+ quoted_marker?

will installing an editable from a VCS URI really be removed in pip 25?

As noted (above)[https://github.com//issues/1289#issuecomment-1510735040], support for the url_req format in editable installs is an implementation issue. We can't guarantee what will happen over the next year, but if we don't implement that support, we'll likely just update the deprecation to push it back a bit.

But at some point, you will need to switch from #egg to url_req. When that point is depends on resources at pip's end, though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
C: dependency resolution About choosing which dependencies to install C: direct url Direct URL references (PEP 440, PEP 508, PEP 610) C: unnamed reqs Handling and processing unnamed requirements state: awaiting PR Feature discussed, PR is needed type: enhancement Improvements to functionality
Projects
None yet
Development

No branches or pull requests

16 participants