Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pip accepts requirements with extras that don't conform to PEP 508 #8288

Closed
pfmoore opened this issue May 21, 2020 · 10 comments · Fixed by #8424
Closed

Pip accepts requirements with extras that don't conform to PEP 508 #8288

pfmoore opened this issue May 21, 2020 · 10 comments · Fixed by #8424
Labels
state: awaiting PR Feature discussed, PR is needed type: deprecation Related to deprecation / removal.

Comments

@pfmoore
Copy link
Member

pfmoore commented May 21, 2020

Description
See #8285. The form setuptools_scm>=3.5[toml] does not conform to PEP 508. Pip accepts it, presumably for historical reasons.

We should deprecate and ultimately reject this form.

Expected behavior
Pip should only accept standard (PEP 508) formats for requirements.

How to Reproduce
See the linked issue.

@triage-new-issues triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label May 21, 2020
@pradyunsg pradyunsg added state: awaiting PR Feature discussed, PR is needed type: deprecation Related to deprecation / removal. labels May 21, 2020
@triage-new-issues triage-new-issues bot removed the S: needs triage Issues/PRs that need to be triaged label May 21, 2020
@jku
Copy link
Contributor

jku commented Jun 10, 2020

I'm trying to get familiar with pip sources and can take a look at this.

I'll probably need a bit of advice along the way though: I've looked at this enough to see that the requirements/specifier handling is a bit redundant: it gets parsed several times with slightly different results and it's not 100% clear which of these parsing passes should check for this. More importantly it looks like the non-deprecated form base[extra]>=1.0 gets mis-parsed in two places...

I'll make a potential fix for just this issue and then comment on the other weirdness once I understand it better.

@jku
Copy link
Contributor

jku commented Jun 10, 2020

Actually commenting now on the strange behaviour I'm seeing so I remember later:

  • parse_req_from_line() returns no extras for input like "base[extra]>=1.0" in the returned RequirementParts
>>> req_parts = parse_req_from_line("base[extra]>=1.0", None)
>>> print (req_parts.requirement.extras, req_parts.extras)
{'extra'} set()
>>> req_parts = parse_req_from_line("base>=1.0[extra]", None)
>>> print (req_parts.requirement.extras, req_parts.extras)
set() {'extra'}
  • that RequirementParts is used to build an InstallRequirement: that will however end up having extras (even if it was explicitly given an empty extras set as input)
  • parse_req_from_line() uses _strip_extras() that is clearly buggy in context of "base[extra]>=1.0" input -- maybe this shouldn't be executed for plain requirement specifiers?

@jku
Copy link
Contributor

jku commented Jun 11, 2020

I think the major confusion with the methods in question is that there seems to be undocumented redundancy:

  • what does it mean when a RequirementParts or InstallRequirement has both "extras" and "requirement.extras"
  • what does it mean when a RequirementParts or InstallRequirement has both "markers" and "requirement.marker"

@jku
Copy link
Contributor

jku commented Jun 14, 2020

So InstallRequirements does this:

    if extras:
        self.extras = extras
    elif req:
        self.extras = {
            pkg_resources.safe_extra(extra) for extra in req.extras
        }
    else:
        self.extras = set()

I still can't find an explanation -- why are there two ways to define extras?

EDIT: after lots of git blame I think this is the origin of multiple ways to give extras to InstallRequirement: #1236. Still don't understand why.

@uranusjr
Copy link
Member

uranusjr commented Jun 15, 2020

Man, I think we should just rewrite RequirementParts and get rid of parts.extras and parts.marker entirely. It is only used by parse_req_from_line and parse_req_from_editable, both are well-scoped and can be replaced safely.

I am not entirely sure, but assume most of the extras and marker cruft here are due to history baggage. InstallRequirement and its various constructors predate PEP 508 and the packaging library. pip had its own parsing logic at the time, and was refactored to use packaging afterwards, but a lot of the logic lingers to avoid breaking stuff (and because nobody really have much time to do the cleanups).

@pradyunsg
Copy link
Member

pradyunsg commented Jun 15, 2020

I'm on board. We will have to go through a deprecation cycle though, since this is changing user-facing behavior.

@jku
Copy link
Contributor

jku commented Jun 15, 2020

Right, that's the feel I got from the code as well -- although for a newcomer parse_req_from_line() is not well specified: refactoring anything seems scary because the input can be so many different things...

@uranusjr
Copy link
Member

uranusjr commented Jun 15, 2020

Trust me, I feel as uneasy as you are to the prospect. I feel less afriad of breaking things with experience, but still break as many things as when I started.

@pfmoore
Copy link
Member Author

pfmoore commented Jun 15, 2020

+1 from me too. And @jku don't worry, those nervous feelings are the sign of someone who knows what they are doing! The people I'd be worried about are people who look at this code and think it all seems reasonably straightforward 🙂

On a more serious note, please feel free to ask if you're unsure. And trust your instincts - they've been good so far.

@jku
Copy link
Contributor

jku commented Jun 18, 2020

For others information: I have some ideas and might try the refactoring at some point but for now I'm not working on it -- feel free to have a go.

I'll have another go with the PR #8424 now though: getting this specific misparsing marked as deprecated will be useful in any case.

bors bot added a commit to duckinator/emanate that referenced this issue Jul 30, 2020
153: Update pip to 20.2 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.1.1** to **20.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.2
   ```
   =================

Deprecations and Removals
-------------------------

- Deprecate setup.py-based builds that do not generate an ``.egg-info`` directory. (`6998 &lt;https://github.com/pypa/pip/issues/6998&gt;`_, `8617 &lt;https://github.com/pypa/pip/issues/8617&gt;`_)
- Disallow passing install-location-related arguments in ``--install-options``. (`7309 &lt;https://github.com/pypa/pip/issues/7309&gt;`_)
- Add deprecation warning for invalid requirements format &quot;base&gt;=1.0[extra]&quot; (`8288 &lt;https://github.com/pypa/pip/issues/8288&gt;`_)
- Deprecate legacy setup.py install when building a wheel failed for source
  distributions without pyproject.toml (`8368 &lt;https://github.com/pypa/pip/issues/8368&gt;`_)
- Deprecate -b/--build/--build-dir/--build-directory. Its current behaviour is confusing
  and breaks in case different versions of the same distribution need to be built during
  the resolution process. Using the TMPDIR/TEMP/TMP environment variable, possibly
  combined with --no-clean covers known use cases. (`8372 &lt;https://github.com/pypa/pip/issues/8372&gt;`_)
- Remove undocumented and deprecated option ``--always-unzip`` (`8408 &lt;https://github.com/pypa/pip/issues/8408&gt;`_)

Features
--------

- Log debugging information about pip, in ``pip install --verbose``. (`3166 &lt;https://github.com/pypa/pip/issues/3166&gt;`_)
- Refine error messages to avoid showing Python tracebacks when an HTTP error occurs. (`5380 &lt;https://github.com/pypa/pip/issues/5380&gt;`_)
- Install wheel files directly instead of extracting them to a temp directory. (`6030 &lt;https://github.com/pypa/pip/issues/6030&gt;`_)
- Add a beta version of pip&#39;s next-generation dependency resolver.

  Move pip&#39;s new resolver into beta, remove the
  ``--unstable-feature=resolver`` flag, and enable the
  ``--use-feature=2020-resolver`` flag. The new resolver is
  significantly stricter and more consistent when it receives
  incompatible instructions, and reduces support for certain kinds of
  :ref:`Constraints Files`, so some workarounds and workflows may
  break. More details about how to test and migrate, and how to report
  issues, at :ref:`Resolver changes 2020` . Maintainers are preparing to
   ```
   
  
  
   ### 20.2b1
   ```
   ===================

Bug Fixes
---------

- Correctly treat wheels containing non-ASCII file contents so they can be
  installed on Windows. (`5712 &lt;https://github.com/pypa/pip/issues/5712&gt;`_)
- Prompt the user for password if the keyring backend doesn&#39;t return one (`7998 &lt;https://github.com/pypa/pip/issues/7998&gt;`_)

Improved Documentation
----------------------

- Add GitHub issue template for reporting when the dependency resolver fails (`8207 &lt;https://github.com/pypa/pip/issues/8207&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
bors bot added a commit to duckinator/emanate that referenced this issue Jul 31, 2020
153: Update pip to 20.2 r=duckinator a=pyup-bot


This PR updates [pip](https://pypi.org/project/pip) from **20.1.1** to **20.2**.



<details>
  <summary>Changelog</summary>
  
  
   ### 20.2
   ```
   =================

Deprecations and Removals
-------------------------

- Deprecate setup.py-based builds that do not generate an ``.egg-info`` directory. (`6998 &lt;https://github.com/pypa/pip/issues/6998&gt;`_, `8617 &lt;https://github.com/pypa/pip/issues/8617&gt;`_)
- Disallow passing install-location-related arguments in ``--install-options``. (`7309 &lt;https://github.com/pypa/pip/issues/7309&gt;`_)
- Add deprecation warning for invalid requirements format &quot;base&gt;=1.0[extra]&quot; (`8288 &lt;https://github.com/pypa/pip/issues/8288&gt;`_)
- Deprecate legacy setup.py install when building a wheel failed for source
  distributions without pyproject.toml (`8368 &lt;https://github.com/pypa/pip/issues/8368&gt;`_)
- Deprecate -b/--build/--build-dir/--build-directory. Its current behaviour is confusing
  and breaks in case different versions of the same distribution need to be built during
  the resolution process. Using the TMPDIR/TEMP/TMP environment variable, possibly
  combined with --no-clean covers known use cases. (`8372 &lt;https://github.com/pypa/pip/issues/8372&gt;`_)
- Remove undocumented and deprecated option ``--always-unzip`` (`8408 &lt;https://github.com/pypa/pip/issues/8408&gt;`_)

Features
--------

- Log debugging information about pip, in ``pip install --verbose``. (`3166 &lt;https://github.com/pypa/pip/issues/3166&gt;`_)
- Refine error messages to avoid showing Python tracebacks when an HTTP error occurs. (`5380 &lt;https://github.com/pypa/pip/issues/5380&gt;`_)
- Install wheel files directly instead of extracting them to a temp directory. (`6030 &lt;https://github.com/pypa/pip/issues/6030&gt;`_)
- Add a beta version of pip&#39;s next-generation dependency resolver.

  Move pip&#39;s new resolver into beta, remove the
  ``--unstable-feature=resolver`` flag, and enable the
  ``--use-feature=2020-resolver`` flag. The new resolver is
  significantly stricter and more consistent when it receives
  incompatible instructions, and reduces support for certain kinds of
  :ref:`Constraints Files`, so some workarounds and workflows may
  break. More details about how to test and migrate, and how to report
  issues, at :ref:`Resolver changes 2020` . Maintainers are preparing to
   ```
   
  
  
   ### 20.2b1
   ```
   ===================

Bug Fixes
---------

- Correctly treat wheels containing non-ASCII file contents so they can be
  installed on Windows. (`5712 &lt;https://github.com/pypa/pip/issues/5712&gt;`_)
- Prompt the user for password if the keyring backend doesn&#39;t return one (`7998 &lt;https://github.com/pypa/pip/issues/7998&gt;`_)

Improved Documentation
----------------------

- Add GitHub issue template for reporting when the dependency resolver fails (`8207 &lt;https://github.com/pypa/pip/issues/8207&gt;`_)
   ```
   
  
</details>


 

<details>
  <summary>Links</summary>
  
  - PyPI: https://pypi.org/project/pip
  - Changelog: https://pyup.io/changelogs/pip/
  - Homepage: https://pip.pypa.io/
</details>



Co-authored-by: pyup-bot <github-bot@pyup.io>
Co-authored-by: Ellen Marie Dash <me@duckie.co>
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 13, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
state: awaiting PR Feature discussed, PR is needed type: deprecation Related to deprecation / removal.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants