packaging does not expect 'extra' in marker (need vendor upgrade)

[ice-tong]

Description

Hi, I've been doing some hacks on pip recently, like taking extra requirements into account when checking_install_conflicts.
I have rewritten the method of pkg_resources.Distribution, and every time I get the dependency, I will return the extra mim requirements.
Here are the sample code:

from contextlib import contextmanager
from pip._internal.commands import create_command


@contextmanager
def patch_pkg_resources_distribution() -> Generator:
    from pip._vendor.pkg_resources import Distribution, parse_requirements

    origin_requires = Distribution.requires

    def patched_requires(self, extras=()):
        deps = origin_requires(self, extras)
        if 'mim' in self.extras:
            mim_extra_requires = origin_requires(self, ('mim',))
            filter_invalid_marker(mim_extra_requires)
            deps += mim_extra_requires
        return deps

    Distribution.requires = patched_requires  # type: ignore
    yield

    Distribution.requires = origin_requires  # type: ignore
    return


install_args = [...]
with patch_pkg_resources_distribution():
    create_command('install')(install_args)

I got this error during check_install_conflicts:

ERROR: Error while checking for conflicts. Please file an issue on pip's issue tracker: https://github.com/pypa/pip/issues/new
Traceback (most recent call last):
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/commands/install.py", line 531, in _determine_conflicts
    return check_install_conflicts(to_install)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 111, in check_install_conflicts
    package_set, should_ignore=lambda name: name not in whitelist
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_internal/operations/check.py", line 78, in check_package_set
    missed = req.marker.evaluate()
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 304, in evaluate
    return _evaluate_markers(self._markers, current_environment)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 234, in _evaluate_markers
    lhs_value = _get_env(environment, lhs.value)
  File "/mnt/data/xiaoyancong/miniconda3/envs/open-mmlab/lib/python3.7/site-packages/pip/_vendor/packaging/markers.py", line 216, in _get_env
    f"{name!r} does not exist in evaluation environment."
pip._vendor.packaging.markers.UndefinedEnvironmentName: 'extra' does not exist in evaluation environment.

The reason for the above error is that this code does not consider that the marker of the requirement obtained from METADATA may contain extras. Is this behavior a bug?

A sample METADTA that contains Requires-Dist:

...
Requires-Dist: matplotlib
Requires-Dist: mmcls (>=0.20.1)
Requires-Dist: numpy
Requires-Dist: packaging
Requires-Dist: prettytable
Provides-Extra: all
Requires-Dist: cityscapesscripts ; extra == 'all'
Requires-Dist: matplotlib ; extra == 'all'
Requires-Dist: mmcls (>=0.20.1) ; extra == 'all'
Requires-Dist: numpy ; extra == 'all'
Requires-Dist: packaging ; extra == 'all'
Requires-Dist: prettytable ; extra == 'all'
Requires-Dist: codecov ; extra == 'all'
Requires-Dist: flake8 ; extra == 'all'
Requires-Dist: interrogate ; extra == 'all'
Requires-Dist: pytest ; extra == 'all'
Requires-Dist: xdoctest (>=0.10.0) ; extra == 'all'
Requires-Dist: yapf ; extra == 'all'
Provides-Extra: build
Provides-Extra: mim
Requires-Dist: mmcls (>=0.20.1) ; extra == 'mim'
Requires-Dist: mmcv-full (<=1.6.0,>=1.4.4) ; extra == 'mim'
Provides-Extra: optional
Requires-Dist: cityscapesscripts ; extra == 'optional'
Provides-Extra: tests
Requires-Dist: codecov ; extra == 'tests'
Requires-Dist: flake8 ; extra == 'tests'
Requires-Dist: interrogate ; extra == 'tests'
Requires-Dist: pytest ; extra == 'tests'
Requires-Dist: xdoctest (>=0.10.0) ; extra == 'tests'
Requires-Dist: yapf ; extra == 'tests'
...

Expected behavior

The requirement parsed from METADATA should consider the invalid situation of the marker?
For example, in src/pip/_internal/operations/check.py#L78 we can handle the invalid marker?

...
for req in package_detail.dependencies:
    name = canonicalize_name(req.name)
    
    # Check if it's missing
    if name not in package_set:
        missed = True
        if req.marker is not None:
            try:
                missed = req.marker.evaluate()
            except:
                missed = False
            if missed:
                missing_deps.add((name, req))
        continue
    ....

pip version

22.1.2

Python version

3.7

OS

ubuntu

How to Reproduce

See above

Output

see above

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This looks like a duplicate of #9643

[uranusjr]

packaging already recently fixed this, but we need to pull in the change. The change is still unreleased so that needs to happen as well.

[pradyunsg]

You can pass {"extra": ""} to the evaluate call for your experimentation, which will let you move forward with that.

@uranusjr I don't think we need a tracking issue for doing a vendoring upgrade. If you're in agreement, please feel free to close this out. :)

[uranusjr]

Sure if you also remember to release packaging before the pip release 🙂 That’s the part I’m less sure about.

[pradyunsg]

Ouch, but fair. 😝

[pfmoore]

It doesn't look like packaging will get released for the 22.3 release, so I'm moving this to the 23.0 milestone.