pip 22 TypeError: missing required 'use_deprecated_html5lib'

[mauritsvanrees]

I am trying out the latest pip version (22.0.2) in the Plone core development buildout, see this PR and hit an error with zc.buildout 3.0.0rc1:

Traceback...
  File ".../lib/python3.9/site-packages/zc/buildout/patches.py", line 120, in process_url
    plinks = list(parse_links(html_page))
TypeError: wrapper_wrapper() missing 1 required positional argument: 'use_deprecated_html5lib'

Full traceback is in that PR.

It seems likely that the process_url patch in zc.buildout needs to be updated to a new feature in pip:

Changed PackageFinder to parse HTML documents using the stdlib html.parser.HTMLParser class instead of the html5lib package.
For now, the deprecated html5lib code remains and can be used with the --use-deprecated=html5lib command line option. However, it will be removed in a future pip release. (#10291)

[mauritsvanrees]

BTW, I tried reproducing this with a much smaller buildout config, instead of the huge Plone one, but the following worked without errors:

$ python3.9 -m venv .
$ bin/pip install -U pip setuptools zc.buildout
$ cat buildout.cfg 
[buildout]
parts = test
[test]
recipe = zc.recipe.egg
eggs = pygments
$ bin/buildout 
Creating directory '/Users/maurits/tmp/pip22/parts'.
Creating directory '/Users/maurits/tmp/pip22/develop-eggs'.
Installing test.
Getting distribution for 'pygments'.
Got Pygments 2.11.2.
Generated script '/Users/maurits/tmp/pip22/bin/pygmentize'.

[ale-rt]

I can reproduce it:

[ale@flo pip22]$ pip freeze --all
pip==22.0.3
pkg_resources==0.0.0
setuptools==60.7.1
wheel==0.37.1
zc.buildout==3.0.0rc1
🌠 ~/tmp/pip22  
[ale@flo pip22]$ cat buildout.cfg 
[buildout]
parts = test

[test]
recipe = zc.recipe.egg
eggs = pygments
🌠 ~/tmp/pip22  
[ale@flo pip22]$ buildout 
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/_distutils_hack/__init__.py:30: UserWarning: Setuptools is replacing distutils.
  warnings.warn("Setuptools is replacing distutils.")
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning:  is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: text-python-1.1.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: python-1.1.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: openid-2.2.5 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: bcrypt-0.4 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: debugger-3.1.0 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: Levenshtein-0.12.2 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: pptx-0.6.18 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: ng-0.9.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: blind-except-0.1.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: except-0.1.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: 1.2.7-20070827-preview is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: preview is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: quotes-2.0.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: debugger-3.2.1 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: archive-0.2 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: docx-0.8.10 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: dateutil-1.5 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: gettext-1.0 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: quotes-1.0.0 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: todo-0.7 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: print-3.1.0 is an invalid version and will not be supported in a future release
  warnings.warn(
/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/pkg_resources/__init__.py:122: PkgResourcesDeprecationWarning: gettext-4.0 is an invalid version and will not be supported in a future release
  warnings.warn(
While:
  Installing.
  Getting section test.
  Initializing section test.
  Installing recipe zc.recipe.egg.
  Getting distribution for 'zc.recipe.egg>=2.0.6'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 2250, in main
    getattr(buildout, command)(args)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 739, in install
    [self[part]['recipe'] for part in install_parts]
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 739, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1372, in __getitem__
    options._initialize()
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1480, in _initialize
    self.initialize()
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1486, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1429, in _install_and_load
    zc.buildout.easy_install.install(
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 965, in install
    return installer.install(specs, working_set)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 690, in install
    for dist in self._get_dist(requirement, ws):
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 550, in _get_dist
    dist, avail = self._satisfied(requirement)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 384, in _satisfied
    best_available = self._obtain(req, source)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 473, in _obtain
    if index.obtain(requirement) is None:
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/setuptools/package_index.py", line 493, in obtain
    self.find_packages(requirement)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/setuptools/package_index.py", line 477, in find_packages
    self.scan_url(self.index_url + requirement.unsafe_name + '/')
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/setuptools/package_index.py", line 818, in scan_url
    self.process_url(url, True)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/zc/buildout/patches.py", line 120, in process_url
    plinks = list(parse_links(html_page))
TypeError: wrapper_wrapper() missing 1 required positional argument: 'use_deprecated_html5lib'

The error goes away when I run it with the -N flag (that picks the eggs from the cache).

[ale-rt]

Maybe you have newest=false in your .buildout/default.cfg, what happens if you try:

buildout -U

?

[ale-rt]

Btw if I change plinks = list(parse_links(html_page, use_deprecated_html5lib=False or Tue does not matter)) then I have:
this error:

  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/setuptools/__init__.py", line 174, in __init__
    super().__init__(dist)
  File "/home/ale/tmp/pip22/py39/lib/python3.9/site-packages/setuptools/_distutils/cmd.py", line 57, in __init__
    raise TypeError("dist must be a Distribution instance")
TypeError: dist must be a Distribution instance

[mauritsvanrees]

Aha, I have eggs-directory and download-cache in my ~/.buildout/default.cfg, which is what makes it work for me. When I override those settings in my buildout.cfg it fails in the same way as for you.

eggs-directory = eggs
download-cache = downloads

[mauritsvanrees]

When I temporarily fix it in the patch, I get the new same error as you. With a pdb:

 54  	        from distutils.dist import Distribution
 55  	
 56  	        if not isinstance(dist, Distribution):
 57  ->	            raise TypeError("dist must be a Distribution instance")
(Pdb) dist
<setuptools.dist.Distribution object at 0x10d0604c0>
(Pdb) Distribution
<class 'distutils.dist.Distribution'>

So setuptools complains that dist is a setuptools Distribution instead of a distils Distribution.
The dist here is Pygments. Maybe it does stuff wrong in its setup.py/cfg.
See for example stack overflow.