Unable to install packages which have no sdist

[icemac]

I want to install https://pypi.org/project/playwright/ using buildout.
This is only successful as long the release contains a source distribution.
As newer releases of that package no longer provide an sdist, buildout fails to install them.

See the following example: (According to #567 pip 21.0.1 is the newest version supported by buildout.)

$ python3.7 -m venv playwright-buildout
$ cd playwright-buildout
$ cat > buildout.cfg << EOF
[buildout]
parts = test

[test]
recipe = zc.recipe.egg
eggs =
    playwright
EOF
$ bin/pip install -U pip==21.0.1 zc.buildout==3.0.0b2
...
$ bin/buildout
Creating directory '.../playwright-buildout/parts'.
Creating directory '.../playw-buildout/develop-eggs'.
Installing test.
Getting distribution for 'playwright'.
Got playwright 0.162.2.
Getting distribution for 'greenlet==1.0a1'.
Got greenlet 1.0a1.
Getting distribution for 'pyee>=8.0.1'.
Got pyee 8.2.2.

Thus the installed version of playwright is 0.162.2 whereas the current version – while writing this issue – is 1.14.1.
Requiring playwright >= 1 leads to:

$ bin/buildout
Installing test.
Getting distribution for 'playwright>=1'.
While:
  Installing test.
  Getting distribution for 'playwright>=1'.
Error: Couldn't find a distribution for 'playwright>=1'.

Whereas it can be installed on my machine using pip:

$ bin/pip install "playwright>=1"
Collecting playwright>=1
  Using cached https://development.verdi.de/devpi/root/pypi/%2Bf/41f/158a850af1e0a/playwright-1.14.1-py3-none-macosx_11_0_universal2.whl (18.0 MB)
...
Installing collected packages: websockets, pyee, greenlet, playwright
Successfully installed greenlet-1.1.1 playwright-1.14.1 pyee-8.2.2 websockets-9.1

[gotcha]

I do not manage to reproduce your issue.

If I specify playwright = 1.14.1 in the [versions] section, everything works fine for me.

I wonder if you are trying to use playwright>=1 in buildout config file. Which is not a use case covered.

[icemac]

That's strange: I tried again with the following content for buildout.cfg:

[buildout]
parts = test
eggs-directory = eggs
abi-tag-eggs = false

[versions]
playwright = 1.14.1

[test]
recipe = zc.recipe.egg
eggs =
    playwright

And I got:

$ bin/buildout
Updating test.
Getting distribution for 'playwright==1.14.1'.
While:
  Updating test.
  Getting distribution for 'playwright==1.14.1'.
Error: Couldn't find a distribution for 'playwright==1.14.1'.

Setting abi-tag-eggs to true does not change the behavior for me.
Maybe this happens because there is just a univeral2 wheel for Darwin 11 which might not be supported by pip==21.0.1.

[gotcha]

@icemac Can you share the output of bin/buildout -vv?

[icemac]

@gotcha Sorry for the long delay. I upgraded to zc.buildout 3.0.0b3 but the error is still the same. This is what -vv shows, I cannot see anything special here:

$ bin/buildout -vv
Using buildout.cfg and bin/buildout.
Installing 'zc.recipe.egg'.
Picked: zc.recipe.egg = 2.0.7

Configuration data:
[buildout]
abi-tag-eggs = false
allow-hosts = *
allow-picked-versions = true
allow-unknown-extras = false
bin-directory = /.../playwright-buildout/bin
develop-eggs-directory = /.../playwright-buildout/develop-eggs
directory = /.../playwright-buildout
download-cache = /Users/mac/python/cache
eggs-directory = /.../playwright-buildout/eggs
executable = /.../playwright-buildout/bin/python3.7
extends-cache = /Users/mac/python/cache/extends
find-links =
install-from-cache = false
installed = /.../playwright-buildout/.installed.cfg
log-format =
log-level = INFO
newest = false
offline = false
parts = test
parts-directory = /.../playwright-buildout/parts
prefer-final = true
python = buildout
show-picked-versions = false
socket-timeout =
update-versions-file =
use-dependency-links = true
verbosity = 20
versions = versions
[test]
_b = /.../playwright-buildout/bin
_d = /.../playwright-buildout/develop-eggs
_e = /.../playwright-buildout/eggs
bin-directory = /.../playwright-buildout/bin
develop-eggs-directory = /.../playwright-buildout/develop-eggs
eggs = playwright
eggs-directory = /.../playwright-buildout/eggs
recipe = zc.recipe.egg
[versions]
playwright = 1.14.1
zc.buildout = >=3.0.0b3
zc.recipe.egg = >=2.0.6

Installing test.
Installing 'playwright'.
We have no distributions for playwright that satisfies 'playwright==1.14.1'.
Getting distribution for 'playwright==1.14.1'.
While:
  Installing test.
  Getting distribution for 'playwright==1.14.1'.
Error: Couldn't find a distribution for 'playwright==1.14.1'.

[gotcha]

@icemac Which OS/version are you on ?

[icemac]

@gotcha I am on MacOS 11.6.

[gotcha]

@icemac Did you try to install with pip install --no-cache-dir playwright==1.14.1 ?

I'd like to confirm that latest pip would succeed to install playwright==1.14.1 on your MacOS 11.6 machine.

[jamadden]

pip works for me on an intel 11.6 machine; the wheel is universal so it should also work for M1 macs, given a recent enough pip.

$ pip --version
pip 21.1.1 from /...lib/python3.10/site-packages/pip (python 3.10)
$ pip install --no-cache-dir playwright==1.14.1
Collecting playwright==1.14.1
  Downloading playwright-1.14.1-py3-none-macosx_11_0_universal2.whl (18.0 MB)
     |████████████████████████████████| 18.0 MB 6.9 MB/s
Collecting websockets>=8.1
  Downloading websockets-10.0.tar.gz (81 kB)
     |████████████████████████████████| 81 kB 51.9 MB/s
Requirement already satisfied: greenlet>=1.0.0 in ./src (from playwright==1.14.1) (1.2.0.dev0)
Collecting pyee>=8.0.1
  Downloading pyee-8.2.2-py2.py3-none-any.whl (12 kB)
Building wheels for collected packages: websockets
  Building wheel for websockets (setup.py) ... done
  Created wheel for websockets: filename=websockets-10.0-cp310-cp310-macosx_11_0_x86_64.whl size=94988 sha256=b3a54f973a87f1ca21ed6516e128177d0b57562f3e1fe184250411d2d3a7028d
  Stored in directory: /.../T/pip-ephem-wheel-cache-jen3q6qn/wheels/d5/56/64/2ecbd74736716d6bba72d533296fac530c4530a84f932bcd81
Successfully built websockets
Installing collected packages: websockets, pyee, playwright
WARNING: You are using pip version 21.1.1; however, version 21.3 is available.
You should consider upgrading via the '/.../bin/python -m pip install --upgrade pip' command.

[gotcha]

@jamadden Do you see the same issue as @icemac when running the buildout hereabove ?

[jamadden]

I do see the same issue.

$ mktmpenv `which python3.10`
…
Collecting pip
  Using cached pip-21.3-py3-none-any.whl (1.7 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 21.2.4
    Uninstalling pip-21.2.4:
      Successfully uninstalled pip-21.2.4
Successfully installed pip-21.3
...
Successfully installed MarkupSafe-2.0.1 alabaster-0.7.12 appnope-0.1.2 astroid-2.6.6 babel-2.9.1 backcall-0.2.0 bleach-4.1.0 build-0.7.0 certifi-2021.10.8 chardet-4.0.0 charset-normalizer-2.0.7 check-manifest-0.47 colorama-0.4.4 coverage-6.0.2 decorator-5.1.0 docrepr-0.1.1 docutils-0.17.1 elpy-1.999 idna-3.3 imagesize-1.2.0 importlib-metadata-4.8.1 ipython-7.28.0 isort-5.9.3 jedi-0.18.0 jinja2-3.0.2 keyring-23.2.1 lazy-object-proxy-1.6.0 matplotlib-inline-0.1.3 mccabe-0.6.1 packaging-21.0 parso-0.8.2 pep517-0.11.0 pep8-1.7.1 pexpect-4.8.0 pickleshare-0.7.5 pkginfo-1.7.1 prompt-toolkit-3.0.20 ptyprocess-0.7.0 pyflakes-2.4.0 pygments-2.10.0 pylint-3.0.0a4 pyparsing-3.0.0rc2 pyroma-3.2 pytz-2021.3 readme-renderer-30.0 requests-2.26.0 requests-toolbelt-0.9.1 rfc3986-1.5.0 six-1.16.0 snowballstemmer-2.1.0 sphinx-4.2.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-2.0.0 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.5 toml-0.10.2 tomli-1.2.1 tqdm-4.62.3 traitlets-5.1.0 twine-3.4.2 urllib3-1.26.7 wcwidth-0.2.5 webencodings-0.5.1 wrapt-1.12.1 zest.releaser-6.22.1 zipp-3.6.0 zope.dottedname-4.3
This is a temporary environment. It will be deleted when you run 'deactivate'.
$ cat >> buildout.cfg
[buildout]
parts = test
eggs-directory = eggs
abi-tag-eggs = false

[versions]
playwright = 1.14.1

[test]
recipe = zc.recipe.egg
eggs =
    playwright

^D
$ pip install 'zc.buildout>2'
Collecting zc.buildout>2
  Downloading zc.buildout-3.0.0b3-py2.py3-none-any.whl (165 kB)
     |████████████████████████████████| 165 kB 5.1 MB/s
Requirement already satisfied: pip in ./lib/python3.10/site-packages (from zc.buildout>2) (21.3)
Requirement already satisfied: wheel in ./lib/python3.10/site-packages (from zc.buildout>2) (0.37.0)
Requirement already satisfied: setuptools>=8.0 in ./lib/python3.10/site-packages (from zc.buildout>2) (58.0.4)
Installing collected packages: zc.buildout
Successfully installed zc.buildout-3.0.0b3
$ buildout -vv
Creating directory '//tmp-546d5f3fbb63f0eb/eggs'.
Creating directory '//tmp-546d5f3fbb63f0eb/parts'.
Creating directory '//tmp-546d5f3fbb63f0eb/develop-eggs'.
Installing 'zc.buildout', 'setuptools', 'pip', 'wheel'.
We have a develop egg: zc.buildout 3.0.0b3
We have a develop egg: setuptools 58.0.4
We have a develop egg: pip 21.3
We have a develop egg: wheel 0.37.0
Installing 'zc.recipe.egg'.
We have no distributions for zc.recipe.egg that satisfies 'zc.recipe.egg>=2.0.6'.
Getting distribution for 'zc.recipe.egg>=2.0.6'.
Download cache has zc.recipe.egg 2.0.7 at: /Users/jmadden/Projects/buildout-cache/dist/zc.recipe.egg-2.0.7.tar.gz
Running pip install:
"//tmp-546d5f3fbb63f0eb/bin/python" "-c" "import sys; sys.path[0:0] = []; from pip._internal.cli.main import main; main()" "install" "--no-deps" "-t" "//tmp-546d5f3fbb63f0eb/eggs/tmpb18m2lw1" "/Users/jmadden/Projects/buildout-cache/dist/zc.recipe.egg-2.0.7.tar.gz"
path=[]

Processing //buildout-cache/dist/zc.recipe.egg-2.0.7.tar.gz
  Preparing metadata (setup.py) ... done
Building wheels for collected packages: zc.recipe.egg
  Building wheel for zc.recipe.egg (setup.py) ... done
  Created wheel for zc.recipe.egg: filename=zc.recipe.egg-2.0.7-py2.py3-none-any.whl size=30633 sha256=99272f3627da8df8547bd923941a2707361ee4dc14222b7d2742bc7dcfc51236
  Stored in directory: //Caches/pip/wheels/83/5d/34/7b35347e67f934841971cf9e4dd77aace93fb3203e61ef1bf1
Successfully built zc.recipe.egg
Installing collected packages: zc.recipe.egg
Successfully installed zc.recipe.egg-2.0.7
Got zc.recipe.egg 2.0.7.
Picked: zc.recipe.egg = 2.0.7

Configuration data:
[buildout]
abi-tag-eggs = false
allow-hosts = *
allow-picked-versions = true
allow-unknown-extras = false
bin-directory = //tmp-546d5f3fbb63f0eb/bin
develop-eggs-directory = //tmp-546d5f3fbb63f0eb/develop-eggs
directory = //tmp-546d5f3fbb63f0eb
download-cache = /Users/jmadden/Projects/buildout-cache
eggs-directory = //tmp-546d5f3fbb63f0eb/eggs
executable = //tmp-546d5f3fbb63f0eb/bin/python
find-links =
install-from-cache = false
installed = //tmp-546d5f3fbb63f0eb/.installed.cfg
log-format =
log-level = INFO
newest = true
offline = false
parts = test
parts-directory = //tmp-546d5f3fbb63f0eb/parts
prefer-final = true
python = buildout
show-picked-versions = false
socket-timeout =
update-versions-file =
use-dependency-links = true
verbosity = 20
versions = versions
[test]
_b = //tmp-546d5f3fbb63f0eb/bin
_d = //tmp-546d5f3fbb63f0eb/develop-eggs
_e = //tmp-546d5f3fbb63f0eb/eggs
bin-directory = //tmp-546d5f3fbb63f0eb/bin
develop-eggs-directory = //tmp-546d5f3fbb63f0eb/develop-eggs
eggs = playwright
eggs-directory = //tmp-546d5f3fbb63f0eb/eggs
recipe = zc.recipe.egg
[versions]
playwright = 1.14.1
zc.buildout = >=3.0.0b3
zc.recipe.egg = >=2.0.6

Installing test.
Installing 'playwright'.
We have no distributions for playwright that satisfies 'playwright==1.14.1'.
Getting distribution for 'playwright==1.14.1'.
While:
  Installing test.
  Getting distribution for 'playwright==1.14.1'.
Error: Couldn't find a distribution for 'playwright==1.14.1'.

The problem persists if I upgrade setuptools to the lastest 58.2.0.

[jamadden]

Given this part of the failing backtrace:

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/zc/buildout/easy_install.py(696)install()
    694
    695         for requirement in requirements:
--> 696             for dist in self._get_dist(requirement, ws):
    697                 self._maybe_add_setuptools(ws, dist)
    698

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/zc/buildout/easy_install.py(554)_get_dist()
    552         # Maybe an existing dist is already the best dist that satisfies the
    553         # requirement
--> 554         dist, avail = self._satisfied(requirement)
    555
    556         if dist is None:

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/zc/buildout/easy_install.py(347)_satisfied()
    345                          req.project_name, str(req))
    346
--> 347             return None, self._obtain(req, source)
    348
    349         # Note that dists are sorted from best to worst, as promised by

 //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/zc/buildout/easy_install.py(476)_obtain()
    474         from IPython.terminal.debugger import set_trace; set_trace() ## DEBUG ##
    475
--> 476         if index.obtain(requirement) is None:
    477             # Nothing is available.
    478             print(">>> INDEX OBTAIN IS NONE")

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/setuptools/package_index.py(485)obtain()
    483     def obtain(self, requirement, installer=None):
    484         self.prescan()
--> 485         self.find_packages(requirement)
    486         for dist in self[requirement.key]:
    487             if dist in requirement:

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/setuptools/package_index.py(469)find_packages()
    467
    468     def find_packages(self, requirement):
--> 469         self.scan_url(self.index_url + requirement.unsafe_name + '/')
    470
    471         if not self.package_pages.get(requirement.key):

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/setuptools/package_index.py(810)scan_url()
    808
    809     def scan_url(self, url):
--> 810         self.process_url(url, True)
    811
    812     def _attempt_download(self, url, filename):

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/zc/buildout/patches.py(70)process_url()
     68                 return
     69             else:
---> 70                 dists = list(distros_for_url(url))
     71                 if dists:
     72                     if not self.url_ok(url):

  //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/setuptools/package_index.py(101)distros_for_url()
     99     """Yield egg or source distribution objects that might be found at a URL"""
    100     base, fragment = egg_info_for_url(url)
--> 101     for dist in distros_for_location(url, base, metadata):
    102         yield dist
    103     if fragment:

> //tmp-546d5f3fbb63f0eb/lib/python3.10/site-packages/setuptools/package_index.py(138)distros_for_location()

We are here:

    112 def distros_for_location(location, basename, metadata=None):
    113     """Yield egg or source distribution objects based on basename"""
    114     if basename.endswith('.egg.zip'):
    115         basename = basename[:-4]  # strip the .zip
    116     if basename.endswith('.egg') and '-' in basename:
    117         # only one, unambiguous interpretation
    118         return [Distribution.from_location(location, basename, metadata)]
    119     if basename.endswith('.whl') and '-' in basename:
    120         wheel = Wheel(basename)
    121         if not wheel.is_compatible():
    122             return []
    123         return [Distribution(
    124             location=location,
    125             project_name=wheel.project_name,
    126             version=wheel.version,
    127             # Increase priority over eggs.
    128             precedence=EGG_DIST + 1,
    129         )]
    130     if basename.endswith('.exe'):
    131         win_base, py_ver, platform = parse_bdist_wininst(basename)
    132         if win_base is not None:
    133             return interpret_distro_name(
    134                 location, win_base, metadata, py_ver, BINARY_DIST, platform
    135             )
    136     # Try source distro extensions (.zip, .tgz, etc.)
    137     #
--> 138     for ext in EXTENSIONS:
    139         if basename.endswith(ext):
    140             basename = basename[:-len(ext)]
    141             return interpret_distro_name(location, basename, metadata)
    142     return []  # no extension matched
    143

With argument values:

ipdb> args
location = 'https://pypi.org/simple/playwright/'
basename = ''
metadata = None

And the value of EXTENSIONS:

ipdb> EXTENSIONS
['.tar.gz', '.tar.bz2', '.tar', '.zip', '.tgz']

So this part of setuptools will never look for just wheels.

(I know that's not final, I'm still stepping.)

[jamadden]

The root of the problem is that setuptools doesn't know that universal2 wheels are usable on macOS (playwright only publishes universal2 wheels for macOS). Given a filename like playwright-0.8.1a2-py3-none-macosx_11_0_universal2.whl, setuptools.wheel.Wheel correctly extracts the tags as [('py3', 'none', 'macosx_11_0_universal2')], but it constructs the list of supported tags using its vendored copy of packaging at setuptools._vendor.packaging.tags, and that vendored version is too old to understand universal2. The vendored version of packaging that pip has is newer.

So when we eventually call this API with distros_for_location('https://files.pythonhosted.org/packages//playwright-1.14.1-py3-none-macosx_11_0_universal2.whl#sha256=…,' 'playwright-1.14.1-py3-none-macosx_11_0_universal2.whl'), the wheel.is_compatible() check on line 121 fails.

[jamadden]

Support for universal2 wheels was added in pypa/packaging#319 which was released in version 20.9 in January of this year. setuptools's vendored copy of packaging was last updated to version 20.4 to close pypa/setuptools#2310 in pypa/setuptools#2327 from August of last year.

It'll take a new issue/PR and setuptools release to fix this, unless we want to add setuptools.extern.packaging.tags to the things we monkey-patch and incorporate pypa/packaging#319 that way.

[icemac]

@jamadden Thank you for the investigation.

I feel setuptools would be the right place to fix this. I am not in a hurry.

[gotcha]

@jamadden Did you open a new issue on pypa/setuptools ?

[jamadden]

In the setuptools commit pypa/setuptools@3b87957 the bundled version got updated to 21.2.

[gotcha]

@icemac Can you check if your issue is fixed with setuptools>=59.1.0 ?

[icemac]

@gotcha Updating setuptools (I my case to 59.2.0) solved the problem, I am now able to install playwright==1.14.1 using zc.buildout. — Thank you all for your investigation and help.

Is there still something we should do about this issue or can it be closed now?

[gotcha]

I'll close.

Thanks again to @jamadden !