pip install pytest fails - Mo matching found for py>=1.5.0 for python3.3

[ColinKennedy]

I'm using a tox environment and all of its tests pass in Python 2.7,3.4-6 except for in Python 3.3. It just started happening last night and persisted into this morning, both on my Linux environment locally and also remotely

Here is the log for when I try to run

python3.3 -m pip install pytest

Collecting pytest
  Using cached pytest-3.3.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in /usr/local/lib/python3.3/dist-packages (from pytest)
Collecting attrs>=17.2.0 (from pytest)
  Using cached attrs-17.3.0-py2.py3-none-any.whl
Collecting pluggy<0.7,>=0.5 (from pytest)
  Downloading pluggy-0.5.2.tar.gz
Collecting py>=1.5.0 (from pytest)
  Could not find a version that satisfies the requirement py>=1.5.0 (from pytest) (from versions: 0.9.1, 0.9.2, 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.3.2, 1.3.3, 1.3.4, 1.4.0, 1.4.1, 1.4.2, 1.4.3, 1.4.4, 1.4.5, 1.4.6, 1.4.7.dev3, 1.4.7, 1.4.8, 1.4.9, 1.4.10, 1.4.11, 1.4.12, 1.4.13, 1.4.14, 1.4.15, 1.4.16, 1.4.17, 1.4.18, 1.4.19, 1.4.20, 1.4.21, 1.4.22, 1.4.23, 1.4.24, 1.4.25, 1.4.26, 1.4.27, 1.4.28, 1.4.29, 1.4.30, 1.4.31, 1.4.32, 1.4.33, 1.4.34)
No matching distribution found for py>=1.5.0 (from pytest)

Here's a report of the failing test remotely:
https://travis-ci.org/ColinKennedy/ways/jobs/308277578
That Travis-CI link refers to a log with more output. This pastebin link is that log: https://pastebin.com/rk8EUq0E

OS:
Ubuntu 16.04.2 LTS

I have a feeling this is simple user error on my part but it's hard to tell since I wiped my environment and am still getting the same error, locally and remotely. Any help would be greatly appreciated

[The-Compiler]

Support for Python 3.3 was dropped, see the changelog.

[nicoddemus]

@ColinKennedy if you are using the latest pip it should not try to install pytest-3.3.0 on Python 3.3. Make sure to update pip on Travis first with pip install --upgrade pip before installing your requirements.

[takluyver]

It looks like the metadata did not get published correctly when pytest 3.3.0 was uploaded. Compare py 1.5.2, which has 'Requires Python' metadata on the page, with pytest 3.3.0, which does not.

Unfortunately, this means that installing pytest fails on Python 3.3 rather than automatically falling back to an older version.

[RonnyPfannschmidt]

it seems the wheel metadata is incorrect

[takluyver]

Unfortunately, the only way I can think of to fix this is to make a new release of pytest which can be installed on Python 3.3, and has a version number higher than 3.3.0.

Otherwise, it might get fixed in pip if it gets a proper dependency resolver - that was a GSoC project this year, but I haven't heard much about it.

[nicoddemus]

I used the automatic procedure detailed in HOWTORELEASE to generate and upload the package, what could have caused the missing header? An old pip or wheel version installed in the virtual environment I used?

[RonnyPfannschmidt]

@nicoddemus pip cant cause it - wheel looked recent, perhaps setuptools?

[nicoddemus]

I will verify the actual versions when I get home to see if they were the cause.

[takluyver]

wheel, setuptools or maybe twine (if you're using that).

[nicoddemus]

Here's the pip list of the virtual environment I used:

Package             Version                         Location
------------------- ------------------------------- --------------------
apipkg              1.4
argon2-cffi         16.3.0
attrs               17.3.0
beautifulsoup4      4.6.0
bleach              2.0.0
certifi             2017.7.27.1
cffi                1.10.0
Chameleon           3.1
chardet             3.0.4
check-manifest      0.35
click               6.7
coverage            4.4.2
defusedxml          0.5.0
devpi               2.2.0
devpi-client        3.0.0
devpi-common        3.1.0
devpi-server        4.3.0
devpi-web           3.2.0
docutils            0.14
execnet             1.4.1
gitdb2              2.0.2
GitPython           2.1.5
html5lib            0.999999999
hupper              1.0
hypothesis          3.37.0
idna                2.5
incremental         17.5.0
invoke              0.20.4
itsdangerous        0.24
Jinja2              2.9.6
MarkupSafe          1.0
passlib             1.7.1
PasteDeploy         1.5.2
pip                 9.0.1
pkginfo             1.4.1
plaster             0.5
plaster-pastedeploy 0.4.1
pluggy              0.4.0
py                  1.5.0
pycparser           2.18
Pygments            2.2.0
pyramid             1.9.1
pyramid-chameleon   0.3
pytest              3.2.6.dev217+ge97c774.d20171123 /home/vagrant/pytest
readme-renderer     17.2
regendoc            0.6.1
repoze.lru          0.6
requests            2.18.3
requests-toolbelt   0.8.0
setuptools          36.2.7
six                 1.10.0
smmap2              2.0.3
toml                0.9.2
towncrier           17.8.0
tox                 2.8.1
tqdm                4.15.0
translationstring   1.3
twine               1.9.1
urllib3             1.22
venusian            1.1.0
virtualenv          15.1.0
waitress            1.0.2
webencodings        0.5.1
WebOb               1.7.3
wheel               0.29.0
Whoosh              2.7.4
zope.deprecation    4.3.0
zope.interface      4.4.2

Now with --outdated:

Package             Version                         Latest    Type  Location
------------------- ------------------------------- --------- ----- --------------------
bleach              2.0.0                           2.1.1     wheel
certifi             2017.7.27.1                     2017.11.5 wheel
cffi                1.10.0                          1.11.2    wheel
Chameleon           3.1                             3.2       sdist
check-manifest      0.35                            0.36      wheel
devpi-client        3.0.0                           3.1.0     wheel
devpi-common        3.1.0                           3.2.0     wheel
devpi-server        4.3.0                           4.3.1     wheel
devpi-web           3.2.0                           3.2.1     wheel
execnet             1.4.1                           1.5.0     wheel
gitdb2              2.0.2                           2.0.3     wheel
GitPython           2.1.5                           2.1.7     wheel
hypothesis          3.37.0                          3.38.9    wheel
idna                2.5                             2.6       wheel
invoke              0.20.4                          0.22.0    wheel
Jinja2              2.9.6                           2.10      wheel
plaster             0.5                             1.0       wheel
plaster-pastedeploy 0.4.1                           0.4.2     wheel
pluggy              0.4.0                           0.6.0     sdist
py                  1.5.0                           1.5.2     wheel
pytest              3.2.6.dev217+ge97c774.d20171123 3.3.0     wheel /home/vagrant/pytest
repoze.lru          0.6                             0.7       wheel
requests            2.18.3                          2.18.4    wheel
setuptools          36.2.7                          38.2.3    wheel
six                 1.10.0                          1.11.0    wheel
toml                0.9.2                           0.9.3.1   sdist
tox                 2.8.1                           2.9.1     wheel
tqdm                4.15.0                          4.19.4    wheel
waitress            1.0.2                           1.1.0     wheel
WebOb               1.7.3                           1.7.4     wheel
wheel               0.29.0                          0.30.0    wheel
zope.interface      4.4.2                           4.4.3     wheel

[takluyver]

Look at this file, I see that pytest releases are made through devpi. I've never used that; maybe it doesn't propagate the 'Requires-Python' metadata to PyPI.

[RonnyPfannschmidt]

@takluyver the metadata is taken from the wheel, and the wheel doesnt contain it

[jaraco]

Unfortunately, the only way I can think of to fix this is to make a new release of pytest which can be installed on Python 3.3, and has a version number higher than 3.3.0.

Or alternatively, publish a new release 3.3.1 that has the proper metadata and pull 3.3.0 from PyPI any distributions that are missing the metadata (wheel only apparently).

[takluyver]

I think you would have to delete 3.3.0 entirely, both wheel and sdist, to solve the problem that way.

In the simple index, pip is looking for a data-requires-python attribute. You can see that no distributions currently have that (compare py). I think that the metadata from the first distribution to be uploaded is applied to that release as a whole.

[jaraco]

I dare say this issue is a critical one. Some projects have adapted by dropping support for Python 2.6 and 3.3. Others may have worked around it by pinning to older versions of pytest, which isn't a sustainable approach, especially when the metadata provides a better mechanism for filtering releases based on supported Pythons. I don't have access to manage pytest releases myself, or I'd help directly.

Is there any reason not to immediately cut a 3.3.1 from 3.3.0 and remove 3.3.0 from PyPI?

And to prevent the resurgence of this issue, I suggest that releases should be cut mechanically using infrastructure resources like Travis CI to cut releases in a well-defined and visible environment. I've applied this approach in all of the projects I maintain and I'd be happy to help set that up for pytest too.

[RonnyPfannschmidt]

@jaraco main blocker for this currently is the lack of commit/push automation in combination with regendoc/towncrier - else we could just use the combination of setuptools_scm and automate the uploads from travis

[jaraco]

pytest 3.3.1 was released and has the same issue, so that's another version that's going to need to be pulled to address this issue.

[nicoddemus]

Hi @jaraco!

Is there any reason not to immediately cut a 3.3.1 from 3.3.0 and remove 3.3.0 from PyPI?

We just released 3.3.1 yesterday with the correct metadata. 👍 If we all agree we should remove 3.3.0 from PyPI, let's do it then, I'm not 100% certain this is a good approach because I read somewhere online that you ideally should never drop a release from PyPI, but I remember the alternative was to just push a new release to solve the issue which is not possible in our case.

And to prevent the resurgence of this issue, I suggest that releases should be cut mechanically using infrastructure resources like Travis CI to cut releases in a well-defined and visible environment. I've applied this approach in all of the projects I maintain and I'd be happy to help set that up for pytest too.

@RonnyPfannschmidt I think @jaraco's point here is that the package generation should be done in a controlled environment, not necessarily the entire process be automated. If that's the suggestion, I think we could just push tags to an approved "release PR" (#2997) and the package generation/publication could be done from Travis.

Besides the technical issues to address, there's some resistance from @hpk42 and @obestwalter (IIRC) about automating releases like this; they prefer some manual checking first.

Particularly I'm a big fan of the "push-tag-to-release" workflow. :)

[nicoddemus]

pytest 3.3.1 was released and has the same issue, so that's another version that's going to need to be pulled to address this issue.

Are you sure? I verified the package myself (both wheel and .tar.gz) and they did contain the "Python Requires" metadata. 😮

[jaraco]

Are you sure?

I'm not certain, but I don't see the Requires-Python in the metadata listing for pytest

Whereas I do see it for py.

[RonnyPfannschmidt]

@nicoddemus how did you upload them?

@jaraco when the wheel is uploaded as second file, pypi skips the data, also twine seems to be required
in the actual wheel the data is present this time

[nicoddemus]

Just verified again:

PKG-INFO in tar.gz has Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
pytest-3.3.1.dist-info/METADATA in the wheel also has this string

[jaraco]

pytest releases are made through devpi

Is it possible that this release flow is implicated, and that py doesn't use this flow?

[nicoddemus]

@RonnyPfannschmidt using devpi, according to our automated tasks.

Having the Requires-Python meta-data on the package is not sufficient?

Is it possible that this release flow is implicated, and that py doesn't use this flow?

I'm pretty sure py was uploaded using twine.

[nicoddemus]

@takluyver that seems possible, thanks for looking into it

[nicoddemus]

Just realized another disadvantage about dropping 3.3.0 and 3.3.1 from PyPI: that would break frozen "requirements.txt" of people that pin to those packages for deployment. ☹️

[jaraco]

Yes, it's disruptive... and the longer 3.3.0/1 is out there, the more the disruption. I suggest that 3.3.2 should be released for as long as 3.3.0 was released with a warning that those versions go away, then they go away.

[jaraco]

Alternatively, maybe we can ask PyPI if we can hack into their DB and update the metadata for those packages.

[nicoddemus]

A thought: would it be possible for people still supporting Python 2.6 and 3.3 to conditionally pin their tox.ini or requirements based on Python version? I ask because we did it for some time because mock dropped Python 2.6 support in mock 1.1:

pytest/tox.ini

Lines 30 to 36 in a220a40

	[testenv:py26]
	commands = pytest --lsof -rfsxX {posargs:testing}
	# pinning mock to last supported version for python 2.6
	deps =
	hypothesis<3.0
	nose
	mock<1.1

[fschulze]

I think one issue here is that it's not possible to update metadata on packages on PyPI anymore. Before one was able to fix the Readme or other stuff via the "register" call. That is now gone. I'm pretty sure nobody wants to put that back for PyPI and instead wants to get warehouse out and fix it there. So I think it would be reasonable to ask the to fix it in the Database for 3.3.0 and 3.3.1. In the meantime, I'm looking into the devpi push issue devpi/devpi#480

[fschulze]

Was something fixed on PyPI? The new pytest is now blocked on older PyPy3 https://travis-ci.org/fschulze/devpi/jobs/313430195

[fschulze]

Doh, I misunderstood. With the correct metadata it should use the older release.

[fschulze]

I updated devpi.net with a pre-release. I made a test upload of pytest at https://devpi.net/fschulze/dev/pytest/3.3.1 and it now has the metadata.

[obestwalter]

Besides the technical issues to address, there's some resistance from @hpk42 and @obestwalter (IIRC) about automating releases like this; they prefer some manual checking first.

No resistance from me, really. the more automatic a release is, the less one can do wrong :) togetherwith pre-releases that would actually be my ideal scenario.

[nicoddemus]

Alternatively, maybe we can ask PyPI if we can hack into their DB and update the metadata for those packages.
So I think it would be reasonable to ask the to fix it in the Database for 3.3.0 and 3.3.1.

That would be ideal solution: it will fix those releases without changing the actual package, and we won't risk further disruption for users using supported Python versions.

Do you guys know who we can contact regarding this request? @dstufft?

[fschulze]

Judging from https://pypi.org/help/ the warehouse issue tracker may be a good place. https://github.com/pypa/warehouse/issues

[nicoddemus]

Thanks @fschulze! Opened up pypi/warehouse#2700.

[RonnyPfannschmidt]

@nicoddemus so what do we do about the bad uploads - they break the builds and by now pulling will break pinned builds

[nicoddemus]

The warehouse guys seem to be happy to help us out, so we should go with that solution. Em dom, 24 de dez de 2017 08:32, Ronny Pfannschmidt < notifications@github.com> escreveu:

…

@nicoddemus <https://github.com/nicoddemus> so what do we do about the bad uploads - they break the builds and by now pulling will break pinned builds — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2966 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABCO_InoXUlrr-nwKKUf7Z_9eW0faYyZks5tDihRgaJpZM4QtzSl> .

[nicoddemus]

3.3.2 is out, we should be good as soon as @ewdurbin gets a chance to update the metadata. 👍

[nicoddemus]

Thanks to the awesome help from @ewdurbin the metadata is now fixed for 3.3.0 and 3.3.1! 👍

[jaraco]

Thanks for all the hard work on this one.

[chinu23081994]

Just verified again:

PKG-INFO in tar.gz has Requires-Python: >=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*
pytest-3.3.1.dist-info/METADATA in the wheel also has this string

metadata is missing