Dropping pypy from our test matrix

[sirosen]

📣 I am advocating for removal of pypy from our matrix. Details and rationale follow.

@webknjaz, I'm seeing CI failures starting today which appear to be specific to pypy on Windows. (example)
They may or may not be resolvable -- I haven't even tried yet -- but I also think it's not worth the effort because I have been thinking for a while that we should drop pypy from our matrix anyway.

For broader context, some of the pypy maintainers themselves are stepping back from the project (although reports of pypy's death seem to be exaggerated), conda-forge dropped pypy, and pip itself does not test against pypy.

On the one hand, pypy being in the matrix improves our compatibility posture. On the other hand, it imposes ongoing cost, and our underlying dependencies are not testing against it, so it increases our build fragility. I think the cost/benefit tradeoff favors removal.

[webknjaz]

I wouldn't drop it but rather bump to pypy-3.11, keeping it non-EOL: pypa/cibuildwheel#2518. We should probably bump endoflife-date/endoflife.date#3364 that requests an EOL tracking page similar to https://endoflife.date/python, otherwise, it's sometimes difficult to say what's EOL w/o scanning each release notes entry or watching what cibuildwheel drops…

@StanFromIreland has recently become a PyPy Core Dev, AFAIU.

[StanFromIreland]

Hi!

Indeed the project has slowed down, but it sure ain't dead :-) I will admit I'm obviously biased, but I suggest keeping PyPy compatibility. I also suggest updating to PyPy3.11, however. If PyPy's flaky and causing issues, then I'd suggest not making it a required check.

If you find any bugs in PyPy itself, please open an issue!

[sirosen]

Okay! I am well enough convinced that we can keep it but switch to allow failures if we must.

We're continuing to work on our broken build. pypy is only loosely implicated there -- our tests are known-flaky and for some reason (which we don't fully have a handle on yet) are failing consistently on a Windows x PyPy case. Once we have the busted build under control, I want to circle back on this discussion and lock-in a decision for the short-to-medium term.

@StanFromIreland, by way of encouragement, many of us really like having a second implementation we can test against! Even if we aren't active PyPy users, it's important to us from an ecosystem perspective.
My desire to drop builds is primarily in pursuit of stable CI. If we can't keep CI green, we aren't really doing good and useful testing anyway! 😅

[StanFromIreland]

That's good to hear :-)