AppVeyor CI: Add Python 3.11 jobs

[EwoutH]

AppVeyor deployed new Windows images with Python 3.11 support (appveyor/ci#3844), which means we can use it to build Python 3.11 Windows wheels for lxml. This PR adds three Python 3.11 jobs to the matrix, for the x86, x86-64 and arm64 platforms

Part of Bug #1977998. Partly replaces #355.

I tested the jobs on my branch, and the workflow passes.

I would suggest after this PR, to backport it to the 4.9 maintenance branch and release a new 4.9.2 version which includes these Python 3.11 Windows wheels.

[cclauss]

@scoder Your review, please.

[cclauss]

LGTM

[armando-herastang]

Great. Waiting on these Python 3.11 Windows wheels for lxml. Have some workflows and apps not being able to build from source.

[govindashrit]

Waiting !!!!

[cclauss]

@armando-herastang @govindashrit Please go to https://github.com/lxml/lxml/pull/360/files and click Review changes and then Approve to give project maintainers confidence that you have studied these changes and are confident that they are safe and will improve the project.

[armando-herastang]

LGTM

[jgstew]

just a change to which versions are built. Would really like to see this from lxml going forward as windows builds are kind of a pain.

[EwoutH]

Hi @scoder! This PR got quite a bit of interest, and is as you can see quite a small CI change. Could you review this PR?

[govindashrit]

Looks fine.

[EwoutH]

Hi @scoder! This has now been a week in production, and it looks quite stable! Could a new patch release (4.9.2) be tagged and the Python 3.11 wheels be uploaded? It would be great to have them on PyPI.

[zlogz101]

@EwoutH +1 Looking forward to getting 4.9.2 release with windows Py3.11 wheel included. To be honest, it blocked me to upgrade Py3.11 due to missing wheel.

[cm253]

Also waiting for the lxml 4.9.2 wheels for Py3.11, so we can switch to Python 3.11 finally.

[elandorr]

Could someone please explain where exactly this process got stuck yet again? The one place where wheels really matter, there aren't any: Windows. (Due to current Pythonland actively refusing to cooperate with mingw, and MS spyware not being acceptable.)

Normally we don't rush to update, but it's been half a year, and 3.11 has some nice speed gains by default.

Thanks

[cm253]

Normally we don't rush to update, but it's been half a year, and 3.11 has some nice speed gains by default.

I think there are quite a lot of Python libs that depends on lxml, so this is also some kind of blocker for upgrading to Python 3.11, at least for heterogeneous dev teams developing on/for the Windows platform.

We would also like to move to Python 3.11 because of the performance gains.

[scoder]

Could someone please explain where exactly this process got stuck yet again?

See #355.

[scoder]

The one place where wheels really matter, there aren't any: Windows.

Just to counter that argument, wheels definitely matter much more on Linux than on Windows (or macOS). The amount of computing time saved by downloading lxml's Linux wheels, compared to downloading sources and building them, is tremendous. Here are the PyPI download stats for the last month, showing 90% Linux downloads, 2% Windows, and 0.6% macOS, where the remaining 7% are probably source downloads:

Downloads by system:
  Linux  :    1710444.5 / day ( 90.1%)
  Windows:      37564.7 / day ( 92.1%)
  Darwin :      12941.5 / day ( 92.7%)
  null   :     137846.6 / day (100.0%)
  other  :        163.1 / day (100.0%)

[elandorr]

Could someone please explain where exactly this process got stuck yet again?

See #355.

Thanks @scoder. Trying to help as an outsider is not useful, then.

Just to counter that argument, wheels definitely matter much more on Linux than on Windows (or macOS). The amount of computing time saved

You're certainly right about computing time and energy. However, my point is usability. On Linux, the worst case is usually quite okay. If there's no exotic dependency hell, it's usually as simple as installing your distro's equivalent of build-essential.
You can tell other people to do that, it works with default repos, manageable.

On Windows it's a nightmare. There simply is no way to do it without violating the system's integrity - at all. Best you can do is use this: https://github.com/throwawaymicrosoft1/Visual-CPP-Build-Tools-OFFLINE-installer. And that's so far beyond reason, unless I really desperately need a lib right this second, that's not coming on any system I admin. It's faster to even create a tiny Linux VM.

If distutils/setuptools supported an acceptable standard like MinGW as it did in the past, this issue wouldn't be a total roadblock.
MinGW links to e.g. https://winlibs.com/. Download exactly what you need, start compiling in 5 minutes. No spyware, no bloat, handles every standard setup OOTB. Works well, but not in Pythonland.

[EwoutH]

@scoder I would suggest getting Python 3.11 wheels out first, because as seen above, a lot of users are requesting them. Then when that's done we can look at upgrading to VS 2022 (#355 and lxml/libxml2-win-binaries@3fb0529).

This AppVeyor build passed, so I would suggest tagging e8f088a as 4.9.2, and uploading these artifacts to PyPI.

[bastimeyer]

@EwoutH 4.9.2 has already been published:
https://github.com/lxml/lxml/releases/tag/lxml-4.9.2
https://pypi.org/project/lxml/4.9.2/#files

[elandorr]

Thank you! Our windows stuff is working again.

[EwoutH]

4.9.2 has already been published:

Didn't see it yet, awesome!

[cm253]

We upgraded now to Python 3.11 on all platforms, lxml 4.9.2 works everywhere now 🥇

[anthrotype]

everywhere.. except arm64 architecture on macOS (Apple Silicon M1/M2 chips) 😿

the cp311-universal2 wheel is broken, it pip installs fine but then it fails with ImportError -- dlopen complains about missing symbols supposedly from libxml2, even though it seems to be statically linked (doesn't show up in otool -L).
If I force python to run in arch -x86_64 emulation mode, it can successfully import lxml.etree

https://bugs.launchpad.net/lxml/+bug/1913032

[cm253]

everywhere.. except arm64 architecture on macOS (Apple Silicon M1/M2 chips) 😿

I'm using a M1 chip. Looks like I had 4.9.1 installed. Removed it and installed the 4.9.2 wheel, and now I also get the error 😢

lxml/etree.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_exsltDateXpathCtxtRegister'

[cclauss]

Can you revert with python3.11 -m pip install lxml==4.9.1 ?

[anthrotype]

forcing to build from source with pip install --no-binary=lxml lxml==4.9.2 also works for me, but defeats the purpose of having the binary wheel. I think it's better that it is removed from PyPI.

[cm253]

Or in my case PIP_NO_BINARY=lxml pipenv install lxml==4.9.2 worked for me.