Add image for musllinux_1_2 policy

[repsejnworb]

Added a musllinux_1_2 policy

This is more of an Alpine thing than musl.
Some wheels built with musllinux_1_1 (Alpine 3.12) don't work on Alpine 3.14+ due to https://git.alpinelinux.org/aports/tree/main/python3/bpo-43112.patch?h=v3.14.2

On that same note there is python/cpython#24502

[repsejnworb]

Moved to get access to BASE_POLICY from utils

[henryiii]

How often are new versions released? (That is, how many years between 1_1 and 1_2?)

[repsejnworb]

How often are new versions released? (That is, how many years between 1_1 and 1_2?)

https://musl.libc.org/releases.html
I'm not more familiar with it than that listing.

[henryiii]

Okay, 2014 for 1_1 and 2020 for 1_2.

This release moves all 32-bit archs to 64-bit time_t, enabling them to represent times beyond January of 2038.

There are no new requirements on kernel version, and this is not a hard ABI break, but the type changes do impact compatibility between code built against previous versions of musl and code built against musl 1.2. Users upgrading 32-bit systems should read the detailed time64 release notes. 64-bit systems are not affected.

Pretty small change causing the 1.1 to 1.2 change, through!

[repsejnworb]

Pretty small change causing the 1.1 to 1.2 change, through!

Oh I see now that I forgot to add some description as to why I made the PR, I've updated it now.
This is more an Alpine thing than it is musl.

[repsejnworb]

I'm unsure why the ppc64le build failed, https://github.com/pypa/manylinux/runs/4253538659, any hints?

The openssl package is the same for Alpine-3.12 and Alpine-3.14, and me just quickly doing the checks on a ppc64le/alpine:3.14 image similar to what build-openssl.sh does works (as it seems to fail on the if [ "${SMALLEST}" = "${OPENSSL_MIN_VERSION}" ]; then ?)

[dvarrazzo]

I have recently heard about this issue and verified that it affects the psycopg 3.0.3 and 3.0.4 binary packages, which distribute musllinux wheels too.

I made a test, described here and ISTM that just distributing both the names works. It works with a symlink too, but I'm not sure about the portability of zipping a symlink.

Does creating a musllinux_1_2 really solve the problem? Does it need to be accompanied by a suitable pip new version, understanding what tag to require based on the SOABI? I can see a long and painful transition period here.

Do you suggest to withdraw the existing musllinux packages for 3.9, 3.10, and wait for the new tag, or to release 3.0.5 with both the names available?

[repsejnworb]

Does creating a musllinux_1_2 really solve the problem?

Well rather it is "solved" by building with an Alpine that has the patch (https://git.alpinelinux.org/aports/tree/main/python3/bpo-43112.patch?h=v3.14.2).
It is just a circumstance that this also includes an updated musl .

Does it need to be accompanied by a suitable pip new version, understanding what tag to require based on the SOABI? I can see a long and painful transition period here.
Do you suggest to withdraw the existing musllinux packages for 3.9, 3.10, and wait for the new tag, or to release 3.0.5 with both the names available?

I don't know :0

It is a weird situation and I am not frequent in open source etc.
I just solved my own situation this way and I don't see another way forward - so I thought I'd make a PR.

[lithammer]

I'll try to shed some light on why this is currently necessary. The problem is that current musllinux_1_1 wheels won't install on Alpine 3.14 because of what's described here:

• https://discuss.python.org/t/a-mess-with-soabi-tags-on-musllinux/11688
• bpo-43112: detect musl as a separate SOABI python/cpython#24502 (seems like Alpine has patched this themselves as of 3.14)

In short, the SOABI tag on Alpine 3.12 (the current base image used for musllinux_1_1) is gnu. And in 3.14 it's musl, so the wheel will install just fine but then cause an ImportError at runtime because it doesn't find the correct .so file(s).

One way to solve this is to build on 3.14, but that requires the musllinux_1_2 tag (because 3.14 has musl-1.2). However, since musl-1.2 was actually introduced in 3.13 that still uses the gnu SOAPI, it gets tricky.

It's a bit of a mess right now it seems. Coverage solved it by not publishing Python 3.9 wheels because that's the Python version on Alpine 3.14 (this is also the solution I suggested for psycopg).

I'm not sure if Python and/or Alpine has any plans to backport the SOAPI fixes to earlier versions of Python/Alpine either.

I made a test, described here and ISTM that just distributing both the names works. It works with a symlink too, but I'm not sure about the portability of zipping a symlink.

I guess this is the other solution. But that's more of a workaround than a fix for the future.

Does creating a musllinux_1_2 really solve the problem? Does it need to be accompanied by a suitable pip new version, understanding what tag to require based on the SOABI?

Yes it solves the problem for Alpine 3.14 (and any future versions on musl-1.2). Or the solution is actually to build on 3.14, the musllinux_1_2 tag is just a necessary side effect of that. No new Pip version is required.

[dvarrazzo]

One way to solve this is to build on 3.14, [...] However, since musl-1.2 was actually introduced in 3.13 that still uses the gnu SOAPI, it gets tricky.

That's what I mean: musl 1-2 has no relation with the SOAPI, so it will only be a solution when a few things will upgrade, which I understand to be 1) python fixing the broken SOAPI and 2) manylinux images released with such python.

I guess [releasing both names] this is the other solution. But that's more of a workaround than a fix for the future.

I don't advocate this as a long term solution. Just as a stitch for the present, to bridge creating 1-1 packages using the current infrastructure and working both with Python releases affected by BPO 43112 and when they will be fixed. I don't think the hack will be needed for 1-2, as the only broken combo is musl 1-1 + gnu SOAPI.

IMO this would make the experience of the end users much smoother, as they will have familiar releases fixing bugs ("psycopg 3.0.5 fixes the bug with manylinux such and such", instead of withdrawing packages from pypi, which frankly is unheard of, by me at least) and for packagers as they won't have to feel the blunt of p*ssed off users whose working workflows have been broken.

Or the solution is actually to build on 3.14, the musllinux_1_2 tag is just a necessary side effect of that.

These packages wouldn't work on python:3.9-alpine and 3.10 docker images, (which is worth noticing are the ones most likely ones being used right now), as the SOAPI will mismatch the other way, but will start working after a certain bugfix python release, correct?

[henryiii]

Adding a new musllinux_1_2 platform will add a ton of new binaries to be stored on PyPI, extra platforms to be maintained here, and extra building time across all the CI systems for everyone who wants to support both (and many packages will likely want to). I don't think this is a a great solution for a bug unrelated to the musl version. If it enabled C++2x or something, but not just to solve a bug. It seems to me a better solution would be possibly in auditwheel, to fix the incorrect names and then add a symlink to the names. Maybe we could try to get Alpine to backport the patch to this version? Etc.

[henryiii]

Also, note that this is also the wrong meaning for a new distribution - manylinux1 works on manylinux2010. While in this case, musllinux_1_2 and musllinux_1_1 are completely incompatible, and all maintainers are expected to either build both or build just 1_2, rather than building 1_1 and having that be forward compatible.

[lithammer]

It seems to me a better solution would be possibly in auditwheel, to fix the incorrect names and then add a symlink to the names. Maybe we could try to get Alpine to backport the patch to this version? Etc.

That sounds like a good approach!

[repsejnworb]

It seems to me a better solution would be possibly in auditwheel, to fix the incorrect names and then add a symlink to the names. Maybe we could try to get Alpine to backport the patch to this version? Etc.

That sounds a lot better than this yeah.

Let's close this PR in favor of that.

[dvarrazzo]

Auditwheel would have the problem that the .whl files are 🤬 zip, which don't support symlinks well (not the zipfile module at least: see https://bugs.python.org/issue18595, https://bugs.python.org/issue27318).

A better place to fix the issue might be pip: my proposal at pypa/pip#10678.

[henryiii]

This has been fixed by Alpine 3.14 and 3.15; they can load musllinux wheels again with the official SOABI, even though they still produce binaries with an unsupported SOABI. This should not be merged, besides not adding anything useful in terms of new features, if we are using official CPython, this would not change the SOABI and would be identical to musllinux_1_1, and if we are using Alpine's built-in Python, that would produce a currently invalid SOABI that would not load with any other CPython, like from python.org.

[repsejnworb]

Fantastic job getting it all sorted at the correct place!

Closing my PR.