refactor _generic_api to use EXT_SUFFIX

[mattip]

Fixes #606 where PyPy would like to change its non-compliant SOABI tag (pypy36-pp73) to a compliant one (pypy39-pp73-x86_64-linux-gnu).

The basic code is taken from wheel.bdist_wheel's version.
changed to use EXT_SUFFIX instead of SOABI.

A few other changes:

• return a list instead of a iterator, since all uses of the function converted to a list anyway
  - add a fallback for older cpython to call _cpython_abis, which means passing in PythonVersion
• clarify the typical value of interpreter_name, since it actually is the 'cp' or 'pp' short name.

If I was starting over, PyPy's ABI tag would be pp39_73 rather than pypy39_73 to save two letters. Maybe someday ...

Edit (11-Nov-2022): refactored to use EXT_SUFFIX

[uranusjr]

Does this not need to still support the old format so currently existing wheels can still work?

[mattip]

Does this not need to still support the old format so currently existing wheels can still work?

Yes. The code should be backward compatible: the names should not change.

[mattip]

Does this not need to still support the old format so currently existing wheels can still work?

I added tests to make sure that on PyPy the current SOABI and the future SOABI give the same value as they currently do. There should be no change here for CPython.

[henryiii]

According to PEP 3149, EXT_SUFFIX and SOABI are equivalent, minus the extension. If they are not, it's a bug - and historically, EXT_SUFFIX is much more likely to be right:

For future reference going forward, sysconfig.get_config_var('EXT_SUFFIX') is the recommended way to spell this for CPython3.8+ and all current versions of PyPy. See CPython issues https://bugs.python.org/issue39825 (for windows) and https://bugs.python.org/issue42604 (for FreeBSD and AIX).

There's also a chance SOABI will be blank on Windows, while EXT_SUFFIX is safe. Why not just use EXT_SUFFIX, then always strip the extension and the platform part? Then you don't need the conditional logic to see if there is a platform part.

[mattip]

Where is the quote from? I don't see it in the linked PEP Whoops - you were quoting me on the cmake issue https://gitlab.kitware.com/cmake/cmake/-/issues/21070#note_876947.

[henryiii]

Yes, sorry, there's a link just below to the same issue, one comment down.

[mattip]

I wonder where I got that information (that EXT_SUFFIX is to be preferred over SOABI). It seems that in the posix configure script SOABI comes first. I do agree that if they are not equivalent (plus-minus a suffix) it is a bug, so it would seem for the purposes of this PR that SOABI is easier, no need to remove the suffix.

[henryiii]

I would assume it's because EXT_SUFFIX is used when loading the module, so it has to be correct. While SOABI is misused here, so any implementation (PyPy, and maybe all others?) had to have incorrect SOABI's in the past to make packaging tags work. Are there other implementations to check? Pyodide is correct, but it doesn't use Pip, and it counts as cpython, so there's no issue. I tried to install IronPython 3.4, but have no idea where it went after installing it. ¯\_(ツ)_/¯

Also the warning for using SO is:

DeprecationWarning: SO is deprecated, use EXT_SUFFIX

So it's pointing to EXT_SUFFIX as the replacement, not SOABI.

[mattip]

@henryiii is there something to change in this PR to answer your comment?

[mattip]

I wouldn't want to switch SOABI for EXT_SUFFIX without tacit approval from the package maintainers.

[henryiii]

My question was why not use EXT_SUFFIX? It is much more likely to be right, and it's suggested as the replacement for SO anyway, even though SOABI is a closer match. Then you don't have to do special processing to check to see if it's a malformed SOABI, you can just strip the extension suffix & leading dot and now you know it's a valid SOABI.

[henryiii]

Without approval from packaging maintainers, the current version is better than the status quo, so no unless packaging maintainers chime in. Setuptools/distutils rely on EXT_SUFFIX, like in https://github.com/pypa/setuptools/blob/a2c4c578e4921be7670448fc1cbf3f81792d86e4/setuptools/_distutils/command/build_ext.py#L710.

[brettcannon]

I wouldn't want to switch SOABI for EXT_SUFFIX without tacit approval from the package maintainers.

If it leads to better results I'm up for trying it out. Is there anyone else to loop in here and ask how they might be affected?

[henryiii]

Are there any other Python implementations that might be using this? I couldn't figure out how to find IronPython 3.4 after installing it (and besides, that currently maxes out at Python 3.4, this would only affect 3.7+), and pyodide doesn't use this AFAICT (it "is" CPython in a sense).

[mattip]

I am refactoring this to use EXT_SUFFIX. I changed the description and title to match.

[mattip]

Is this correct or would the reviewers prefer the function return [] for an invalid EXT_SUFFIX (which should never happen)? I prefer exceptions when reality clashes with my expectations, but maybe that is too optimistic?

[brettcannon]

If you can't function w/o the information then an exception makes sense (have not thought about which one).

[mattip]

CI is passing.

[isuruf]

cc @timfel from GraalPy

[brettcannon]

Looks like @timfel from GrallPy is the only person anyone could think of, so let's give them a chance to comment and then we can go from there.

[timfel]

I'm happy with this for GraalPy.

[brettcannon]

Maybe someday sysconfig will could grow new functions python_425_tag, abi_425_tag and platform_425_tag to report the appropriate PEP 425 tags. Then the interpreter would be self-reporting and this would all become redundant.

Please feel free to open an issue at python/cpython if we want to generalize some things, but moving all tag code upstream might not be doable as I can see some folks getting upset with embedding that much packaging knowledge into the stdlib (but we won't know until we ask 🤷).

[brettcannon]

So, are we all in agreement on this PR? Last chance to speak up before I begin reviewing for technical/code reasons and not on results reasons.

[dnicolodi]

A version of meson-python has been released incorporating changes in the spirit of this PR. I don't think we will go back on these changes. Note that this means that SciPy (and other less prominent packages) is now uninstallable on GraalPy, unless meson-python is pinned to a previous version or other tricks are applied.

[brettcannon]

@dnicolodi

Note that this means that SciPy (and other less prominent packages) is now uninstallable on GraalPy

... until the PR is merged and released, correct?

[dnicolodi]

Note that this means that SciPy (and other less prominent packages) is now uninstallable on GraalPy

... until the PR is merged and released, correct?

Correct. But that's going to be a while.

[mattip]

Squashed review commits and rebase off main to clear merge conflict

[pradyunsg]

@brettcannon I'm gonna defer to you on this one completely. :)

[pradyunsg]

@mattip @brettcannon Any suggestions on what would be a good changelog entry for this?

[mattip]

"Refactor _generic_api to use EXT_SUFFIX instead of SOABI"

[pradyunsg]

Hmm... I'm not sure that means much for end users, and mentioning an internal function name in the changelog isn't particularly meaningful either...

What's the user-facing behaviour change here?

[mattip]

Hopefully this change has absolutely no user facing changes. It is a code refactor to make the code more robust to the single-source of truth (EXT_SUFFIX) instead of using aliases (SOABI) that may be wrong.