gh-88402: Add new sysconfig variables on Windows (GH-110049) #110049

colesbury · 2023-09-28T19:39:56Z

This adds LIBRARY, LDLIBRARY, LIBPL, SOABI, and Py_NOGIL variables to sysconfig on Windows. Note that Py_NOGIL is only defined in --disable-gil builds.

Issue: Define SOABI, LIBRARY, LDLIBRARY and LIBPL on Windows #88402

This adds `LIBRARY`, `LDLIBRARY`, `LIBPL`, `SOABI`, and `Py_NOGIL` variables to sysconfig on Windows. Note that `Py_NOGIL` is only defined in `--disable-gil` builds.

colesbury · 2023-09-28T19:40:59Z

Lib/sysconfig.py

+        vars['SOABI'] = soabi
+
+        # e.g., check for a "t" (for "threading") in SOABI
+        if re.match(r'cp\d+t', soabi):


The string processing is a bit icky. It would be convenient to just pass these values (SOABI, Py_NOGIL) from a C to Python, but it wasn't clear to me where to put that (i.e., in sys, _imp, _winapi or someplace else)

I agree. Maybe a good option would be putting them on _winapi and then simply loading them in _init_non_posix?

sys.flags?

Isn't sys.flags for command line flags?

sys.implementation then? We added these namespaces specifically to be extensible

But this information (SOABI, Py_NOGIL, etc) doesn't seem quite right there, I think. Though maybe this is just my interpretation.

I have a prototype locally to export this information in _winapi, avoiding any public API changes. It doesn't feel amazing either, but 🤷.

Really, this should just be exposed by sysconfig directly, which may involve a C _sysconfig module that exports this information out to Python, or something else. It's unfortunate that there's such a large discrepancy between how Makefile and Windows builds work 😞. This is kind of a mess, and we should probably try to unify things, it's just... hard, and a lot of work. I'm a bit tired of all this 😕

I moved some of these to _winapi. A _sysconfig C module might also make sense, but it would be very small.

Lib/sysconfig.py

Lib/test/test_sysconfig.py

Misc/NEWS.d/next/Library/2023-09-28-12-32-57.gh-issue-88402.hoa3Gx.rst

Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>

zooba · 2023-10-04T16:07:45Z

Modules/_winapi.c

+[clinic start generated code]*/
+
+static PyObject *
+_winapi__sysconfig_vars_impl(PyObject *module)


I don't like this being here, I'd rather it be in the sys module. It's not a Windows API

What do you think about putting it in a _sysconfig C module?

I think we're probably past due having one of those. We already smuggle constants into getbuildinfo.c (git info) and sysmodule.c (VPATH) on Windows, so may as well put them all into a _sysconfig.c that is importable.

Make it a built-in module though, not its own .pyd. PC/config.c lists the modules that are directly linked in, so it'll look like one of those.

You can see the smuggling in PCbuild/pythoncore.vcxproj (search for GITVERSION= and VPATH=). Doesn't necessarily have to be done the same way, but copying it is going to be the easiest way to ensure incremental builds keep working normally.

I've moved the computation to a _sysconfig C module. Thanks for the pointer to PC/config.c -- that made it easier. I don't think we need to smuggle data from PCbuild/pythoncore.vcxproj like we do for GITVERSION and VPATH here.

zooba · 2023-10-04T20:47:35Z

In principle, I'm happy with this change (and if we don't move more build-related variables into _sysconfig now, then let's leave an open issue to do it at some point in the future).

I'll need more time to do a thorough review, but if someone else is happy with the details, don't wait for me.

FFY00 · 2023-10-04T20:56:24Z

Lib/sysconfig.py

+    # Add EXT_SUFFIX, SOABI, and Py_NOGIL
+    vars.update(_sysconfig.config_vars())


(note for the future)

I think we can move this to _init_config_vars, I am sure it will end up there soon anyway. The only thing to keep in mind is that we allow the POSIX config vars module (eg. _sysconfigdata__linux_x86_64-linux-gnu) to be overwritten (to help in cross-compilation) by setting _PYTHON_SYSCONFIGDATA_NAME, and we should raise an error if the value given by the user is different from what we computed.
To be even more cautious, I think we should probably save the original sysconfigdata name in the C module, as attributes like sys.implementation, etc., are often monkey patched when cross-compiling. If needed, people working on cross-compilation should also monkey patch the _sysconfig module explicitly, or better, just monkey patch sysconfig.get_config_vars.

You don't need to do it in this PR, I just wanted to write this down, especially to document the non-obvious issue with _PYTHON_SYSCONFIGDATA_NAME.
I don't think relying on Makefile variables is great, for a ton of reasons, meaning the new sysconfig API (GH-103480) will very likely use _sysconfig.config_vars on other platforms, so I expect the _PYTHON_SYSCONFIGDATA_NAME issue to be relevant soon.

FFY00

Thanks @colesbury 😊

I think the PR is in good enough shape to merge.

A possible improvement would be moving the suffix calculation from dynload_*.c to pycore_internaldl.h, which would allow us to pull both the SOABI and EXT_SUFFIX values directly, without having to add C globals to the module implementation.

Modules/_sysconfig.c

FFY00 · 2023-10-04T21:28:04Z

Modules/_sysconfig.c

+    if (_PyImport_DynLoadFiletab[0]) {
+        if (add_string_value(config, "EXT_SUFFIX", _PyImport_DynLoadFiletab[0]) < 0) {
+            Py_DECREF(config);
+            return NULL;
+        }
+    }


This is one of the things I wanted to avoid 😅

I think it's probably okay, but one thing to keep in mind is that EXT_SUFFIX is usually used as the extension to give native modules. We likely want to make sure we are setting it to the one that includes SOABI, not a different one (eg. the abi3 one). In this case, I think we should probably be okay since we know Windows will always have one, and put it in the first entry.

Modules/_sysconfig.c

Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>

It should be zero or one and always defined.

FFY00 · 2023-10-04T21:49:19Z

@colesbury could you please let me know if you are happy to merge this now, or if you still want to make any changes based on the rest of the review feedback? Any of the mentioned possible improvements could be done later, so I don't want to block this. I'm just asking if you are planning on updating the PR further so that I don't merge it before you have a chance to push the changes. Thanks!

colesbury · 2023-10-04T21:54:16Z

@FFY00 - I'm happy to make the suggested improvements.

colesbury · 2023-10-04T21:59:23Z

@FFY00, ok I'm done making the changes from your feedback (assuming the tests pass). Let me know if this matches what you were thinking.

FFY00 · 2023-10-04T22:03:32Z

It looks good, I am gonna schedule it for merging. Thanks again!

FFY00 · 2023-10-04T22:05:46Z

The generated files check seems to be failing due to the clinic output not being up-to-date.

The C _sysconfig module no longer defines EXT_SUFFIX for POSIX systems

vstinner · 2023-10-05T23:19:50Z

#define PYD_TAGGED_SUFFIX PYD_DEBUG_SUFFIX "." PYD_SOABI ".pyd"

Would it be possible to move the d tag as part of the "SOABI" flags? Maybe it's time to add sys.abiflags to Windows?

FFY00 · 2023-10-05T23:23:43Z

Yes, I think that makes sense, particularly, I'd like to align the Windows tags with POSIX. I just haven't gotten around to look deeper into the reason we do it this way currently.

zooba · 2023-10-06T13:48:35Z

The tag is actually _d and not just d. I think we need a deprecation cycle to change the naming convention here (and there are going to be a significant number of flow on changes in the launcher and various tests, so be prepared for a bit more work than just one define).

pythongh-88402: Add new sysconfig variables on Windows

30814e4

This adds `LIBRARY`, `LDLIBRARY`, `LIBPL`, `SOABI`, and `Py_NOGIL` variables to sysconfig on Windows. Note that `Py_NOGIL` is only defined in `--disable-gil` builds.

colesbury added OS-windows topic-sysconfig 3.13 bugs and security fixes labels Sep 28, 2023

colesbury requested a review from FFY00 as a code owner September 28, 2023 19:39

bedevere-app bot added the awaiting review label Sep 28, 2023

bedevere-app bot mentioned this pull request Sep 28, 2023

Define SOABI, LIBRARY, LDLIBRARY and LIBPL on Windows #88402

Closed

colesbury commented Sep 28, 2023

View reviewed changes

Lib/sysconfig.py Show resolved Hide resolved

colesbury commented Sep 28, 2023

View reviewed changes

Lib/sysconfig.py Show resolved Hide resolved

colesbury mentioned this pull request Sep 29, 2023

--disable-gil test failures due to ABI flag #110119

Closed

FFY00 mentioned this pull request Oct 1, 2023

python module: migrate away from the use of distutils mesonbuild/meson#11133

Merged

FFY00 reviewed Oct 1, 2023

View reviewed changes

Lib/sysconfig.py Outdated Show resolved Hide resolved

FFY00 reviewed Oct 1, 2023

View reviewed changes

Lib/test/test_sysconfig.py Show resolved Hide resolved

FFY00 reviewed Oct 1, 2023

View reviewed changes

Misc/NEWS.d/next/Library/2023-09-28-12-32-57.gh-issue-88402.hoa3Gx.rst Outdated Show resolved Hide resolved

Update 2023-09-28-12-32-57.gh-issue-88402.hoa3Gx.rst

3f2a744

Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>

FFY00 mentioned this pull request Oct 2, 2023

Are static library builds on Windows still supported? #110234

Open

colesbury mentioned this pull request Oct 4, 2023

test_importlib and test_peg_generator fails on AMD64 Windows Server 2022 NoGIL 3.x #110359

Closed

Compute some sysconfig variables in _winapi

0ce92b2

colesbury mentioned this pull request Oct 4, 2023

PEP 703 -- Making the Global Interpreter Lock Optional in CPython #108219

Open

colesbury requested review from brettcannon, ericsnowcurrently, ncoghlan and warsaw as code owners October 4, 2023 14:35

colesbury requested a review from FFY00 October 4, 2023 14:37

zooba reviewed Oct 4, 2023

View reviewed changes

brettcannon removed their request for review October 4, 2023 16:30

colesbury added 2 commits October 4, 2023 15:41

Move Python/importdl.h to Include/internal/pycore_importdl.h

aa396e5

Add _sysconfig module

89f891c

colesbury requested a review from a team as a code owner October 4, 2023 20:10

colesbury requested a review from kumaraditya303 as a code owner October 4, 2023 20:10

colesbury requested a review from zooba October 4, 2023 20:13

FFY00 reviewed Oct 4, 2023

View reviewed changes

FFY00 approved these changes Oct 4, 2023

View reviewed changes

bedevere-app bot added awaiting core review and removed awaiting review labels Oct 4, 2023

colesbury and others added 2 commits October 4, 2023 17:39

Update Modules/_sysconfig.c

14d201c

Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>

Fix "Py_NOGIL" sysconfig variable.

34fe412

It should be zero or one and always defined.

Compute PYD_TAGGED_SUFFIX and PYD_SOABI in pycore_importdl.h

db30ede

FFY00 changed the title ~~gh-88402: Add new sysconfig variables on Windows~~ gh-88402: Add new sysconfig variables on Windows (GH-110049) Oct 4, 2023

FFY00 enabled auto-merge (squash) October 4, 2023 22:03

colesbury added 2 commits October 4, 2023 18:05

regen clinic

f0fb235

Fix sysconfig test on POSIX systems.

0a84d76

The C _sysconfig module no longer defines EXT_SUFFIX for POSIX systems

FFY00 merged commit cf6f23b into python:main Oct 4, 2023
23 checks passed

bedevere-app bot removed the awaiting core review label Oct 4, 2023

colesbury deleted the sysconfig_win branch October 6, 2023 15:40

JamesParrott mentioned this pull request May 25, 2024

sysconfig.get_config_var("LDLIBRARY") is now defined in Windows Python v3.13.0b1 spiraldb/ziggy-pydust#343

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-88402: Add new sysconfig variables on Windows (GH-110049) #110049

gh-88402: Add new sysconfig variables on Windows (GH-110049) #110049

colesbury commented Sep 28, 2023 •

edited by bedevere-app bot

colesbury Sep 28, 2023

FFY00 Oct 1, 2023

zooba Oct 2, 2023

FFY00 Oct 2, 2023

zooba Oct 2, 2023

FFY00 Oct 2, 2023

colesbury Oct 4, 2023

zooba Oct 4, 2023

colesbury Oct 4, 2023

zooba Oct 4, 2023

zooba Oct 4, 2023

colesbury Oct 4, 2023

zooba commented Oct 4, 2023

FFY00 Oct 4, 2023 •

edited

FFY00 left a comment

FFY00 Oct 4, 2023

FFY00 commented Oct 4, 2023

colesbury commented Oct 4, 2023

colesbury commented Oct 4, 2023

FFY00 commented Oct 4, 2023

FFY00 commented Oct 4, 2023

vstinner commented Oct 5, 2023

FFY00 commented Oct 5, 2023

zooba commented Oct 6, 2023

		# Add EXT_SUFFIX, SOABI, and Py_NOGIL
		vars.update(_sysconfig.config_vars())

gh-88402: Add new sysconfig variables on Windows (GH-110049) #110049

gh-88402: Add new sysconfig variables on Windows (GH-110049) #110049

Conversation

colesbury commented Sep 28, 2023 • edited by bedevere-app bot

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zooba commented Oct 4, 2023

FFY00 Oct 4, 2023 • edited

Choose a reason for hiding this comment

FFY00 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FFY00 commented Oct 4, 2023

colesbury commented Oct 4, 2023

colesbury commented Oct 4, 2023

FFY00 commented Oct 4, 2023

FFY00 commented Oct 4, 2023

vstinner commented Oct 5, 2023

FFY00 commented Oct 5, 2023

zooba commented Oct 6, 2023

colesbury commented Sep 28, 2023 •

edited by bedevere-app bot

FFY00 Oct 4, 2023 •

edited