PyRun_String not exported in python38.dll

[cgohlke]

BPO	37189
Nosy	@pfmoore, @vstinner, @tjguk, @benjaminp, @zware, @eryksun, @zooba, @miss-islington
PRs	bpo-37189: Export old PyRun_XXX() functions #14142 [3.8] bpo-37189: Export old PyRun_XXX() functions (GH-14142) #14177

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-06-18.10:06:56.449>
created_at = <Date 2019-06-07.07:55:23.953>
labels = ['3.8', '3.9', 'OS-windows']
title = 'PyRun_String not exported in python38.dll'
updated_at = <Date 2019-06-18.10:06:56.440>
user = 'https://bugs.python.org/cgohlke'

bugs.python.org fields:

activity = <Date 2019-06-18.10:06:56.440>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-06-18.10:06:56.449>
closer = 'vstinner'
components = ['Windows']
creation = <Date 2019-06-07.07:55:23.953>
creator = 'cgohlke'
dependencies = []
files = []
hgrepos = []
issue_num = 37189
keywords = ['patch']
message_count = 18.0
messages = ['344905', '344982', '344989', '344993', '345056', '345792', '345809', '345871', '345873', '345915', '345919', '345926', '345928', '345929', '345930', '345931', '345934', '345975']
nosy_count = 10.0
nosy_names = ['paul.moore', 'vstinner', 'tim.golden', 'pyscripter', 'benjamin.peterson', 'cgohlke', 'zach.ware', 'eryksun', 'steve.dower', 'miss-islington']
pr_nums = ['14142', '14177']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37189'
versions = ['Python 3.8', 'Python 3.9']

[cgohlke]

While testing third party packages on Python 3.8.0b1 for Windows, I noticed that the PyRun_String function is no longer exported from python38.dll.

Is this intentional? I can't see this mentioned at <https://docs.python.org/3.8/whatsnew/3.8.html\> or <https://docs.python.org/3.8/c-api/veryhigh.html#c.PyRun_String\>

This change breaks existing code. But then PyRun_String is easy to replace with PyRun_StringFlags.

[zooba]

Guessing Victor may have touched something in this area.

Was it exported before? Or did we just have a macro for it? Maybe the macro was moved to an internal header by mistake?

[cgohlke]

PyRun_String was exported at least since python23.dll.

Python.Net relies on it at <https://github.com/pythonnet/pythonnet/blob/master/src/runtime/runtime.cs#L858\>

[zware]

A look through git log -p looks like bpo-34646 is likely to be related.

[benjaminp]

It shouldn't break existing code because PyRun_String has a macro expansion to PyRun_StringFlags. ABI compatibility between major releases is not provded.

[pyscripter]

This does break PyScripter Python for Delphi as well. The question whether this change was intentional in which case it would need to be explained and documented, or accidental and will be reversed begs an answer.

[vstinner]

Attached PR 14142 fix bpo-34646 regression.

It shouldn't break existing code because PyRun_String has a macro expansion to PyRun_StringFlags. ABI compatibility between major releases is not provided.

Many applications don't use Python header files, but access directly libpython. For example, py2app uses dlsym():
https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/apptemplate/src/main.c

PyInstaller uses GetProcAddress() on Windows or dlsym() on other platforms:

• https://github.com/pyinstaller/pyinstaller/blob/1844d69f5aa1d64d3feca912ed1698664a3faf3e/bootloader/src/pyi_python.h
• https://github.com/pyinstaller/pyinstaller/blob/1844d69f5aa1d64d3feca912ed1698664a3faf3e/bootloader/src/pyi_pythonlib.c

That's why PyRun_String() is defined as an alias using a macro *and* as a function in pythonrun.c:

#undef PyRun_String
PyObject *
PyRun_String(const char *str, int s, PyObject *g, PyObject *l)
{
    return PyRun_StringFlags(str, s, g, l, NULL);
}

[vstinner]

A look through git log -p looks like bpo-34646 is likely to be related.

New changeset e502451 by Benjamin Peterson in branch 'master':
closes bpo-34646: Remove PyAPI_* macros from declarations. (GH-9218)
e502451

Extract of this change:

 
 #undef PyRun_String
-PyAPI_FUNC(PyObject *)
+PyObject *
 PyRun_String(const char *str, int s, PyObject *g, PyObject *l)
 {
     return PyRun_StringFlags(str, s, g, l, NULL);
 }
 

On Windows, this change removed dllexport from PyRun_String(). My PR 14142 adds it back, to pythonrun.h.

[zooba]

I haven't fully tested this, but a suitable test using ctypes might look like:

py = ctypes.PyDLL("", handle=sys.dllhandle)
missing = {name for name in EXPECTED_NAMES if not hasattr(py, name)}
# assert 'missing' is empty

[vstinner]

I tested the following code:
---

import ctypes, sys 
names = """
PyRun_String
PyRun_AnyFile
PyRun_AnyFileEx
PyRun_AnyFileFlags
PyRun_SimpleString
PyRun_SimpleFile
PyRun_SimpleFileEx
PyRun_InteractiveOne
PyRun_InteractiveLoop
PyRun_File
PyRun_FileEx
PyRun_FileFlags
"""
api = ctypes.pythonapi
api2 = ctypes.PyDLL("", handle=sys.dllhandle)
for name in names.split():
    if not hasattr(api, name) or not hasattr(api2, name):
        print("MISSING NAME", name)

---

Current output:

Missing names ['PyRun_AnyFile', 'PyRun_AnyFileEx', 'PyRun_File', 'PyRun_FileEx',
'PyRun_FileFlags', 'PyRun_InteractiveLoop', 'PyRun_InteractiveOne', 'PyRun_Simp
leFile', 'PyRun_SimpleFileEx', 'PyRun_SimpleString', 'PyRun_String']

With my PR 14142:

Missing names []

[eryksun]

api = ctypes.pythonapi
api2 = ctypes.PyDLL("", handle=sys.dllhandle)

Those should be the same. In Windows, pythonapi is defined as PyDLL("python dll", None, sys.dllhandle).

Wouldn't it be better to add a function to _testcapi that checks GetProcAddress(PyWin_DLLhModule, name)?

[vstinner]

"Those should be the same."

Well, I wasn't 100% sure so I tested both. At least, I can now confirm that missing symbols are now exposed in both :-D

[zooba]

Ah, that's what ctypes.pythonapi is :) I looked at PyDLL first and figured it out from there.

Should we add a regression test to avoid this happening in the future?

[vstinner]

New changeset 343ed0f by Victor Stinner in branch 'master':
bpo-37189: Export old PyRun_XXX() functions (bpo-14142)
343ed0f

[vstinner]

Should we add a regression test to avoid this happening in the future?

I'm not sure where to add such test, nor which kind of test is needed. I mean, should we only test that the symbol is present? Or should we also test the ABI? Or even write a functional test?

Since I didn't know, I just merged my fix, to make sure that it lands before 3.8beta2.

[vstinner]

My notes on checking an ABI:
https://pythoncapi.readthedocs.io/stable_abi.html#check-for-abi-changes

I didn't check if https://abi-laboratory.pro/tracker/timeline/python/ supports Windows or not.

[miss-islington]

New changeset 8cb8d5d by Miss Islington (bot) in branch '3.8':
bpo-37189: Export old PyRun_XXX() functions (GH-14142)
8cb8d5d

[vstinner]

I close the issue. The initial issue has been fixed (PyRun_String is now exported again in the Python DLL).

Eryk Sun:

Wouldn't it be better to add a function to _testcapi that checks GetProcAddress(PyWin_DLLhModule, name)?

I have no idea.

Steve Dower:

Should we add a regression test to avoid this happening in the future?

As I explained, I'm not sure where to put such test, I'm not sure what should be tested. If someone wants to work on that, I suggest to open a separated issue.