-
-
Notifications
You must be signed in to change notification settings - Fork 31.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Incorrect shared library extension on linux #60958
Comments
I'm using Python3 as available in Fedora rawhide Attempting to build a project using python3/distutils, I noticed that Detailed analysis: setup.py: --- In function -> Where does shared_lib_extension get defined?
|
So, from what I can see, historically the SO extension was taken from sysconfig.py, see [1] lines 24 and 60. Then, the CCompiler class got overhauled, and the value was hardcoded to ".so", see [2], but the "compiler.shared_lib_extension = SO" statement remained, presumably oversight. So technically, this is also affecting python 2.x, though in that case it does not make any difference, since python2 does not use versioned so-names for binary modules. [1] http://hg.python.org/cpython/file/2802fb52e99b/Lib/distutils/unixccompiler.py |
This is issue introduced with implementation of SOABI. Build of standard extensions is protected by following code: class PyBuildExt(build_ext):
def __init__(self, dist):
build_ext.__init__(self, dist)
self.failed = []
def build_extensions(self):
# Detect which modules should be compiled
old_so = self.compiler.shared_lib_extension
# Workaround PEP 3149 stuff
self.compiler.shared_lib_extension = os.environ.get("SO", ".so")
try:
missing = self.detect_modules()
finally:
self.compiler.shared_lib_extension = old_so
.... I think that PEP-3149 is not accurate . For historical reasons (backward compatibility) SO must remain same as OS specific suffix and and new variable is required for python specific suffix. |
For reference, quoting PEP-3149: >>> sysconfig.get_config_var('SO')
'.cpython-32mu.so' |
Matthias, Barry, and I looked at this at pycon today. It looks a bit like the original intent was to have and then CPython extension module suffixes would be: if SOABI:
so_ext = ''.join(".", SOABI, SO)
else:
so_ext = SO This would need to be used in distutils/commands/build_ext.py We weren't sure if there are other places in the code that would need it as well but a quick build of a module which uses libraries and needs C extensions showed that this seems to work. The one worrisome question is whether more people have come to rely on the SO variable holding the extension module suffix or if more code was broken by the extension module suffix replacing the library suffix in the SO variable. Answering that might better show us whether to change these variables back to their original meanings or to create brand new variables that have the correct values. We also discovered the reason the current version appears to work with python-pillow on Ubuntu boxes but not Fedora. The find_library_files() code first checks for library names that would match with the "shared" library suffix. If that fails, it falls back to looking for the "static" library suffix. On Fedora, there are no static libraries so this function just fails to find the library. On Ubuntu, the code finds the static libraries and returns that. This causes the code in python-pillow to attempt to link to the library with -ljpeg, -lpng, etc... Since the shared libraries actually are present, the compiler and linker use the shared versions even though python only found the static versions. |
the patch so.diff
For trunk maybe consider removing the SO macro, I think it's better to change it back to it's original value for 3.1 and 3.2, knowing that it will probably break some Debian packaging, but that can be fixed too. |
Toshio and Matthias: This approach seems sane to me, Nick asked me to review this ticket. I'm not coming up with any objections. +1 for retiring SO at some point after 3.2, and EXT_SUFFIX and SHLIB_SUFFIX. What documentation needs to be changed? PEP-3149 (maybe just reference this being obsolete)? What other documentation? |
proposed PEP change. Or should the change itself be documented in the PEP? |
New changeset 8f91014c8f00 by doko in branch '3.2':
New changeset 14f0c9e595a5 by doko in branch '3.3':
New changeset f6a6b4eed5b0 by doko in branch 'default':
|
fixed, pep update is pending |
So sysconfig.get_config_var('SO') will change in a micro release? Won't this break working user code? Give unexpected file names? |
I just tried building trunk on MacOS 10.6.8, and make install fails with this traceback: Traceback (most recent call last):
File "./setup.py", line 2175, in <module>
main()
File "./setup.py", line 2170, in main
"Tools/scripts/2to3", "Tools/scripts/pyvenv"]
File "/Users/akuchling/source/p/cpython/Lib/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 917, in run_commands
self.run_command(cmd)
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
cmd_obj.run()
File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install.py", line 566, in run
self.run_command(cmd_name)
File "/Users/akuchling/source/p/cpython/Lib/distutils/cmd.py", line 313, in run_command
self.distribution.run_command(command)
File "/Users/akuchling/source/p/cpython/Lib/distutils/dist.py", line 936, in run_command
cmd_obj.run()
File "/Users/akuchling/source/p/cpython/Lib/distutils/command/install_lib.py", line 93, in run
outfiles = self.install()
File "./setup.py", line 2068, in install
self.set_file_modes(outfiles, 0o644, 0o755)
File "./setup.py", line 2079, in set_file_modes
if filename.endswith(self.shlib_suffix): mode = sharedLibMode
TypeError: endswith first arg must be str or a tuple of str, not NoneType
make: *** [sharedinstall] Error 1 The following patch fixes the traceback, but I don't know why shlib_suffix is None. -> hg diff
diff -r 9328e2b8a397 setup.py
--- a/setup.py Tue Apr 02 22:13:49 2013 +0200
+++ b/setup.py Tue Apr 02 19:10:27 2013 -0400
@@ -2076,7 +2076,9 @@
for filename in files:
if os.path.islink(filename): continue
mode = defaultMode
- if filename.endswith(self.shlib_suffix): mode = sharedLibMode
+ if (self.shlib_suffix is not None and
+ filename.endswith(self.shlib_suffix)):
+ mode = sharedLibMode
log.info("changing mode of %s to %o", filename, mode)
if not self.dry_run: os.chmod(filename, mode) |
Did you start with a clean build directory and rerun ./configure? What ./configure options did you use? Make sure that an existing install isn't getting in the way during a build, particularly with --enable-shared. With a current tip of the default branch, I've built all three standard OS X configurations (default unix (non-shared), --enable-shared, and --enable-framework) on OS X 10.8 and "make install" succeeds for all of them with sysconfig.get_config_var("SHLIB_SUFFIX") returning ".so" as expecting. |
is SHLIB_SUFFIX=".so" really correct on mac? |
are you speculating, or is your comment based on some testing? MacOS had this value before, and apparently it did work before that change. |
I'm going by what says in configure: The extension of shared libraries on macos is .dylib in most cases (e.g libtool based libraries and as mentioned python itself) Maybe its just a documentation/naming issue. |
just to clarify its not any issue in python, python is working fine with .so |
This change may be responsible for bpo-17869 |
False alarm, disregard my previous message. |
Amaury’s questions are still unanswered:
|
There were already 5 releases of Python 3.3 (starting with 3.3.1) with this change included. |
New changeset ccb679e5ae0e by Éric Araujo in branch 'default': |
distutils.sysconfig.get_config_var('SO') was deprecated in python 3.4, and is unset in python 3.11.0. Causes PYTHON3_SO=None -> _RNA.so builds as "_RNANone" -> build failure: ``` ImportError: cannot import name '_RNA' from partially initialized module 'RNA' ``` See: python/cpython#60958, python/cpython#63754 and [deprecation notice here](https://github.com/python/cpython/blob/dd53b79de0ea98af6a11481217a961daef4e9774/Doc/whatsnew/3.4.rst#deprecations-in-the-python-api). Further, module [distutils](https://docs.python.org/3/library/distutils.html) is deprecated, and has planned removal in 3.12. Functions get_python_inc, get_python_lib are removed, see: python/cpython#92584.
distutils.sysconfig.get_config_var('SO') was deprecated in python 3.4, and is unset in python 3.11.0. Causes PYTHON3_SO=None -> _RNA.so builds as "_RNANone" -> build failure: ``` ImportError: cannot import name '_RNA' from partially initialized module 'RNA' ``` See: python/cpython#60958, python/cpython#63754 and [deprecation notice here](https://github.com/python/cpython/blob/dd53b79de0ea98af6a11481217a961daef4e9774/Doc/whatsnew/3.4.rst#deprecations-in-the-python-api). Also, module [distutils](https://docs.python.org/3/library/distutils.html) is deprecated and has planned removal in 3.12. Functions get_python_inc, get_python_lib are removed, see: python/cpython#92584.
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: