ctypes.util.find_library fails when ldconfig/glibc not available (e.g., AIX) #70626
Comments
|
I have successful enough with python 2.7.10 (for building cloud-init), including it finding openssl libraries during the installation od setuptools (before installing pip). I have also been able to assemble saltstack - BUT - salt-master and salt-minion fail to start because ctypes.util.find_library() always returns 'None'. in the util.py file the code reached is: +219 else: (I have not researched _get_soname or _findLib_gcc but neither of these "feel right" as AIX, by default, does not end library (archives) with .so (archives end with .a and archive members frequently end with .so) That this is/has not been reported more frequently may be because python programmers are avoiding ctypes - when portability is essential. I hope that, just as for Solaris - where an alternate program is used - that AIX can have a block: so that if ldconfig is not available the command /usr/bin/dump could be used instead, and/or search LIBPATH and/or (when not empty) and/or ldd (for programs). Ideally, /sbin/ldconfig would not be need at all! dump output: root@x064:[/data/prj/gnu/bashRC1-4.4]dump -Xany -H /opt/bin/python /opt/bin/python: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER root@x064:[/usr/bin]dump -Xany -H /usr/lib/libcrypto.a /usr/lib/libcrypto.a[libcrypto.so]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER /usr/lib/libcrypto.a[libcrypto.so.0.9.8]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER /usr/lib/libcrypto.a[libcrypto.so.1.0.0]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER /usr/lib/libcrypto.a[libcrypto64.so]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER /usr/lib/libcrypto.a[libcrypto64.so.0.9.8]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER /usr/lib/libcrypto.a[libcrypto64.so.1.0.0]: VERSION# #SYMtableENT #RELOCent LENidSTR #IMPfilID OFFidSTR LENstrTBL OFFstrTBL INDEX PATH BASE MEMBER ldd outpath: root@x064:[/usr/bin]ldd /usr/lib/libcrypto.a |
|
p.s. On a debian (on POWER) system, the function is working, but the test seems a bit broken as well, i.e., cdll.LoadLibrary("libm.so") is not working even though if os.name == "posix":
# find and load_version
print find_library("m")
print find_library("c")
print find_library("bz2")has successfully printed.
root@ipv4:/home/michael# python -m ctypes.util
libm.so.6
libc.so.6
libbz2.so.1.0
Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/usr/lib/python2.7/ctypes/util.py", line 287, in <module>
test()
File "/usr/lib/python2.7/ctypes/util.py", line 282, in test
print cdll.LoadLibrary("libm.so")
File "/usr/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libm.so: cannot open shared object file: No such file or directory
root@ipv4:/home/michael# uname -a
Linux ipv4.rootvg.net 3.16.0-4-powerpc64 #1 SMP Debian 3.16.7-ckt9-3 (2015-04-23) ppc64 GNU/Linux
root@ipv4:/home/michael# ldconfig -p | grep libm.so
libm.so.6 (libc6, OS ABI: Linux 2.6.32) => /lib/powerpc-linux-gnu/libm.so.6
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm.so.6
lrwxrwxrwx 1 root root 12 Apr 15 2015 /lib/powerpc-linux-gnu/libm.so.6 -> libm-2.19.so
root@ipv4:/home/michael# ls -l /lib/powerpc-linux-gnu/libm-2.19.so
-rw-r--r-- 1 root root 743784 Apr 15 2015 /lib/powerpc-linux-gnu/libm-2.19.so |
|
Further testing... I added an extremely simple hack in util.py so that a archive (AIX library) name got returned. Also in this case - the print cdll.LoadLibrary("libc.a") fails. +74 if os.name == "posix" and sys.platform == "darwin": +275 else: root@x064:[/tmp]python -m ctypes.util
libm.a
libc.a
libbz2.a
Traceback (most recent call last):
File "/opt/lib/python2.7/runpy.py", line 162, in _run_module_as_main
"__main__", fname, loader, pkg_name)
File "/opt/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/lib/python2.7/ctypes/util.py", line 282, in <module>
test()
File "/opt/lib/python2.7/ctypes/util.py", line 276, in test
print cdll.LoadLibrary("libc.a")
File "/opt/lib/python2.7/ctypes/__init__.py", line 443, in LoadLibrary
return self._dlltype(name)
File "/opt/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: 0509-022 Cannot load module /usr/lib/libc.a.
0509-103 The module has an invalid magic number.So, what needs to be returned so that cdll.LoadLibrary could use that result? (e.g., I know that the default libm.a has no shared members on AIX 5.3 TL7 - but libc.a does (shr.o and more, but not all!). |
|
Last message (back to debian, and minor changes to learn expected behavior) if sys.platform == "darwin":
print cdll.LoadLibrary("libm.dylib")
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
else:
print cdll.LoadLibrary("libm.so.6")
# print cdll.LoadLibrary("libcrypt.so")
print find_library("crypt")
x = find_library("crypt")
print cdll.LoadLibrary(x)returns: |
|
FYI: getting objdump is not too hard... root@x064:[/data/prj/gnu/binutils-2.25.1]v/null /usr/lib/libcrypto.a < libcrypto.so: file format aixcoff-rs6000 libcrypto.so.0.9.8: file format aixcoff-rs6000 libcrypto.so.1.0.0: file format aixcoff-rs6000 libcrypto64.so: file format aix5coff64-rs6000 libcrypto64.so.0.9.8: file format aix5coff64-rs6000 libcrypto64.so.1.0.0: file format aix5coff64-rs6000 But ldconfig (as a command within glibc) will be a real headache - and I would home unnecessary. In closing, I hope the AIX command /usr/bin/dump will be adequate as an alternative of objdump. Where I am still a bit lost is with the "open", i.e. _dlopen(). Suggestions/hints for debugging are welcome. |
aixtools
changed the title
|
The _dlopen call in __init__.py I have been able to fix (hack) with the following: root@x064:[/data/prj/aixtools/python/python-2.7.10/Lib/ctypes]diff -u __init__.py /opt/lib/python2.7/ctypes/__init__.py
--- __init__.py 2015-05-23 16:09:01 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py 2016-02-26 08:40:19 +0000
@@ -11,6 +11,7 @@
from _ctypes import _Pointer
from _ctypes import CFuncPtr as _CFuncPtr
from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
from _ctypes import ArgumentError
@@ -32,6 +33,11 @@
if int(_os.uname()[2].split('.')[0]) < 8:
DEFAULT_MODE = RTLD_GLOBAL
+if _os.name == "posix" and _sys.platform.startswith("aix"):
+ RTLD_NOW = 0x00000002
+ RTLD_MEMBER = 0x00040000
+ DEFAULT_MODE |= (RTLD_NOW | RTLD_MEMBER)
+
from _ctypes import FUNCFLAG_CDECL as _FUNCFLAG_CDECL, \
FUNCFLAG_PYTHONAPI as _FUNCFLAG_PYTHONAPI, \
FUNCFLAG_USE_ERRNO as _FUNCFLAG_USE_ERRNO, \I have an additional hack in util.py so that, e.g., returns: When that is passed to dlopen (plus RTLD_MEMBER) the dlopen succeeds. <CDLL '/usr/lib/libcrypto.a/(libcrypto.so)', handle 6 at 30146670> With some help in util.py, to do some sensible searching, being able to establish if in 32 or 64-bit mode, etc. somehing nice can be made here (imho). p.s. can any verify whether this is limited to python 2.7? Or should python 3.X be added as well? |
|
ctypes util.py "simply" needs support for AIX. There already is special support for Windows, Darwin, BSDs, Solaris. Is the question about the technical details for equivalent functionality on AIX or about adding a stanza to Lib/ctpyes/util.py? |
|
it was, partly, about the technical details - but I feel I have found the key bits - /full/path/libNAME.a(libNAME.so) + dlflags RTLD_NOW + RTLD_MEMBER for "native" AIX support. Additionally, a patch should not break what might be working for some (via google I read of suggestions for other languages, e.g., java, where .so members are extracted or installed side-by-side with the .a archive) - as the IBM run-rime loader also accepts/looks for both .a and .so. And/or for when /sbin/ldconfig is available. And when available, is this used as last case, or preferred? Lastly, we cannot assume we will know the name of the member based on the name of the library. In many cases, e.g., libiconv.a the IBM names are shr.o, shr4.o and shr4_64.o - and these are the only member names unless GNU libiconv.a has been added (and then libconv.a also contains libiconv.so.2 (so version support is also desired!)) in both 32 and 64 bit mode. One technical detail I have not been able to discover yet: how to determine whether in 32-bit or 64-bit mode. This will be important for libraries that have shr4.o and shr4_64.o as the members that need to be dlopened. Lastly: about adding the "stanza": python is a new language for me. Any assistance with a patch - to keep it properly 'pythonized' In short, I shall continue my studies/learning. Assistance from anyone wiser than me is welcome! |
|
AIX traditionally used member names like shr.o or shr<version>.o or shr<aix_release>.o insider the archive, with _64 designating a 64 bit object when there is a naming collision. GNU libtool defaults to the SO name and version number insider the archive. AIX objects (and shared objects) contain a bit in the header that specifies 32 bit or 64 bit. Both 32 bit and 64 bit objects are intended to be archived together. The linker only processes objects of the correct mode. AIX shared objects contain a bit that specifies if the object may be used at link-edit time or only should be used for loading. This is controlled by the AIX strip -e/-E option (yes, I know, strange place to hide that option). This combination of features allows all of the libraries to be placed in a single /usr/lib directory and all of the objects to be collected into a single archive, avoiding /usr/lib64 and explosion of shared objects and symbolic links clutter. Various packages have created /usr/local/lib64 anyway using Linux/Solaris/SVR4-style naming. |
|
The following demonstrates the basics. cdll.LoadLibrary will load a .so file if it can be located (you will have to believe me as I forgot to include the test in this last batch) Further, when given the AIX format for loading a member of an archive in ctypes/utils.py What is not done, and probably different from ldconfig behavior!
And many many thanks for the strip -e/-E info. Hard to find! root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/__init__.py /opt/lib/python2.7/ctypes/__init__.py
--- python-2.7.11/Lib/ctypes/__init__.py 2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/__init__.py 2016-02-29 17:36:43 +0000
@@ -11,6 +11,7 @@
from _ctypes import _Pointer
from _ctypes import CFuncPtr as _CFuncPtr
from _ctypes import __version__ as _ctypes_version
+# from _ctypes import RTLD_LOCAL, RTLD_GLOBAL, RTLD_NOW ## fails
from _ctypes import RTLD_LOCAL, RTLD_GLOBAL
from _ctypes import ArgumentError
@@ -356,6 +357,14 @@
if use_last_error:
flags |= _FUNCFLAG_USE_LASTERROR
+ if _sys.platform.startswith("aix"):
+ RTLD_NOW = 0x00000002
+ RTLD_MEMBER = 0x00040000
+ mode |= RTLD_NOW
+ if name is not None:
+ if name[-1] == ')':
+ mode |= RTLD_MEMBER
+
class _FuncPtr(_CFuncPtr):
_flags_ = flags
_restype_ = self._func_restype_
root@x064:[/data/prj/aixtools/src]diff -u python-2.7.11/Lib/ctypes/util.py /opt/lib/python2.7/ctypes/util.py
--- python-2.7.11/Lib/ctypes/util.py 2015-12-05 19:46:56 +0000
+++ /opt/lib/python2.7/ctypes/util.py 2016-02-29 17:42:41 +0000
@@ -84,6 +84,23 @@
continue
return None
+if os.name == "posix" and sys.platform.startswith("aix"):
+ def _findLib_aix(name):
+ paths='/usr/lib:/lib:/usr/lpp/xlC/lib'
+ for dir in paths.split(":"):
+ shlib = os.path.join(dir, "lib%s.so" % name)
+ if os.path.exists(shlib):
+ return shlib
+ libfile = os.path.join(dir, "lib%s.a" % name)
+ if os.path.exists(libfile):
+ shlib = "lib%s.a(lib%s.so)" % (name, name)
+ return shlib
+
+ return None
+
+ def find_library(name):
+ return _findLib_aix(name)
+
elif os.name == "posix":
# Andreas Degert's find functions, using gcc, /sbin/ldconfig, objdump
import re, tempfile, errno
@@ -267,7 +284,16 @@
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
+ elif sys.platform.startswith("aix"):
+ print find_library("crypto")
+ x = find_library("crypto")
+ print cdll.LoadLibrary(x)
+ print cdll.LoadLibrary("libcrypto.a(libcrypto.so)")
+ print cdll.LoadLibrary("libc.a(shr.o)")
+ print find_library("crypt")
+ print cdll.LoadLibrary("libcrypt.a(shr.o)")
else:
+ print cdll.LoadLibrary("libc.a")
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so")
print find_library("crypt")p.s. Anyone know what flag to set in vi so that tabs are never inserted and/or spaces converted to tabs. Python seems to hate tab indents. |
|
Oops: forgot the example output: root@x064:[/data/prj/aixtools/src]python -m ctypes.util |
|
The patch works when installed on top of pre-compiled version (e.g., copy the two files to /opt/lib/python2.7/ctypes/ but there seems to be a nasty side-effect when trying to build. make completes, but make install fails during a compile step. Please assist with howto see how/why ctypes/util is impacting this part. And - for my first adventure into python please provide the needed (expected) feedback. Michael make install DESTDIR=/var/aixtools/aixtools/python/2.7.11.2 > .buildaix/install.out Listing /var/aixtools/aixtools/python/2.7.11.2/opt/lib/python2.7/xml/sax ... So, zipfile.py seems to exit non-zero. :( |
|
Ah, good news - the build is successful ONCE I open the file ./Lib/ctypes/util.py and find the (hiding) tab characters and replace them with 8 space characters. (And I had tried so hard to check for that in advance). So, if you apply the patch - it may need some love. I would be grateful for people (even just one!) looking at it. Michael |
|
fyi: just completed a test both as 32 and 64 bit build. However, openssl-1.0.1.514 does not work in 64-bit mode (packaging error). Fortunately, openssl-1.0.1.515 (released 02-March-2016) fixes that. In short, 64-bit build is dependent on openssl-1.0.1.515 Here is the corrected patch (tabs removed). name: python.Lib.ctypes.160317.patch |
|
Maybe bpo-21826 is relevant (slowdown on AIX caused by missing ldconfig) |
|
These are very different issues. However, this patch may resolve both! ldconfig (-p if I recall) lists where (shared) libraries have been installed (imho, this is a GNU tool approach) - whereas AIX would use dump -H to find library paths embedded in a program and/or shared library. Until this patch, to use shared libraries on AIX the members of an archive needed to be extracted from the .a archive, and for 64-bit members, a separate directory (e.g. /usr/lib64) is needed. With this patch find_library() (actually cdll.LoadLibrary() can load members from either both .so and .a libraries, as is normal for AIX. So, in a way, this would also solve https://bugs.python.org/issue21826 as ldconfig is no longer needed (nor called) on AIX. p.s. As it is well longer than a month - I would appreciate that someone actually look at the patch and tell me how it can be improved! :) |
|
The most recent patch seems to follow AIX semantics correctly. |
|
I notice the patch is against Python 2, and uses Python 2 specific syntax. I am unsure if this kind of change is acceptable for 2.7, or if it would have to only go into 3.6. Hopefully this version of the patch will notify the review system that it is against Python 2. |
|
I don’t know anything about AIX specific stuff, but I left some general comments in the code review. Is there any chance that AIX people would be relying on the current behaviour that I understand uses _findSoname_ldconfig() and _findLib_gcc()? Is this new functionality covered by the test suite? E.g. in /Lib/ctypes/test/test_find.py, there are tests that call find_library() for GL, GLU, and gle. Are those libraries common on AIX? As discussed in bpo-9998, it seems a lot of people use find_library() to help convert a build-time library name to a run-time shared library name that can be passed to CDLL() or LoadLibrary(). E.g. on Linux: >>> find_library("python2.7") # As used in cc . . . -lpython2.7
'libpython2.7.so.1.0'
>>> cdll.LoadLibrary("libpython2.7.so.1.0")
<CDLL 'libpython2.7.so.1.0', handle 7f58e7495000 at 7f58e573ac90>Does your patch support this kind of use case? |
|
I have not looked specifically, at least not that I remember, for differences in util/ctypes in python2 and python3. Will do so tomorrow. I did just look briefly at the library, rather archive, built by default as libpython2.7.a - it is static members only, i.e., my build using xlc (i.e., not using gcc) does not build a shared object, so cdll.LoadLibrary and/or find_library will not find anything for python2.7.. Neither will m, or libm, on a default AIX system (with no other gcc based packages installed - these also install a gnu rte where the utilities and libs you mention might include. The few python packages I have found, packaged by others, tend to reload everything yet again, not depending on anything that may already be there. And to use shared libraries they are extracting the members from the .a archives into two directories - when they support both 32 and 64-bit targets. My intent is to examine the program to discover where libraries should be and find the member name that is most likely. Also, if LIBPATH is defined, those directories are searched first for a match. In short, the key difference is to look at the program (probably python) for the blibpath string in the application as well as python (from memory, sys.* calls) to build a list of directories to search. findLibrary('foo') first finds libfoo.a, then looks in libfoo.a for shr*.o members, libfoo.so, libfoo.so.X and/or libfoo.so.X.Y, etc.. I need to check that findLibrary('foo.so') continues to work. At one time it did, just have not looked at this for several weeks and I forget if it still works. That is what I shall make sure stays in the "testing" part of the patch. Michael |
|
The obvious (but easy to fix) problem I forsee with Python 3 is the print() calls. If you use print("") and print(arg), that will work with both 2 and 3. There may be more complications with bytes vs text stdout if we change the os.popen() calls to use subprocess.Popen. I didn’t mean to use libpython2.7 specifically. Substitute any shared library that is widely available across platforms; maybe “crypto” is a better example. CDLL(find_library("crypto")) loads "libcrypto.so.1.0.0" on my Linux computer. It looks like you got the equivalent working for AIX; I was just checking. FWIW it looks like your parsing of sys.executable to find library search paths is similar to searching the runpath (or RPATH) on ELF files, as proposed in bpo-19317. And it seems AIX’s LIBPATH environment variable is similar to LD_LIBRARY_PATH on Linux, proposed to be searched in bpo-9998. Also, I understand the equivalent OS X environment variables DYLD_(FALLBACK)_LIBRARY_PATH are already used. |
|
On 4/28/2016 11:56 PM, Michael Felt wrote:
I am reworking the logic - as many use cdll.LoadLibrary without ever More asap. |
|
Question - before I submit a patch. A. Is there a PEP I should read re: ctypes/util and/or ctypes/cdll? B. I show two different behaviors of responding - My question is, what does the community think should be the response? My preference is to bring the request back to it's stub - (strip lib from the beginning, and .so* or .a from the suffix, so that it is in it's -lFOO form - as if being offered to a linker (with -lFOO, we do not use -lFOO.so.6 (e.g., for libc.so.6). With my current working version - this is the output of the test in util.py: root@x064:[/data/prj/aixtools/python/python-2.7.11.3/Lib/ctypes]../../python ./util.py Additional Tests for AIX call cdll.LoadLibrary("foo") The test looks like: def test():
from ctypes import cdll
if os.name == "nt":
print cdll.msvcrt
print cdll.load("msvcrt")
print find_library("msvcrt")
if os.name == "posix":
# find and load_version
print find_library("m")
print find_library("c")
print find_library("bz2")## print cdll.m # load
if sys.platform == "darwin":
print cdll.LoadLibrary("libm.dylib")
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
else:
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so")
print find_library("crypt")
if sys.platform.startswith("aix"):
print "\nAdditional Tests for AIX"
print "call find_library(\"foo\")"
print "c: ", find_library("c")
print "c.a: ", find_library("c.a")
print "c.so: ", find_library("c.so")
print "libc: ", find_library("libc")
print "libc.a: ", find_library("libc.a")
print "libc.so.6: ", find_library("libc.so.6")
print "crypt: ", find_library("crypt")
print "crypto: ", find_library("crypto")
print "crypto.so: ", find_library("crypto.so")
print "libcrypto.so: ", find_library("libcrypto.so")
###
print "\ncall cdll.LoadLibrary(\"foo\")"
print "m: ", cdll.LoadLibrary("m")
print "libm.so: ", cdll.LoadLibrary("libm.so")
print "c: ", cdll.LoadLibrary("c")
print "c.a: ", cdll.LoadLibrary("c.a")
print "libc.a: ", cdll.LoadLibrary("libc.a")
print "libc.so.6: ", cdll.LoadLibrary("libc.so.6")
print "libc.so.9: ", cdll.LoadLibrary("libc.so.9")
print "bz2: ", cdll.LoadLibrary("bz2")
print "libbz2: ", cdll.LoadLibrary("libbz2")
print "crypt: ", cdll.LoadLibrary("crypt")
print "crypto: ", cdll.LoadLibrary("crypto")
print "crypto.so: ", cdll.LoadLibrary("crypto.so")
print "libcrypto.so: ", cdll.LoadLibrary("libcrypto.so")
print "iconv: ", cdll.LoadLibrary("iconv")
print "intl: ", cdll.LoadLibrary("intl")
if __name__ == "__main__":
test()Thank you for your considered comments. My goal is ease of use for (porting) existing projects to AIX, i.e., ideally without code change. |
|
reworked patch. To assist port to Python3 that changes in __init__.py and util.py are minimal. There is a new file: aixutil.py I have only tested on Python-2.7, so there may be issues for Python3. My goal is to have a single file for both versions. The main change in util.py is lots of specific (behavior) tests that I have encountered (and initially failed) from projects using ctypes.cdll. I feel confident that this will work "ASIS" with nearly all existing projects. Note: normal return is not a full path name. A full path name is only returned when an archive + member could not be found, but a file with the name requested could be found. This "fullpath" is to be compatible with existing code 'demanding' unpacked archives. |
|
implements ctypes.aixutil.find_library() |
|
''' The above don’t seem right to me, unless compiling with “cc -llibc.so.6” etc works on AIX. ''' These doesn’t look right. What happened to the library name? With your new aixutil.py file, it might be good to give it an underscore (_) prefix, to indicate it is an internal module rather than part of the ctypes API. So your code would do import ctypes._aixutil as aix |
|
Your new patch calls find_library() internally in CDLL(); why? My understanding is CDLL() is a fairly lightweight wrapper around the dlopen() call. On Linux, you either pass a full library file name, or an SO-name. Both these strings can be discovered for compiled objects using e.g.: $ ldd build/lib.linux-x86_64-2.7-pydebug/_ssl.so
linux-vdso.so.1 (0x00007fff567fe000)
libssl.so.1.0.0 => /usr/lib/libssl.so.1.0.0 (0x00007f598474c000)
libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f59842d4000)
. . .So in Python, the SO-name or full path can be used, but not the compile-time name, unless you first pass it through find_library(): >>> CDLL("libcrypto.so.1.0.0") # soname
<CDLL 'libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f16658f34d0>
>>> CDLL("/usr/lib/libcrypto.so.1.0.0") # Full path
<CDLL '/usr/lib/libcrypto.so.1.0.0', handle 7f1665e1eb90 at 7f1663cddcd0>
>>> CDLL("crypto") # Compile-time name
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/ctypes/__init__.py", line 365, in __init__
self._handle = _dlopen(self._name, mode)
OSError: crypto: cannot open shared object file: No such file or directory
>>> find_library("crypto") # Some people pass the result of this to CDLL()
'libcrypto.so.1.0.0' |
|
On 27-Aug-16 09:11, Martin Panter wrote:
Once I understood the boundaries of find_library("foo") I limited myself The linker is not looking for a particular member name: it only uses Basically, libtool has standardised how members are named, i.e., My goal for find_library() is to find the most likely name based on some Finally, we had different goals - my focus was on writing something that In closing: I am not trying to redesign find_library(). My hope is that most python
|
|
Regarding the regular expressions, I (or someone else unfamiliar with AIX) may be able to adjust them if you can explain what you are trying to achieve. Take the first one I commented on <https://bugs.python.org/review/26439/diff/17689/Lib/ctypes/_aixutil.py#newcode132\> for example. You added a comment: # '\[%s_*64\.so\]' % name, -> has either _64 or 64 added to name As written, this will match many strings including [<NAME>___64.so] However the annotation leads me to belive you want it to match two cases only: [<NAME>64.so] I do not know whether to fix the annotation (has 64 preceded by any number of underscores), or whether to fix the regular expression (_?64). |
|
On 04/09/2016 06:11, Martin Panter wrote:
The later - _?64. Working on this today. Thank you for the correction. |
|
Not always as elegant as I would wish (do not like the idea of "while 1:" and later a break... But, all in all, much improved by redoing the processing of the subprocess output and getting rid of more noise. Hope this meets your approval! p.s. This is a patch compared to previous patch. If a diff compared to Python-3.6.0a4 I can remake the patch. |
|
Sigh - missed the feature cutoff that would have made this easier to get into python... Anyway - have learned a few new things about python def: syntax and removed some bits that I thought were suitable for variable "initialization" - but tend to be static (not what I was thinking). Also, "new and improved" comments about what the code is doing. Thank you all for your patience and feedback - especially Martin! |
|
New changeset 01885f78b299 by Martin Panter in branch '2.7': New changeset 0db4403e62c4 by Martin Panter in branch '3.5': New changeset 4b7e51998a90 by Martin Panter in branch '3.6': New changeset f496fb6bf4a0 by Martin Panter in branch 'default': |
|
Hi Michael, I have done some cleanup and modifications to your patch. The result is in aix-library.161001.patch, which has all the changes, i.e. it is not based on another patch. More significant changes I made:
I did test it a bit on Linux with faked dump -H output, but I may have made mistakes that I did not pick up. Also, this still needs documentation, and I think some more tests for the test suite exercising various aspects of find_library() would be nice if possible. Another thing: in the last few patches, you dropped the definition of RTLD_MEMBER from Modules/_ctypes/_ctypes.c. Is that intended, or just a temporary thing? |
|
On 01-Oct-16 08:44, Martin Panter wrote:
But I shall also write up the AIX dlopen() process to explain how both
|
|
I have spent the last two hours trying to run the test - however, it fails with: root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
File "util.py", line 102, in <module>
import ctypes._aix as aix
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
from . import util
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'I have noticed several issues with the file that used to be named just but is now: ./build/lib.aix-5.3-3.6/sysconfigdata_m_aix5.py I am guessing something is wrong there - I am going to try copying only _aix.py to the Python2 branch and see if it works there -- and also dig deeper into what is going wrong with Python3.6* |
|
Curious. When in 32-bit mode changing line 15 of _aix.py to +14 import re, os, sys The Lib/ctypes/util.py works. In 64-bit mode it does not: root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
m :: None
c :: libc.a(shr_64.o)
Traceback (most recent call last):
File "util.py", line 355, in <module>
test()
File "util.py", line 330, in test
print("bz2\t:: %s" % find_library("bz2"))
File "util.py", line 104, in find_library
return aix.find_library(name)
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 255, in find_library
(base, member) = find_shared(libpaths, name)
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 247, in find_shared
member = get_member(re.escape(name), members)
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 189, in get_member
member = get_version(name, members)
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 170, in get_version
return util._last_version(versions, '.')
NameError: name 'util' is not defined+++++ both 32 and 64-bit report:
root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python util.py
Traceback (most recent call last):
File "util.py", line 102, in <module>
import ctypes._aix as aix
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/_aix.py", line 15, in <module>
from . import util
File "/data/prj/python/python-3.6.0.177/Lib/ctypes/util.py", line 102, in <module>
import ctypes._aix as aix
AttributeError: module 'ctypes' has no attribute '_aix'This last condition also occurs in Python2 |
|
Have a way to have both 64-bit and 32-bit modes working. root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python root@x064:[/data/prj/python/python-3.6.0.177/Lib/ctypes]../../python Will post patch asap. Going to shorten some lines going well over 76-char lime-limit. |
|
Besides correcting a small error, also my attempt to follow the This is a patch compered to Python-3.6b1 |
|
Just some minor comments on aix-library.161004.patch: Instead of _util.py, I wonder if the new file should have a different name, like _util_common.py, to avoid being too similar to util.py. +def get_shared(input): Needs a newline or new sentance to separate “output” and “the character”. +def get_legacy(members): Spelling: preferred [single F, double R] +def get_version(name, members): Should be: a "distinguishing" [not “an”] |
|
I would like to say: b) I still believe it is a "bug" plain and simple - it (find_library()) cannot work in "normal" situations. Why "IBM" or others never addressed this is beyond me - but - imho - that is a poor argument for not fixing it. FYI: As I have documented in ... this does not mean that ctypes.CDLL is incapable of loading a library. If you know the 'magic', aka what is not being provided by this "implementation dependent" module - it is possible to load either a .so file (actually a file by any name as long as it is a shared archive) - or an AIX-style .a archive with shared libraries stored internally as "archive members" - again, any member name is acceptable. The bug: The "default code" behind an "else:" block depends on the presence of the program "/sbin/ldconfig" to create a list of shared libraries. Normally, this program is not installed on an AIX system (certainly not production - maybe a system with gcc environment and using the gcc ld rather than the AIX ld (the latter being normal). Using python -m trace -C /tmp --count Lib/src/util.py The following (key) counts are reported. This is all that is being called - so when /sbin/ldonfig is not installed (which is usual) - so this code can only return 'None'. Environment: AIX 5.3, AIX 6.1, AIX 7.1 - all identical root@x064:[/]ls -l /sbin/ld* michael@x071:[/home/michael]ls /sbin/ld* root@x062:[/]ls /sbin/ld* So, obviously the sys.popen() call below will always 'fail'. The basic trace from util.py: And how this is called: And with python2-2.7.13 - the counts start the same - but the result is the same - by definition 'None' because subprocess.Popen() also ======== a) this is an "issue" aka bug, plain and simple - even it is has been ignored as such (other issues only complained about poorer performance because /sbin/ldconfig was not available). Please do not say - not fixing it because noone ever complained before. Otherwise, what is the point of ever accepting bug-reports aka issues if they can just be ignored. b) I want to thank Martin for his help on many (PEP-8 et al) improvements to my initial proposals for a patch. c) more important to me right now is that this be recognized as a bug - that should have been reported and resolved years ago. Personally, I do not care why noone ever reported it BUT I would like to see it properly identified that the default code is a specific implementation that is no way related to a normal AIX system - and an AIX-specific implementation is needed for normal operation of python2 and/or python3. Thank you for your time and thought! |
|
OOPS: I have a ... above, meant to be a link to a message. I also needed to 'patch' util.py - with 32-bit build with something like this:
--- src/python-2.7.13/Lib/ctypes/util.py 2016-12-17 20:05:05 +0000
+++ python-2.7.13.0/Lib/ctypes/util.py 2017-01-13 13:29:12 +0000
@@ -299,6 +299,10 @@
print cdll.LoadLibrary("libcrypto.dylib")
print cdll.LoadLibrary("libSystem.dylib")
print cdll.LoadLibrary("System.framework/System")
+ elif sys.platform[:3] == "aix":
+ from ctypes import CDLL
+ RTLD_MEMBER = 0x00040000
+ print CDLL("libc.a(shr.o)", RTLD_MEMBER)
else:
print cdll.LoadLibrary("libm.so")
print cdll.LoadLibrary("libcrypt.so")And 64-bit with: The respective output is: root@x064:[/data/prj/python/python-2.7.13.0]./python Lib/ctypes/util.py root@x064:[/data/prj/python/python-2.7.12.0]./python Lib/ctypes/util.py |
|
Michael Felt: if you still want the code compatible with Python 2 and 3 (and others are happy with that), I suggest documenting that in a code comment. |
|
Michael Felt: I merged your PR 🎉🍰✨ ! Thank you very much for your patience and perseverance. 80 comments on this bug, 117 comments and 19 commits in the PR... wow! AIX is not dead 😄 I leave the bug open since Martin, Mariatta and me were interested to make further minor coding style changes. |
|
Although I'm unable to double check for the moment, feels like the "SVR4" support still is incomplete: Remember that even the libNAME.so file may be an archive, or a symlink to an archive, with the real Shared Object as member having the F_LOADONLY flag set and usually named shr.o or shr_64.o, besides an Import File shr.imp or shr_64.imp, which actually is used at linktime and referring to the real Shared Object: This is necessary to emulate the "DT_SONAME" feature seen with ELF shared libraries. For reference, please have a look at the shared libraries created by recent libtool when --with-aix-soname=svr4 configure option is set. |
Sorry, I don't know what is SVR4. Is it a version of AIX? If it's rarely used, maybe a different issue should be opened to ask for an enhancement. |
|
SVR4 - stands for AT&T System V Release 4 - the beginning (as I understand it) of shared libraries as (indivudual) .so files in UNIX. SVR3 (and earlier) used .a files (aka archives with members inside). |
|
Within this context, the "svr4" label originates in the "-bsvr4" AIX linker flag, and actually is another (yet fully documented by the ld(1) man page) method for creating shared libraries on AIX to support filename based shared library versioning, which is known as the DT_SONAME tag with the ELF binary format. For details please refer to the GCC install doc: |
|
New changeset c0919c2 by Mariatta in branch 'master': |
|
will open a new PR for ./Doc/library/ctypes.rst |
|
imho - this should have status "closed" |
|
Thank you Michael Felt for your big "Implement find_library() support in ctypes/util for AIX" contribution, commit c5ae169! I close the issue. |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: