ctypes.util.find_library("libc") fails

[doko42]

BPO	42580
Nosy	@doko42, @vstinner, @eryksun, @Redeyed

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-02-16.15:18:01.344>
created_at = <Date 2020-12-06.10:50:33.504>
labels = ['ctypes', 'invalid', '3.9']
title = 'ctypes.util.find_library("libc") fails'
updated_at = <Date 2021-02-16.15:29:24.694>
user = 'https://github.com/doko42'

bugs.python.org fields:

activity = <Date 2021-02-16.15:29:24.694>
actor = 'eryksun'
assignee = 'none'
closed = True
closed_date = <Date 2021-02-16.15:18:01.344>
closer = 'vstinner'
components = ['ctypes']
creation = <Date 2020-12-06.10:50:33.504>
creator = 'doko'
dependencies = []
files = []
hgrepos = []
issue_num = 42580
keywords = ['3.9regression']
message_count = 14.0
messages = ['382592', '382593', '387102', '387103', '387104', '387106', '387107', '387108', '387109', '387110', '387111', '387112', '387116', '387117']
nosy_count = 5.0
nosy_names = ['doko', 'vstinner', 'eryksun', 'Alex Shpilkin', 'RedEyed']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42580'
versions = ['Python 3.9']

[doko42]

regression compared to 3.8:

$ python3.9
Python 3.9.1rc1 (default, Nov 27 2020, 19:38:39) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> ctypes.util.find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NameError: name 'ctypes' is not defined
>>> import ctypes.util
>>> ctypes.util.find_library("libc")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.9/ctypes/util.py", line 341, in find_library
    _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))
  File "/usr/lib/python3.9/ctypes/util.py", line 147, in _findLib_gcc
    if not _is_elf(file):
  File "/usr/lib/python3.9/ctypes/util.py", line 99, in _is_elf
    with open(filename, 'br') as thefile:
FileNotFoundError: [Errno 2] No such file or directory: b'liblibc.a'

also note that the shared library is installed.

[doko42]

ctypes.util.find_library("c") works in both 3.8 and 3.9

[RedEyed]

can't reproduce

(py39) ➜ ~ ipython
Python 3.9.1 (default, Dec 11 2020, 14:32:07)
Type 'copyright', 'credits' or 'license' for more information
IPython 7.20.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ctypes.util
In [2]: a = ctypes.util.find_library("libc")
In [3]: print(a)
None

[vstinner]

This function is quite complicated on Linux:

        def find_library(name):
            # See issue python/cpython#54207
            return _findSoname_ldconfig(name) or \
                   _get_soname(_findLib_gcc(name)) or _get_soname(_findLib_ld(name))

_findSoname_ldconfig() uses "/sbin/ldconfig -p" and searchs for 'libc6,x86-64' in the output (on x86-64).

_findLib_gcc() uses "gcc -Wl,-t -o tmp -lc" command and search for "lib" in the created "tmp" file.

_findLib_ld() uses "ld -t -lc" command and searchs for "lib" pattern.

The exact code is more complicated :-) You should debug this issue by running these commands manually on Debian.

Note: python3.9 -c 'import ctypes.util; print(ctypes.util.find_library("c"))' commands displays libc.so.6 with Python 3.8, 3.9 and 3.10 on Fedora. I cannot reproduce the issue on Fedora 33 (x86-64).

[vstinner]

_findLib_gcc() uses "gcc -Wl,-t -o tmp -lc" command and search for "lib" in the created "tmp" file.

Oh, it looks for the "lib" in the process stdout (not in the created "tmp" file).

[RedEyed]

Note, that adding "lib" prefix to any library is wrong, that's why it returns "None" or raises the exception.

And it works fine when you call find_library("c")

I guess the issue, that was reported here, is about raising and exception instead of returning "None", but not about finding library, isn't it?

[vstinner]

Oh right, I always use find_library("c"), not find_library("libc").

[RedEyed]

So, can we close it?

[doko42]

ctypes.util.find_library("libc") used to work in 3.8, not working in 3.9. As I said before, ctypes.util.find_library("c") works in both 3.8 and 3.9.

[eryksun]

bpo-41976 added ctypes.util._is_elf() to filter out linker scripts such as "libc.so". The PR was backported to 3.7. _is_elf() assumes the trace result has absolute paths that can be opened, but Matthias is getting the relative filename "liblibc.a" in the result. Whatever the reason, I think if the file can't be opened for reading, then _is_elf() should just return False. For example:

    def _is_elf(filename):
        "Return True if the given file is an ELF file"
        elf_header = b'\x7fELF'
        try:
            with open(filename, 'br') as thefile:
                return thefile.read(4) == elf_header
        except OSError:
            return False

[RedEyed]

ctypes.util.find_library("libc") used to work in 3.8, not working in 3.9. As I said before, ctypes.util.find_library("c") works in both 3.8 and 3.9.

no, it doesn't work (and it shouldn't) neither in python 3.8 nor 3.7

Python 3.8.5 (default, Sep  4 2020, 07:30:14) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.19.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ctypes.util
In [2]: a = ctypes.util.find_library("libc")
In [3]: print(a)
None

Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.16.1 -- An enhanced Interactive Python. Type '?' for help.
Python 3.7.6 (default, Jan  8 2020, 19:59:22) 

    import ctypes.util
    a = ctypes.util.find_library("libc")
    print(a)
    None

as I said, adding prefix "lib" is wrong

[RedEyed]

I mean, find_library relies on gcc linker, when you pass library name to the linker, it automatically ads "lib" prefix to the library
so, when you pass "libc", linker ads "lib" so u have an error with "liblibc" name.

just man ld and see "-l" option:

"
-l namespec
--library=namespec
Add the archive or object file specified by namespec to the
list of files to link. This option may be used any number of
times. If namespec is of the form :filename, ld will search
the library path for a file called filename, otherwise it
will search the library path for a file called libnamespec.a.
"

as you can see, you pass not a library name, but a "namespec"
which then transforms to "libnamespec"

[vstinner]

The documentation is explicit: you must not include the "lib" prefix.
https://docs.python.org/dev/library/ctypes.html#ctypes.util.find_library

See also examples:
https://docs.python.org/dev/library/ctypes.html#finding-shared-libraries

If it worked with "lib" prefix in Python 3.8, it wasn't on purpose.

You should fix your code, but Python works as expected. I close the issue.

[eryksun]

Can't there be a library named "libc" with the shared module "liblibc.so"? It seems to me that the _is_elf() function added in bpo-41976 is breaking the contract that "[i]f no library can be found, returns None". It's not supposed to leak arbitrary exceptions from the implementation if the library isn't found.