Gunicorn 20.0.0: find_library('c') doesn't work in Alpine Linux

[aljinovic]

I was trying to run gunicorn 20.0.0 inside an Alpine Linux docker container (python:3.8-alpine).
Gunicorn 19.9.0 works just fine on the same docker image.

After installing musl, the error libc not found is still present.

Here's a block of code extracted from gunicorn/socketfromfd.py that doesn't work in Alpine:

test.py

import ctypes
from ctypes.util import find_library

_libc_name = find_library('c')
if _libc_name is not None:
    libc = ctypes.CDLL(_libc_name, use_errno=True)
else:
    raise OSError('libc not found')

Dockerfile

FROM python:3.8-alpine

WORKDIR /app

RUN apk add musl

COPY . /app

CMD ["python", "test.py"]

[benoitc]

Right. We should probably test if python=< 3.7 to use socket.from_fd in

gunicorn/gunicorn/socketfromfd.py

Line 65 in 438371e

def fromfd(fd, keep_fd=True):

[lseelenbinder]

First, thanks for a great piece of software!
Second, I can confirm this issue; it has occurred in every installation of 20.0.0 on Alpine.

[naorlivne]

Right. We should probably test if python=< 3.7 to use socket.from_fd in

gunicorn/gunicorn/socketfromfd.py

Line 65 in 438371e

def fromfd(fd, keep_fd=True):

Can confirm that the issue isn't related to a specific Python version, have it happening in 3.6,3.7,3.8 so long as it's in Alpine.

[benoitc]

It's a side effect of using an operating system without all the basic libs in a container unfortunately. Maybe you could use an image using a debian? or an alpine image coming with a libc?

This api is there is pre-3.7 to replace a lack of API for it in Python. Detecting if the api is available (for python >= 3.7) would at least solve the issue for these versions. Before that we need to investigate what could be done.

[naorlivne]

It's a side effect of using an operating system without all the basic libs in a container unfortunately. Maybe you could use an image using a debian? or an alpine image coming with a libc?

This api is there is pre-3.7 to replace a lack of API for it in Python. Detecting if the api is available (for python >= 3.7) would at least solve the issue for these versions. Before that we need to investigate what could be done.

Alpine uses musl libc in place of glibc as part of it being based on busybox, of course using other base images would be a workaround but it will limit gunicorn uses as both alpine & other minimal OS that are growing in popularity (which are often based on it\busybox\etc) all have musl instead of glibc so the affect is likely not limited to alpine (just the most popular one affected).

Sounds to me that something like detecting if musl is installed instead of glibc & if so use whatever solution was in place in 19.9.x is a potential fix that isn't python version specific as we know 19.9.x works well.

[benoitc]

found a related issue in docker-library/python#111

maybe this code can be reused in a way: python/cpython@e3f6778

[MrNaif2018]

There is one possible workaround for that, as twisted users encountered the same:
https://stackoverflow.com/questions/48234723/twisted-server-monkey-patch-file
That piece of code from gunicorn/socketfromfd.py could be rewritten as following:

_libc_name = find_library('c')
if _libc_name is None:
    _libc_name = 'libc.so.6'
libc = ctypes.CDLL(_libc_name, use_errno=True)

Just one of the possible solutions,but probably not the best one.

[tilgovi]

maybe this code can be reused in a way: python/cpython@e3f6778

I think that's this issue: https://bugs.python.org/issue21622

Let's find a workaround and make a patch release.

[benoitc]

@tilgovi the patch I linked above does the trick (and is linked in the bug ticket you added). I will work on a PR that use it.

[benoitc]

@aljinovic can you try fda61b5 . It should fix your issue. Let me know :)

[aljinovic]

I tried it and it doesn't work. I found a bug in your implementation, here's a proposal of the fix:

for suffix in ['so', '*.so.*']:
    for f in glob('{0}.{1}'.format(prefix, suffix)):

[benoitc]

hrm odd, can you paste me the error you had?

[aljinovic]

Your code: find_library("c") returns None.
After my change it returns: libc.musl-x86_64.so.1

[aljinovic]

Also, I had to manually set LD_LIBRARY_PATH first, so maybe this could help:

paths = ['/lib', '/usr/local/lib', '/usr/lib']
if 'LD_LIBRARY_PATH' in os.environ:
    paths = os.environ['LD_LIBRARY_PATH'].split(':') + paths

Taken from:
https://github.com/alpinelinux/aports/blob/202f4bea916b0cf974b38ced96ab8fca0b192e3f/main/python2/musl-find_library.patch

[benoitc]

ok fixing that, not sure about why the glob is nlot working the first way though. I need to test

[benoitc]

@aljinovic should be OK now, let me know

[aljinovic]

Still doesn't work unless I change from:

for suffix in ['so', 'so.*']:

to:

for suffix in ['so', '*.so.*']:

The actual libc name is: libc.musl-x86_64.so.1

[benoitc]

sorry I missed the additional star. e150ca4 should fix it now.

…

On Thu, Nov 21, 2019 at 11:47 AM Ante Aljinovic ***@***.***> wrote: Still doesn't work unless I change from: for suffix in ['so', 'so.*']: to: for suffix in ['so', '*.so.*']: The actual libc name is: libc.musl-x86_64.so.1 — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub <#2160?email_source=notifications&email_token=AAADRISHZX5Q265IJQ5OCKLQUZRMJA5CNFSM4JLJUUP2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEZZJPY#issuecomment-557028543>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADRIXWUCJLU6ZCD2R2Z6DQUZRMJANCNFSM4JLJUUPQ> .

[aljinovic]

Great, that worked, thank you!

[tilgovi]

Nice work everyone!

[benoitc]

can you all test #2208 as i changed the implementation to simply use the python standard library. Any feedback is welcome :)