segfault with gunicorn and a library made with cython bindings

[amirouche]

BPO	42891
Nosy	@vstinner, @amirouche

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-11.18:07:31.672>
created_at = <Date 2021-01-11.14:48:27.268>
labels = ['3.8', 'type-crash']
title = 'segfault with gunicorn and a library made with cython bindings'
updated_at = <Date 2021-01-11.18:07:31.672>
user = 'https://github.com/amirouche'

bugs.python.org fields:

activity = <Date 2021-01-11.18:07:31.672>
actor = 'amirouche'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-11.18:07:31.672>
closer = 'amirouche'
components = []
creation = <Date 2021-01-11.14:48:27.268>
creator = 'amirouche'
dependencies = []
files = []
hgrepos = []
issue_num = 42891
keywords = []
message_count = 7.0
messages = ['384836', '384838', '384839', '384840', '384843', '384845', '384852']
nosy_count = 2.0
nosy_names = ['vstinner', 'amirouche']
pr_nums = []
priority = 'normal'
resolution = 'third party'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'crash'
url = 'https://bugs.python.org/issue42891'
versions = ['Python 3.8']

[amirouche]

Here is a simple way to reproduce:

from lsm import LSM


db = LSM('db.sqlite')

def app(environ, start_response):
    """Simplest possible application object"""

    for (index, (key, value)) in enumerate(db[b'\x00':b'\xFF']):
        pass

    start_response(b'200', {})
    return b''

db.close()

In my real program, if I add 'global db' in the function app, it does not segfault.

program: https://git.sr.ht/~amirouche/segfault
trace: https://wyz.fr/0I-MO

[vstinner]

I put the code in a "bla.p" script and I ran it with Python 3.8 on Fedora 33:

python3.8 -m venv env
env/bin/python -m pip install lsm-db
env/bin/python bla.py

It doesn't crash. Please provide a script reproducing the issue.

Note: lsm-db cannot be installed in Python 3.9 (errors about the removed PyTypeObject.tp_print member in C).

[amirouche]

You need to run the program with the following:

python -X dev -c "from gunicorn.app.wsgiapp import run; run()" --workers=1 foobar:app

where foobar.py is the code from the previous message.

The crash happen when, the function app is executed, hence you need to call in another console:

curl http://localhost:8000

note: lsm package can be installed with pip install lsm-db
note2: lsm db can not be installed with py3.9

[vstinner]

I get a crash without any code, just by running gunicorn. It looks like a bug in gunicorn.

---

$ python3.8 -m venv env
$ env/bin/python -m pip install gunicorn
(..)
Successfully installed gunicorn-20.0.4

$ ./env/bin/python -X dev -c 'from gunicorn.app.wsgiapp import run; run()' foobar:app 
...)

[2021-01-11 16:14:11 +0100] [32107] [ERROR] Exception in worker process
Traceback (most recent call last):
  (...)
  File "/home/vstinner/z/env/lib64/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/home/vstinner/z/env/lib64/python3.8/site-packages/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  (...)
ModuleNotFoundError: No module named 'foobar'

[2021-01-11 16:14:11 +0100] [32107] [INFO] Worker exiting (pid: 32107)
sys:1: ResourceWarning: unclosed <socket.socket fd=5, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 8000)>
ResourceWarning: Enable tracemalloc to get the object allocation traceback
Debug memory block at address p=0x55cfa92154a0: API '1'
3271705103877500672 bytes originally requested
The 7 pad bytes at p-7 are not all FORBIDDENBYTE (0xfd):
at p-7: 0x00 *** OUCH
at p-6: 0x00 *** OUCH
at p-5: 0x00 *** OUCH
at p-4: 0x00 *** OUCH
at p-3: 0x00 *** OUCH
at p-2: 0x00 *** OUCH
at p-1: 0x00 *** OUCH
Because memory is corrupted at the start, the count of bytes requested
may be bogus, and checking the trailing pad bytes may segfault.
The 8 pad bytes at tail=0x2d67c444d794c3a0 are [2021-01-11 16:14:11 +0100] [32113] [INFO] Booting worker with pid: 32113

(...)
---

You should report the issue to gunicorn: https://github.com/benoitc/gunicorn/issues

[amirouche]

ModuleNotFoundError: No module named 'foobar'

That is not a segfault. The problem I am reporting is a segfault.

It can be reproduced with uvicorn as follow:

from lsm import LSM


db = LSM('db.sqlite')


async def app(scope, receive, send):
    assert scope['type'] == 'http'
    global db
    for (index, (key, value)) in enumerate(db[b'\x00':b'\xFF']):
        pass

await send({
    'type': 'http.response.start',
    'status': 200,
    'headers': [
        [b'content-type', b'text/plain'],
    ],
})
await send({
    'type': 'http.response.body',
    'body': b'Hello, world!',
})

db.close()

Run the above program with: python -X dev -m uvicorn bobo:app

Then in another console: curl http://localhost:8000/

Here is the output:

$ python -X dev -m uvicorn bobo:app
INFO:     Started server process [3580316]
INFO:     Waiting for application startup.
INFO:     ASGI 'lifespan' protocol appears unsupported.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)
Fatal Python error: Segmentation fault

Current thread 0x00007fbb6e49b740 (most recent call first):
File "./bobo.py", line 10 in app
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45 in __call__
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 394 in run_asgi
File "/usr/lib/python3.8/asyncio/events.py", line 81 in _run
File "/usr/lib/python3.8/asyncio/base_events.py", line 1851 in _run_once
File "/usr/lib/python3.8/asyncio/base_events.py", line 570 in run_forever
File "/usr/lib/python3.8/asyncio/base_events.py", line 603 in run_until_complete
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/server.py", line 48 in run
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 386 in run
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 362 in main
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 610 in invoke
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 1066 in invoke
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 782 in main
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/click/core.py", line 829 in __call__
File "/home/amirouche/.cache/pypoetry/virtualenvs/lsmdb-cTe2806J-py3.8/lib/python3.8/site-packages/uvicorn/main.py", line 4 in <module>
File "/usr/lib/python3.8/runpy.py", line 87 in _run_code
File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
Segmentation fault (core dumped)

[amirouche]

I tried to reduce the program by replacing gunicorn / uvicorn dependency with threading.Thread and multiprocess.Pool without success.

[amirouche]

The problem is prolly in lsm-db.

ref: coleifer/python-lsm-db#20

Sorry for the noise.