Segfault in the psycopg python only variant. #300
Description
Hi there, I am trying to get pyscopg working on Django (https://github.com/django/django/pull/15687/files), thank you for your work so far on this -- I'll try to get it over the finish line.
One thing I did note though is that if I run the testsuite like this (this is Django's testsuite from my branch with psycopg3 installed):
./runtests.py --settings=test_postgresql --parallel=8
cat test_postgresql.py
# This is an example test settings file for use with the Django test suite.
#
# The 'sqlite3' backend requires only the ENGINE setting (an in-
# memory database will be used). All other backends will require a
# NAME and potentially authentication information. See the
# following section in the docs for more information:
#
# https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/
#
# The different databases that Django supports behave differently in certain
# situations, so it is recommended to run the test suite against as many
# database backends as possible. You may want to create a separate settings
# file for each of the backends you test against.
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "django",
},
"other": {
"ENGINE": "django.db.backends.postgresql",
"NAME": "django2",
},
}
SECRET_KEY = "django_tests_secret_key"
# Use a fast hasher to speed up tests.
PASSWORD_HASHERS = [
"django.contrib.auth.hashers.MD5PasswordHasher",
]
DEFAULT_AUTO_FIELD = "django.db.models.AutoField"
USE_TZ = False
I do get segfaults after a few tests:
./runtests.py --settings=test_postgresql --parallel=8
Testing against Django installed in '/home/florian/sources/django.git/django' with up to 8 processes
Found 15782 test(s).
Creating test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Cloning test database for alias 'default'...
Creating test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
Cloning test database for alias 'other'...
System check identified no issues (17 silenced).
....Fatal Python error: Segmentation fault
Current thread 0x00007ff582c5b740 (most recent call first):
File "/home/florian/sources/django.git/django/db/backends/base/base.py", line 364 in close
File "/home/florian/sources/django.git/django/utils/asyncio.py", line 26 in inner
File "/home/florian/sources/django.git/django/test/testcases.py", line 1461 in tearDownClass
File "/usr/lib64/python3.10/unittest/suite.py", line 306 in _tearDownPreviousClass
File "/usr/lib64/python3.10/unittest/suite.py", line 130 in run
File "/usr/lib64/python3.10/unittest/suite.py", line 84 in __call__
File "/home/florian/sources/django.git/django/test/runner.py", line 366 in run
File "/home/florian/sources/django.git/django/test/runner.py", line 446 in _run_subsuite
File "/usr/lib64/python3.10/multiprocessing/pool.py", line 125 in worker
File "/usr/lib64/python3.10/multiprocessing/process.py", line 108 in run
File "/usr/lib64/python3.10/multiprocessing/process.py", line 315 in _bootstrap
File "/usr/lib64/python3.10/multiprocessing/popen_fork.py", line 71 in _launch
File "/usr/lib64/python3.10/multiprocessing/popen_fork.py", line 19 in __init__
File "/usr/lib64/python3.10/multiprocessing/context.py", line 277 in _Popen
File "/usr/lib64/python3.10/multiprocessing/process.py", line 121 in start
File "/usr/lib64/python3.10/multiprocessing/pool.py", line 326 in _repopulate_pool_static
File "/usr/lib64/python3.10/multiprocessing/pool.py", line 303 in _repopulate_pool
File "/usr/lib64/python3.10/multiprocessing/pool.py", line 212 in __init__
File "/usr/lib64/python3.10/multiprocessing/context.py", line 119 in Pool
File "/home/florian/sources/django.git/django/test/runner.py", line 497 in run
File "/usr/lib64/python3.10/unittest/suite.py", line 84 in __call__
File "/usr/lib64/python3.10/unittest/runner.py", line 184 in run
File "/home/florian/sources/django.git/django/test/runner.py", line 967 in run_suite
File "/home/florian/sources/django.git/django/test/runner.py", line 1045 in run_tests
File "/home/florian/sources/django.git/tests/./runtests.py", line 427 in django_tests
File "/home/florian/sources/django.git/tests/./runtests.py", line 768 in <module>
Extension modules: pywatchman.bser, markupsafe._speedups, PIL._imaging, psycopg2._psycopg, PIL._webp, yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, _cffi_backend (total: 20)
Now I am not 100% sure yet if this is due to my code changes or not, but Python should not crash. Then again since psycopg uses ctypes there is probably a chance of crashes? The other option is that this is a python bug but I am opening it here first and maybe we can figure out if it is psycopg or python. I can also gather core dumps, but it would be great if someone can try verifying it on another machine so I am not chasing heisenbugs.
I also do not observe this issue if I use 'psycopg[binary]', which points even more towards a psycopg ctypes or python bug. I will try to gather more details tomorrow, but if you have any ideas let me know.