New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Running many tests with sqlite in-memory backend fails with Segmentation fault #409

Closed
asfaltboy opened this Issue Oct 26, 2016 · 14 comments

Comments

Projects
None yet
6 participants
@asfaltboy
Contributor

asfaltboy commented Oct 26, 2016

This happen to me a few times since I moved a large test base, of around 450 tests, to py.test (from nose), changing the django backend to either named sqlite db (test.db) or another backend does not seg fail. The failure seems to occur around the 390 test mark; even if I remove the specific tests the fault happened on.

I tried this with py.test 2.9 and latest 3.0.1 or 3.0.3.

tests.py ......
other_tests.py ...... (many more tests, ~390)

Fatal Python error: Segmentation fault

Current thread 0x00007fff73c0e000 (most recent call first):
  File "venv/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 316 in execute
  File "venv/lib/python2.7/site-packages/django/db/backends/utils.py", line 62 in execute
  File "venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 214 in _savepoint_rollback
  File "venv/lib/python2.7/site-packages/django/db/backends/base/base.py", line 254 in savepoint_rollback
  File "venv/lib/python2.7/site-packages/django/db/transaction.py", line 245 in __exit__
  File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 941 in _rollback_atomics
  File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 998 in _fixture_teardown
  File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 853 in _post_teardown
  File "venv/lib/python2.7/site-packages/django/test/testcases.py", line 189 in __call__
  File "venv/lib/python2.7/site-packages/_pytest/unittest.py", line 153 in runtest
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 104 in pytest_runtest_call
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 151 in <lambda>
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 163 in __init__
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 133 in call_and_report
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 79 in runtestprotocol
  File "venv/lib/python2.7/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 264 in __init__
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 247 in _wrapped_call
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 595 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
  File "venv/lib/python2.7/site-packages/_pytest/main.py", line 152 in pytest_runtestloop
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
  File "venv/lib/python2.7/site-packages/_pytest/main.py", line 131 in _main
  File "venv/lib/python2.7/site-packages/_pytest/main.py", line 96 in wrap_session
  File "venv/lib/python2.7/site-packages/_pytest/main.py", line 125 in pytest_cmdline_main
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 596 in execute
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 333 in <lambda>
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 338 in _hookexec
  File "venv/lib/python2.7/site-packages/_pytest/vendored_packages/pluggy.py", line 724 in __call__
  File "venv/lib/python2.7/site-packages/_pytest/config.py", line 57 in main
  File "venv/bin/py.test", line 11 in <module>
/bin/sh: line 1: 23353 Segmentation fault: 11  py.test
@jmillxyz

This comment has been minimized.

jmillxyz commented Nov 2, 2016

I'm getting a similar error, althought I just started a project and only have one (very) basic test, and am using the spatialite plugin for SQLite. Interestingly, my DATABASES setting specifies 'NAME': ':memory:' but both a test_:memory: and test_:memory:-journal file are created in my root directory after running pytest. @asfaltboy do you experience the same behavior?

$ pytest
==================================================================== test session starts ====================================================================
platform darwin -- Python 3.5.2, pytest-3.0.3, py-1.4.31, pluggy-0.4.0
Django settings: config.settings.test (from ini file)
rootdir: [...], inifile: pytest.ini
plugins: cov-2.4.0, django-3.0.0, faulthandler-1.3.0
collected 1 items

apps/firms/tests/test_models.py Fatal Python error: Segmentation fault

Current thread 0x00007fffce84c3c0 (most recent call first):
  File ".../project/lib/python3.5/site-packages/django/db/backends/sqlite3/base.py", line 335 in execute
  File ".../project/lib/python3.5/site-packages/django/db/backends/utils.py", line 62 in execute
  File ".../project/lib/python3.5/site-packages/django/contrib/gis/db/backends/spatialite/base.py", line 75 in prepare_database
  File ".../project/lib/python3.5/site-packages/django/core/management/commands/migrate.py", line 81 in handle
  File ".../project/lib/python3.5/site-packages/django/core/management/base.py", line 345 in execute
  File ".../project/lib/python3.5/site-packages/django/core/management/__init__.py", line 130 in call_command
  File ".../project/lib/python3.5/site-packages/django/db/backends/base/creation.py", line 70 in create_test_db
  File ".../project/lib/python3.5/site-packages/django/test/runner.py", line 743 in setup_databases
  File ".../project/lib/python3.5/site-packages/pytest_django/fixtures.py", line 96 in django_db_setup
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 720 in call_fixture_func
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 815 in pytest_fixture_setup
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 785 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 766 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 439 in getfixturevalue
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 447 in getfuncargvalue
  File ".../project/lib/python3.5/site-packages/pytest_django/plugin.py", line 375 in _django_db_marker
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 720 in call_fixture_func
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 815 in pytest_fixture_setup
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 785 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 529 in _getfixturevalue
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 464 in _get_active_fixturedef
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 439 in getfixturevalue
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 397 in _fillfixtures
  File ".../project/lib/python3.5/site-packages/_pytest/fixtures.py", line 252 in fillfixtures
  File ".../project/lib/python3.5/site-packages/_pytest/python.py", line 1560 in setup
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 453 in prepare
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 100 in pytest_runtest_setup
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 151 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 163 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 151 in call_runtest_hook
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 133 in call_and_report
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 73 in runtestprotocol
  File ".../project/lib/python3.5/site-packages/_pytest/runner.py", line 66 in pytest_runtest_protocol
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 152 in pytest_runtestloop
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 265 in __init__
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 248 in _wrapped_call
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 613 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 131 in _main
  File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 96 in wrap_session
  File ".../project/lib/python3.5/site-packages/_pytest/main.py", line 125 in pytest_cmdline_main
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 614 in execute
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 334 in <lambda>
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 339 in _hookexec
  File ".../project/lib/python3.5/site-packages/_pytest/vendored_packages/pluggy.py", line 745 in __call__
  File ".../project/lib/python3.5/site-packages/_pytest/config.py", line 56 in main
  File ".../project/bin/pytest", line 11 in <module>
[1]    34950 segmentation fault  pytest
@jmillxyz

This comment has been minimized.

jmillxyz commented Nov 2, 2016

Update: my tests now pass after installing pytest-ipdb, I'm on v0.1dev2:

pip install git+git://github.com/mverteuil/pytest-ipdb.git

It installs a fair amount of dependencies -- one of which may have fixed the problem -- but I don't have the time at the moment to dig into which one specifically. Here's the list:

Successfully installed appnope-0.1.0 decorator-4.0.10 ipdb-0.10.1 ipython-5.1.0
ipython-genutils-0.1.0 pexpect-4.2.1 pickleshare-0.7.4 prompt-toolkit-1.0.8 
ptyprocess-0.5.1 pygments-2.1.3 pytest-ipdb-0.1.dev2 simplegeneric-0.8.1 
traitlets-4.3.1 wcwidth-0.1.7
@jamescooke

This comment has been minimized.

jamescooke commented Dec 6, 2016

Hopefully related: I have a test suite running on sqlite - it passes during normal operation. However if I add --pdb then I get the segmentation fault. Could it be something to do with how the stack traces are stored which pytest-ipdb fixed?

@jmillxyz

This comment has been minimized.

jmillxyz commented Dec 6, 2016

Not sure if it's related, but after further testing I only experienced segfaults or bus errors with SQLite > 3.13.0

@jamescooke

This comment has been minimized.

jamescooke commented Dec 6, 2016

I'm on an old sqlite 2.8.17

@blueyed

This comment has been minimized.

Contributor

blueyed commented Dec 6, 2016

If you are using systemd check out coredumpctl gdb (and then bt when inside gdb) to see if there is some more information where it crashes.

@nMustaki

This comment has been minimized.

nMustaki commented Jan 26, 2017

There's a bug in sqlite 3.13 or so which causes segmentation faults, and sadly it's the default version in ubuntu. You should try to update sqlite and see if it fixes your issue :

wget https://github.com/ghaering/pysqlite/archive/2.8.3.tar.gz \
    && wget https://www.sqlite.org/2016/sqlite-autoconf-3150200.tar.gz \
    && tar -xzf sqlite-autoconf-3150200.tar.gz \
    && tar -xzvf 2.8.3.tar.gz \
    && cd sqlite-autoconf-3150200 \
    && ./configure \
    && make \
    && make install \
    && cd .. \
    && cp /usr/local/lib/libsqlite3.so.0.8.6 /usr/lib/x86_64-linux-gnu/libsqlite3.so.0.8.6 \
    && cp -av sqlite-autoconf-3150200/. pysqlite-2.8.3/ \
    && cd ./pysqlite-2.8.3 && python setup.py build_static install \
    && /opt/venv/bin/python setup.py build_static install
@jmillxyz

This comment has been minimized.

jmillxyz commented Jan 27, 2017

@nMustaki Do you have a link/issue for that bug? I'm curious

@nMustaki

This comment has been minimized.

nMustaki commented Jan 27, 2017

@blueyed

This comment has been minimized.

Contributor

blueyed commented Jan 27, 2017

Nobody provides a stacktrace?! 🤷‍♂️
Is it still present in the current release?

@blueyed

This comment has been minimized.

Contributor

blueyed commented Jan 27, 2017

Not really a pytest-django bug though - closing.

Feel free to provide a PR if you have some workaround, and it's relevant though.

@atodorov

This comment has been minimized.

atodorov commented Dec 29, 2017

Just FYI I'm seeing a very similar backtrace with Django 2.0 and sqlite 3.8.2, Python 3.6.3 on Ubuntu (Travis CI) and not on my devel workstation which is RHEL 7 with an older sqlite3. From what I can tell the segafult happens on an UPDATE query.

I've also dropped pytest and running my tests via manage.py test still reproduces the issue. Any hints would be welcome. Thanks.

@asfaltboy

This comment has been minimized.

Contributor

asfaltboy commented Dec 29, 2017

@atodorov you may want to take a look at Django ticket #24080 which seems to indicate this is likely an issue with SQLite rather than with Django.

One commenter points to related ticket on SQLite #7f7f8026eda38 which was fixed in version 3.12.1. I believe I eventually resolved my original issue by upgrading to latest SQLite as well, so I suggest you try that as well.

@atodorov

This comment has been minimized.

atodorov commented Dec 30, 2017

thanks for the links. I've seen the Django ticket and indeed an upgrade of sqlite solved the issue. However I don't think I've been hitting the same sqlite issue as you linked. Anyway update solves it.

ptrstn added a commit to ptrstn/TkDQMDoctor that referenced this issue Aug 15, 2018

Do not use in-memory sqlite database for testing
In-memory sqlite seems to cause segmentation faults as noted here:
pytest-dev/pytest-django#409

Travis-CI gives following error message:
Fatal Python error: Segmentation fault
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment