New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable native upsert on sqlite #4471

Merged
merged 3 commits into from Jan 25, 2019

Conversation

Projects
None yet
2 participants
@erikjohnston
Copy link
Member

erikjohnston commented Jan 25, 2019

As empirically they break everything.

Before we re-enable them we need to have a testing strategy here. We can get away with not testing postgres 9.4 on sytest as we are planning on dropping support for it imminently, but I don't think the same is true for older SQLite versions.

@erikjohnston erikjohnston requested a review from matrix-org/synapse-core Jan 25, 2019

@erikjohnston

This comment has been minimized.

Copy link
Member Author

erikjohnston commented Jan 25, 2019

The unit tests that fail for me:

tests.test_visibility.FilterEventsForServerTestCase.test_large_room
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_client_ips.py", line 181, in test_adding_monthly_active_user_when_space
    self.assertTrue(active)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 395, in assertTrue
    super(_Assertions, self).assertTrue(condition, msg)
  File "/usr/lib/python3.7/unittest/case.py", line 692, in assertTrue
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: None is not true

tests.storage.test_client_ips.ClientIpStoreTestCase.test_adding_monthly_active_user_when_space
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_client_ips.py", line 203, in test_updating_monthly_active_user_when_space
    self.assertTrue(active)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 395, in assertTrue
    super(_Assertions, self).assertTrue(condition, msg)
  File "/usr/lib/python3.7/unittest/case.py", line 692, in assertTrue
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: None is not true

tests.storage.test_client_ips.ClientIpStoreTestCase.test_updating_monthly_active_user_when_space
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_monthly_active_users.py", line 118, in test_can_insert_and_count_mau
    self.assertEqual(1, self.get_success(count))
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/lib/python3.7/unittest/case.py", line 839, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.7/unittest/case.py", line 832, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 1 != 0

tests.storage.test_monthly_active_users.MonthlyActiveUsersTestCase.test_can_insert_and_count_mau
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_monthly_active_users.py", line 102, in test_initialise_reserved_users
    self.assertEqual(self.get_success(active_count), user_num + ru_count)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/lib/python3.7/unittest/case.py", line 839, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.7/unittest/case.py", line 832, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 2 != 4

tests.storage.test_monthly_active_users.MonthlyActiveUsersTestCase.test_initialise_reserved_users
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_monthly_active_users.py", line 267, in test_track_monthly_users_without_cap
    self.assertEqual(2, self.get_success(count))
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 432, in assertEqual
    super(_Assertions, self).assertEqual(first, second, msg)
  File "/usr/lib/python3.7/unittest/case.py", line 839, in assertEqual
    assertion_func(first, second, msg=msg)
  File "/usr/lib/python3.7/unittest/case.py", line 832, in _baseAssertEqual
    raise self.failureException(msg)
twisted.trial.unittest.FailTest: 2 != 0

tests.storage.test_monthly_active_users.MonthlyActiveUsersTestCase.test_track_monthly_users_without_cap
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/test_mau.py", line 68, in test_simple_deny_mau
    self.create_user("kermit3")
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 331, in __exit__
    self._expectedName, self._returnValue)
twisted.trial.unittest.FailTest: SynapseError not raised (None returned)

tests.test_mau.TestMauLimit.test_simple_deny_mau
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/test_mau.py", line 112, in test_trial_delay
    self.do_sync_for_user(token3)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 331, in __exit__
    self._expectedName, self._returnValue)
twisted.trial.unittest.FailTest: SynapseError not raised (None returned)

tests.test_mau.TestMauLimit.test_trial_delay
===============================================================================
[FAIL]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/test_mau.py", line 168, in test_trial_users_cant_come_back
    self.do_sync_for_user(token1)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/trial/_synctest.py", line 331, in __exit__
    self._expectedName, self._returnValue)
twisted.trial.unittest.FailTest: SynapseError not raised (None returned)

tests.test_mau.TestMauLimit.test_trial_users_cant_come_back
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/erikj/git/synapse/tests/handlers/test_user_directory.py", line 63, in test_handle_local_profile_change_with_support_user
    yield self.handler.handle_local_profile_change(regular_user_id, profile_info)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/erikj/git/synapse/synapse/handlers/user_directory.py", line 133, in handle_local_profile_change
    user_id, profile.display_name, profile.avatar_url, None,
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 372, in runInteraction
    *args, **kwargs
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 421, in runWithConnection
    inner_func, *args, **kwargs
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/threadpool.py", line 250, in inContext
    result = inContext.theWork()
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/threadpool.py", line 266, in <lambda>
    inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/context.py", line 122, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/python/context.py", line 85, in callWithContext
    return func(*args,**kw)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 306, in _runWithConnection
    compat.reraise(excValue, excTraceback)
  File "/home/erikj/.synapse3/lib/python3.7/site-packages/twisted/enterprise/adbapi.py", line 297, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 417, in inner_func
    return func(conn, *args, **kwargs)
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 293, in _new_transaction
    r = func(txn, *args, **kwargs)
  File "/home/erikj/git/synapse/synapse/storage/user_directory.py", line 234, in _update_profile_in_user_dir_txn
    lock=False,  # We're only inserter
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 636, in _simple_upsert_txn
    insertion_values=insertion_values,
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 733, in _simple_upsert_txn_native_upsert
    txn.execute(sql, list(allvalues.values()))
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 89, in execute
    self._do_execute(self.txn.execute, sql, *args)
  File "/home/erikj/git/synapse/synapse/storage/_base.py", line 119, in _do_execute
    sql, *args
sqlite3.OperationalError: ON CONFLICT clause does not match any PRIMARY KEY or UNIQUE constraint

tests.handlers.test_user_directory.UserDirectoryTestCase.test_handle_local_profile_change_with_support_user
===============================================================================
[ERROR]
Traceback (most recent call last):
  File "/home/erikj/git/synapse/tests/storage/test_monthly_active_users.py", line 133, in test_user_last_seen_monthly_active
    self.assertGreater(self.get_success(result), 0)
  File "/usr/lib/python3.7/unittest/case.py", line 1236, in assertGreater
    if not a > b:
builtins.TypeError: '>' not supported between instances of 'NoneType' and 'int'

tests.storage.test_monthly_active_users.MonthlyActiveUsersTestCase.test_user_last_seen_monthly_active
-------------------------------------------------------------------------------
@codecov-io

This comment has been minimized.

Copy link

codecov-io commented Jan 25, 2019

Codecov Report

Merging #4471 into develop will increase coverage by 0.02%.
The diff coverage is 100%.

@@             Coverage Diff             @@
##           develop    #4471      +/-   ##
===========================================
+ Coverage    74.72%   74.75%   +0.02%     
===========================================
  Files          336      336              
  Lines        34062    34062              
  Branches      5540     5540              
===========================================
+ Hits         25453    25462       +9     
+ Misses        7039     7026      -13     
- Partials      1570     1574       +4

@erikjohnston erikjohnston merged commit 57c035d into develop Jan 25, 2019

5 checks passed

ci/circleci: sytestpy2merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy2postgresmerged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3merged Your tests passed on CircleCI!
Details
ci/circleci: sytestpy3postgresmerged Your tests passed on CircleCI!
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details

hawkowl added a commit that referenced this pull request Jan 28, 2019

Revert "Merge pull request #4471 from matrix-org/erikj/sqlite_native_…
…upsert"

This reverts commit 57c035d, reversing
changes made to 4a3f138.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment