Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. #12634

c-bata · 2020-03-27T08:54:30Z

I checked the time taken by flush command with 100 tables. See the source code at https://github.com/c-bata/django-fast-mysql-flush

number of records on each table	before	after `reset_sequences=False` (default)	after `reset_sequences=True` (optional)
10	3.302 sec (+/- 0.076)	0.517 sec (+/- 0.019)	3.658 sec (+/- 0.080)
100	3.323 sec (+/- 0.047)	0.575 sec (+/- 0.025)	3.571 sec (+/- 0.077)
1000	3.577 sec (+/- 0.106)	1.046 sec (+/- 0.029)	3.813 sec (+/- 0.073)

Please note that the benchmark script was run on my laptop. So the times should not be taken precisely.

Ticket URL is https://code.djangoproject.com/ticket/31275

c-bata · 2020-03-27T13:36:23Z

Hmm... I don't know why this change breaks tests like:

https://djangoci.com/job/pull-requests-bionic/database=mysql,label=bionic-pr,python=python3.8/4921/testReport/junit/auth_tests.test_checks/ModelsPermissionsChecksTests/test_clashing_custom_permissions/

Is it an existing issue?

charettes · 2020-03-27T13:40:56Z

@c-bata the failure was introduced by your proposed changes but the failure you pointed at seems to be a side effect on another crash.

If you look at the raw stdout output of the suite execution you'll notice that the first failure is

======================================================================
ERROR [0.011s]: test_for_update_after_from (select_for_update.tests.SelectForUpdateTests)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
MySQLdb._exceptions.IntegrityError: (1062, "Duplicate entry '1' for key 'person_id'")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/select_for_update/tests.py", line 33, in setUp
    self.person_profile = PersonProfile.objects.create(person=self.person)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/query.py", line 443, in create
    obj.save(force_insert=True, using=self.db)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/base.py", line 750, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/base.py", line 788, in save_base
    force_update, using, update_fields,
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/base.py", line 891, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/base.py", line 930, in _do_insert
    using=using, raw=raw,
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/manager.py", line 82, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/query.py", line 1235, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/models/sql/compiler.py", line 1381, in execute_sql
    cursor.execute(sql, params)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/django/db/backends/mysql/base.py", line 73, in execute
    return self.cursor.execute(query, args)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 209, in execute
    res = self._query(query)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/cursors.py", line 315, in _query
    db.query(q)
  File "/home/jenkins/workspace/pull-requests-bionic/database/mysql/label/bionic-pr/python/python3.6/tests/.env/lib/python3.6/site-packages/MySQLdb/connections.py", line 239, in query
    _mysql.connection.query(self, query)
django.db.utils.IntegrityError: (1062, "Duplicate entry '1' for key 'person_id'")

Since sql_flush() is heavily used during tests by TransactionTestCase this issue then snowballed to other failures.

c-bata · 2020-03-27T13:44:31Z

Hi @charettes. Thank you for your swift reply! I see. I'll debug the problem.

django/db/backends/mysql/operations.py

c-bata · 2020-03-27T15:16:10Z

The benchmark result of c3c1470 is:

(venv) $ python manage.py bench_sql_flush 10
0th elapsed: 2.023
1th elapsed: 1.837
2th elapsed: 1.791
3th elapsed: 1.932
4th elapsed: 1.962
elapsed: 1.909 sec (+/- 0.09399143936388689)
(venv) $ python manage.py bench_sql_flush 100
0th elapsed: 2.149
1th elapsed: 2.059
2th elapsed: 2.042
3th elapsed: 2.169
4th elapsed: 2.163
elapsed: 2.116 sec (+/- 0.06102950355064902)
(venv) $ python manage.py bench_sql_flush 1000
0th elapsed: 2.852
1th elapsed: 2.993
2th elapsed: 3.093
3th elapsed: 2.779
4th elapsed: 2.856
elapsed: 2.915 sec (+/- 0.12635354027361947)

https://github.com/c-bata/django-fast-mysql-flush

charettes

Not a big fan of the approximate rows heuristic but I can't think of anything better.

django/db/backends/mysql/operations.py

docs/releases/3.1.txt

django/db/backends/mysql/operations.py

c-bata · 2020-04-01T14:26:32Z

I force-pushed to resolve conflicts and squash commits.

django/db/backends/mysql/operations.py

felixxm · 2020-04-02T05:08:25Z

I agree with Simon. @c-bata Do you have time to update this patch? If not I can do this.

c-bata · 2020-04-02T05:32:41Z

I see! I'll update soon!

docs/releases/3.1.txt

…MySQL. Co-Authored-By: Simon Charette <charettes@users.noreply.github.com>

felixxm · 2020-04-02T10:58:59Z

@c-bata Thanks 👍 Welcome aboard ⛵

c-bata force-pushed the ticket-31275 branch from dd20d86 to 12695e4 Compare March 27, 2020 09:56

c-bata changed the title ~~Refs #31275 -- Make sql_flush faster at MySQL DB~~ Refs #31275 -- Make sql_flush() faster on MySQL. Mar 27, 2020

c-bata force-pushed the ticket-31275 branch 2 times, most recently from baae1d5 to 6cd0a83 Compare March 27, 2020 11:14

charettes reviewed Mar 27, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

This comment has been minimized.

Sign in to view

c-bata force-pushed the ticket-31275 branch from 6cd0a83 to c3c1470 Compare March 27, 2020 14:41

charettes reviewed Mar 27, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

c-bata force-pushed the ticket-31275 branch from 901a73a to f37b9d3 Compare March 28, 2020 08:29

charettes approved these changes Mar 28, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

docs/releases/3.1.txt Outdated Show resolved Hide resolved

c-bata force-pushed the ticket-31275 branch from a52ab51 to 8c65f9f Compare March 29, 2020 09:59

felixxm reviewed Mar 30, 2020

View reviewed changes

django/db/backends/mysql/operations.py Show resolved Hide resolved

felixxm reviewed Mar 30, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

felixxm reviewed Mar 30, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

felixxm reviewed Mar 30, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

felixxm reviewed Mar 30, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

c-bata force-pushed the ticket-31275 branch 3 times, most recently from c932ea3 to 91d3e20 Compare April 1, 2020 14:24

charettes reviewed Apr 1, 2020

View reviewed changes

django/db/backends/mysql/operations.py Outdated Show resolved Hide resolved

felixxm changed the title ~~Refs #31275 -- Make sql_flush() faster on MySQL.~~ Fixed #31275 -- Optimized sql_flush() without sequence resetting on MySQL. Apr 2, 2020

felixxm self-assigned this Apr 2, 2020

c-bata force-pushed the ticket-31275 branch 2 times, most recently from 384f57a to ac948a0 Compare April 2, 2020 07:12

c-bata commented Apr 2, 2020

View reviewed changes

docs/releases/3.1.txt Outdated Show resolved Hide resolved

c-bata force-pushed the ticket-31275 branch 3 times, most recently from 99d8419 to 44a779d Compare April 2, 2020 08:34

Fixed #31275 -- Optimized sql_flush() without resetting sequences on …

8903287

…MySQL. Co-Authored-By: Simon Charette <charettes@users.noreply.github.com>

felixxm force-pushed the ticket-31275 branch from 44a779d to 8903287 Compare April 2, 2020 10:58

felixxm changed the title ~~Fixed #31275 -- Optimized sql_flush() without sequence resetting on MySQL.~~ Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. Apr 2, 2020

felixxm merged commit 8903287 into django:master Apr 2, 2020

c-bata deleted the ticket-31275 branch April 2, 2020 11:37

jonatron mentioned this pull request Oct 23, 2020

Fixed #31276 Add 'flush' option to only clear non-empty tables #12482

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. #12634

Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. #12634

c-bata commented Mar 27, 2020 •

edited

Loading

c-bata commented Mar 27, 2020 •

edited

Loading

charettes commented Mar 27, 2020 •

edited

Loading

c-bata commented Mar 27, 2020

This comment has been minimized.

c-bata commented Mar 27, 2020 •

edited

Loading

charettes left a comment

c-bata commented Apr 1, 2020

felixxm commented Apr 2, 2020

c-bata commented Apr 2, 2020

felixxm commented Apr 2, 2020

Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. #12634

Fixed #31275 -- Optimized sql_flush() without resetting sequence on MySQL. #12634

Conversation

c-bata commented Mar 27, 2020 • edited Loading

c-bata commented Mar 27, 2020 • edited Loading

charettes commented Mar 27, 2020 • edited Loading

c-bata commented Mar 27, 2020

This comment has been minimized.

c-bata commented Mar 27, 2020 • edited Loading

charettes left a comment

Choose a reason for hiding this comment

c-bata commented Apr 1, 2020

felixxm commented Apr 2, 2020

c-bata commented Apr 2, 2020

felixxm commented Apr 2, 2020

c-bata commented Mar 27, 2020 •

edited

Loading

c-bata commented Mar 27, 2020 •

edited

Loading

charettes commented Mar 27, 2020 •

edited

Loading

c-bata commented Mar 27, 2020 •

edited

Loading