MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) #248

sensssz · 2016-10-22T18:16:03Z

This branch introduces a new scheduling algorithm (Variance-Aware-Transaction-Scheduling, VATS) for the record lock manager of InnoDB based on MariaDB 10.2. Instead of using First-Come-First-Served (FCFS), the newly introduced algorithm prefers the eldest transaction. A configuration parameter (innodb_lock_schedule_algorithm) is introduced for users to choose between VATS and FCFS (the default one). We've extensively tested this algorithm in many workloads. The algorithm is very simple, and the changes are very local, but it significantly improves performance (in terms of average latency and throughput) and predictability (in terms of reduction of tail and quantile latencies) For more details, please refer to this paper http://arxiv.org/abs/1602.01871

….2-vats Sync upstream and rebase.

….2-vats Rebase

…ert.

janlindstrom

Changes look correct and fine. There is small differences on used style and normal InnoDB style because InnoDB style uses tab for indention not spaces. Could you please fix the indention before I will merge this. Currently on 10.2 there is no xtradb.

janlindstrom · 2016-10-23T05:25:40Z

sql/rpl_parallel.cc

@@ -1410,7 +1410,7 @@ handle_rpl_parallel_thread(void *arg)
  mysql_mutex_unlock(&rpt->LOCK_rpl_thread);

  my_thread_end();
-
+  


This is unnecessary.

janlindstrom · 2016-10-23T05:26:51Z

storage/innobase/include/trx0trx.h

@@ -1096,6 +1096,8 @@ struct trx_t {

 	time_t		start_time;	/*!< time the state last time became
 					TRX_STATE_ACTIVE */
+    clock_t     start_time_micro; /*!< start time of the transaction
+                    in microseconds. */


Indention does not look correct in all places, InnoDB uses tabs not spaces.

janlindstrom · 2016-10-23T05:28:03Z

storage/innobase/lock/lock0lock.cc

+	if (trx_is_high_priority(lock1->trx)) {
+        return true;
+    }
+    if (trx_is_high_priority(lock2->trx)) {


Indention does not look correct here.

janlindstrom · 2016-10-23T05:28:35Z

storage/innobase/lock/lock0lock.cc

+    ulint               space;
+    ulint               page_no;
+	ulint				rec_fold;
+    hash_table_t*       hash;


Indention in this function does not look correct, use tab-characters not spaces.

janlindstrom · 2016-10-23T05:30:32Z

storage/innobase/lock/lock0lock.cc

@@ -4823,7 +5197,7 @@ lock_release(
 		ut_d(lock_check_dict_lock(lock));

 		if (lock_get_type_low(lock) == LOCK_REC) {
-
+            


This looks unnecessary.

janlindstrom · 2016-10-23T05:31:32Z

storage/innobase/lock/lock0lock.cc

@@ -52,12 +53,16 @@ Created 5/7/1996 Heikki Tuuri
 #include "row0mysql.h"
 #include "pars0pars.h"

+#include <inttypes.h>


Where you use this ?

sensssz · 2016-10-23T17:44:15Z

@janlindstrom Hi, Jan. I've replaced all space indention with tab indention in my code. I also undid all changes to XtraDB. Could you kindly take a look?

svoj · 2016-10-24T09:03:20Z

Hi Jiamin,

Thanks for your contribution. These changes were originally proposed for 10.1 in PR#245, Normally there's no need to create separate pull request for 10.2, because it will be merged from 10.1 anyway.

But I guess target version of this PR is not decided currently, so I keep both PR's open.

Thanks,
Sergey

sensssz · 2016-10-24T19:20:48Z

@svoj Hi, Sergey. I created a separate pull request for 10.2 because the locking system has undergone some significant code refactor in it, so the pull request for 10.1 would not be compatible with 10.2. Also from our information it would be easier for you to accept new features in version 10.2.

This patch contains several fixes that hasn't been adopted in the previous patch yet. Do you want me to update that patch as well?

janlindstrom · 2016-10-24T19:35:17Z

Actually, I did accept this also for 10.1. So if there is some fixed needed could you create new pull request for them and make VATS as default.

svoj · 2016-10-24T19:36:33Z

@sensssz, you're right. It perfectly makes sense if code is too different between versions. Please follow up with Jan regarding additional patches.

sensssz · 2016-10-24T21:03:56Z

@janlindstrom Sure I will create a pull request for 10.1. For that version should I also implement it in XtraDB or should I undo all the changes to XtraDB?

barzan · 2016-10-24T21:21:30Z

Why not keep the XtraDB changes (for 10.1)?

vuvova · 2016-10-25T05:09:35Z

@janlindstrom I don't think this should go into 10.1 at all, 10.1 is GA for quite a while, no new features.

janlindstrom · 2016-10-25T12:38:46Z

No undo for XtraDB and for 10.1 for now.

dr-m · 2018-07-05T07:49:37Z

@sensssz Hi Jiamin,
The parameter innodb_lock_schedule_algorithm=vats seems to be causing a debug assertion failure in MariaDB, reported as MDEV-16664.

I see that you also contributed this to MySQL, and it was introduced there under WL#10793, MySQL Bug#84266, mysql/mysql-server#115, mysql/mysql-server@fb056f4

In MySQL 8.0, I found the following bug fixes related to this:
mysql/mysql-server@f395242 (a test case refers to MySQL Bug #89829, which I cannot access).
mysql/mysql-server@2248a3e, mysql/mysql-server@c917825 (these could be addressing a similar issue as MDEV-16664)
mysql/mysql-server@c1990af (memory management issue, apparently specific to the MySQL 8.0 implementation; in MariaDB lock_grant_and_move_on_page() which corresponds to the MySQL 8.0 lock_grant_vats() does not allocate memory)

It looks like MySQL 8.0 employs a threshold based on lock_sys->n_waiting < LOCK_VATS_THRESHOLD (defined as 32), so it would be dynamically choosing between the algorithms FCFS and VATS (also called CATS by them) based on load. This could make it harder to repeat the MDEV-16664 failure scenario in MySQL 8.0.

I would appreciate it if you could comment how the implementations of the algorithm differ between MariaDB 10.2 and MySQL 8.0.

sensssz · 2018-07-05T14:31:18Z

Hi, Marko,

The algorithms we merged into MariaDB is actually different from what we merged into MySQL 8.0. They are based on two of our papers (VATS, CATS). VATS grants locks to transactions according to their age, while CATS does so according to each transaction's dependency set size (the number of transactions being blocked on it). Therefore, CATS's implementation is much more complicated because it also needs to maintain each transaction's dependency set size.

For VATS, the idea is very simple: it inserts transactions to the queue according to their ages so that they are granted in this order. An additional thing I did was to keep the granted locks at the front of the queue.

Is there a specific test case that I can use to reproduce the problem? I will look into this problem as well.

vuvova · 2020-08-17T11:06:00Z

Hi, @sensssz,

sorry that you didn't get a reply earlier.

MDEV-16664 is still not closed. May be you'd like to check the issue itself, it's https://jira.mariadb.org/browse/MDEV-16664

There are few ways to reproduce the problem, in issue comments. Either with an RQG (random query generator, https://github.com/MariaDB/randgen) or simply by running one of the mysql-test cases on repeat.

janlindstrom · 2020-08-17T11:56:01Z

One additional problem with VATS is the fact that it is not compatible with Galera replication where if lock request is done by brute force thread (BF) no other lock request may granted before it as BF transactions may not wait for locks (if there is already granted lock request its holder thread is selected as victim and killed). This fact naturally can be fixed with, if wsrep_thd_is_BF(trx->mysql_thd) we set transaction age so high that no other transaction lock request can be granted before it.

dr-m · 2020-09-18T16:21:03Z

@sensssz, I posted in MDEV-16664 some information from an rr replay trace that I analyzed. It looks like two LOCK_X were wrongly granted on the same record for two concurrently running DELETE statements. Can you please check that analysis and suggest what to check further? The GDB commands watch and break are rather powerful in rr replay, but only when you know what to look for. Somehow the conflicting lock is not being reached via the hash table.

dr-m · 2020-10-02T16:26:35Z

@sensssz Due to lack of response, I am going to deprecate the configuration parameter innodb_lock_schedule_algorithm in MariaDB Server 10.2 (issue a warning if it is set to the value VATS) and remove the parameter in MariaDB Server 10.6.

dr-m · 2020-10-05T08:45:02Z

I added a deprecation and corruption warning in 295e2d5
and removed this code from the MariaDB Server 10.6 branch in b4fb15c.

sensssz added 15 commits October 17, 2016 21:56

Implement VATS in XtraDB and InnoDB.

87abc87

Disable VATS on slave servers during replication.

88519c9

Implement VATS in XtraDB and InnoDB.

0b7c35c

Disable VATS on slave servers during replication.

a9bdea5

Merge branch '10.2-vats' of https://github.com/sensssz/server into 10…

5c40954

….2-vats Sync upstream and rebase.

Change VATS implementation.

5661b5c

Implement VATS in XtraDB and InnoDB.

c455898

Disable VATS on slave servers during replication.

03b3425

Change VATS implementation.

0e60f89

Merge branch '10.2-vats' of https://github.com/sensssz/server into 10…

0f93b53

….2-vats Rebase

Bug fix: remove redundant code from check_deadlock_result. Remove ass…

f62fd5a

…ert.

Add INNODB_LOCK_SCHEDULE_ALGORITHM to sysvars_innodb.result

36841ac

A few bug fixes. Use thd_is_slave_replication.

adaebd2

Add a NULL check for thd_is_replication_slave_thread

04f241a

Remove unnecessary changes. Apply changes to XtraDB.

149581a

janlindstrom requested changes Oct 23, 2016

View reviewed changes

sensssz added 4 commits October 23, 2016 13:17

Style fixes.

fd30d07

Change all space indentions to tab.

a24e625

Remove duplicated comments.

0064d85

Undo changes to XtraDB.

f6da2b4

svoj assigned janlindstrom Oct 24, 2016

svoj changed the title ~~Add new scheduling algorithm for reducing tail latencies (for 10.2)~~ MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) Oct 24, 2016

janlindstrom approved these changes Oct 24, 2016

View reviewed changes

janlindstrom merged commit b09b316 into MariaDB:10.2 Oct 24, 2016

svoj added this to the 10.2 milestone Jul 11, 2017

dr-m pushed a commit that referenced this pull request Nov 10, 2017

Plugins: lost terminator in versioning plugin [fixes #248]

b231a85

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) #248

MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) #248

sensssz commented Oct 22, 2016 •

edited

Loading

janlindstrom left a comment

janlindstrom Oct 23, 2016

janlindstrom Oct 23, 2016

janlindstrom Oct 23, 2016

janlindstrom Oct 23, 2016

janlindstrom Oct 23, 2016

janlindstrom Oct 23, 2016

sensssz commented Oct 23, 2016

svoj commented Oct 24, 2016

sensssz commented Oct 24, 2016

janlindstrom commented Oct 24, 2016

svoj commented Oct 24, 2016

sensssz commented Oct 24, 2016

barzan commented Oct 24, 2016

vuvova commented Oct 25, 2016

janlindstrom commented Oct 25, 2016

dr-m commented Jul 5, 2018 •

edited

Loading

sensssz commented Jul 5, 2018

vuvova commented Aug 17, 2020 •

edited

Loading

janlindstrom commented Aug 17, 2020

dr-m commented Sep 18, 2020

dr-m commented Oct 2, 2020

dr-m commented Oct 5, 2020

		@@ -1410,7 +1410,7 @@ handle_rpl_parallel_thread(void *arg)
		mysql_mutex_unlock(&rpt->LOCK_rpl_thread);

		my_thread_end();

		@@ -4823,7 +5197,7 @@ lock_release(
		ut_d(lock_check_dict_lock(lock));

		if (lock_get_type_low(lock) == LOCK_REC) {

MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) #248

MDEV-11039 - Add new scheduling algorithm for reducing tail latencies (for 10.2) #248

Conversation

sensssz commented Oct 22, 2016 • edited Loading

janlindstrom left a comment

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

janlindstrom Oct 23, 2016

Choose a reason for hiding this comment

sensssz commented Oct 23, 2016

svoj commented Oct 24, 2016

sensssz commented Oct 24, 2016

janlindstrom commented Oct 24, 2016

svoj commented Oct 24, 2016

sensssz commented Oct 24, 2016

barzan commented Oct 24, 2016

vuvova commented Oct 25, 2016

janlindstrom commented Oct 25, 2016

dr-m commented Jul 5, 2018 • edited Loading

sensssz commented Jul 5, 2018

vuvova commented Aug 17, 2020 • edited Loading

janlindstrom commented Aug 17, 2020

dr-m commented Sep 18, 2020

dr-m commented Oct 2, 2020

dr-m commented Oct 5, 2020

sensssz commented Oct 22, 2016 •

edited

Loading

dr-m commented Jul 5, 2018 •

edited

Loading

vuvova commented Aug 17, 2020 •

edited

Loading