Fix #1651657, failback from backup for legacy for small BP instance size - 5.7 #1263

ihanick · 2016-12-25T09:08:11Z

http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/561/
https://bugs.launchpad.net/percona-server/+bug/1651657
#1262 PR 5.6

If buffer pool is bigger than 1GB and there are many buffer pool instances each instance could be smaller than 20MB and causing deadlocks with backoff algorithm.

laurynas-biveinis

In addition to 5.6 comments, a 5.7-specific question: - what happens if we have "backoff" set with a large enough buffer pool and then resize it down so that it needs "legacy"?

ihanick · 2017-03-21T14:26:00Z

http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/738/

merged 5.6 changes
preventing from reducing innodb_buffer_pool size if backoff is enabled and resulting buffer pool instance size will be less than 20MB
using 20MB buffer pool change in innodb_buffer_pool_size_basic to keep test case compatible with backoff

laurynas-biveinis · 2017-03-22T04:12:03Z

The added test is failing intermittently:

innodb.percona_bug1651657 w3 [ fail ]
Test ended at 2017-03-21 10:19:31

CURRENT_TEST: innodb.percona_bug1651657
--- /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/centos6-32/mysql-test/suite/innodb/r/percona_bug1651657.result 2017-03-21 12:25:46.228520091 +0300
+++ /mnt/workspace/mysql-5.7-param/BUILD_TYPE/debug/Host/centos6-32/build/mysql-test/var/3/log/percona_bug1651657.reject 2017-03-21 17:19:31.481432098 +0300
@@ -4,6 +4,9 @@
ERROR 42000: Variable 'innodb_empty_free_list_algorithm' can't be set to the value of 'backoff'
include/assert.inc [Small buffer pool instances should use legacy]
SET @@GLOBAL.innodb_buffer_pool_size = 1189085184;
+Timeout in wait_condition.inc for SELECT SUBSTR(variable_value, 1, 34) = 'Completed resizing buffer pool at '

FROM information_schema.global_status
+WHERE LOWER(variable_name) = 'innodb_buffer_pool_resize_status'
SET GLOBAL innodb_empty_free_list_algorithm="backoff";
SET @@GLOBAL.innodb_buffer_pool_size = 1075838976;
ERROR 42000: Variable 'innodb_buffer_pool_size' can't be set to the value of '1075838976'

laurynas-biveinis · 2017-03-22T04:13:25Z

mysql-test/suite/innodb/t/percona_bug1651657.test

+--source include/assert.inc
+
+call mtr.add_suppression("InnoDB: innodb_empty_free_list_algorithm = 'backoff' requires at least 20MB buffer pool instances.");
+--error 1231


Please use ER_ mnemonics

laurynas-biveinis · 2017-03-22T04:21:48Z

storage/innobase/handler/ha_innodb.cc

+bool
+innodb_empty_free_list_algorithm_allowed(
+	srv_empty_free_list_t	algorithm,
+	ulint new_buf_pool_sz = 0)


long long arg? It's getting called from a "long long" MySQL variable validation function

laurynas-biveinis · 2017-03-22T04:21:56Z

storage/innobase/handler/ha_innodb.cc

+@param[in]	algorithm	desired algorithm from srv_empty_free_list_t
+@param[in]	new_buf_pool_sz	requested buffer pool size
+@return	true if it's possible to enable backoff. */
+static inline


Just static

laurynas-biveinis · 2017-03-22T04:22:41Z

storage/innobase/handler/ha_innodb.cc

+				"because of small buffer pool size. "
+				"In order to use backoff, "
+				"increase buffer pool at least up to 20MB.\n");
+			srv_empty_free_list_algorithm


Indentation

ihanick · 2017-03-28T17:12:49Z

http://jenkins.percona.com/job/mysql-5.7-param/756/

using ER_WRONG_VALUE_FOR_VAR
removed inline
using long long for innodb_empty_free_list_algorithm_allowed buffer pool size parameter
fixed ident for srv_empty_free_list_algorithm
removed buffer pool change (from 1GB to 1080MB), small instances (centos6-32bit) getting out of memory errors (mmap returns errno 12).

I'm getting warnings with 19MB buffer pool instance (1026 buffer pool size, 54 instances) even with legacy flush algorithm:

14:17:44 2017-03-28T14:17:36.086635Z 8 [Warning] InnoDB: Difficult to find free blocks in the buffer pool (338 search iterations)! 0 failed attempts to flush a page! Consider increasing the buffer pool size. It is also possible that in your Unix version fsync is very slow, or completely frozen inside the OS kernel. Then upgrading to a newer version of your operating system may help. Look at the number of fsyncs in diagnostic info below. Pending flushes (fsync) log: 0; buffer pool: 0. 17861 OS file reads, 29281 OS file writes, 8 OS fsyncs. Starting InnoDB Monitor to print further diagnostics to the standard output.

http://jenkins.percona.com/job/mysql-5.7-param/756/BUILD_TYPE=debug,Host=centos6-32/consoleFull

Thus for 5.7 it could be a good idea to limit buffer pool instance size at 20MB+ and remove backoff to legacy failback (current tests are executed with at least 128MB buffer pool instances if multiple buffer pool instances are enabled in 5.7).

laurynas-biveinis

Please restore the bp resize test, it is important and tests an affected code path. In order for the testcase to be lighter on our Jenkins, please reduce the instance count, it could be as small as 2.

Limitting minimum buffer pool size is not desirable as there are users running application tests with minimally-sized MySQL instances

laurynas-biveinis · 2017-03-29T02:54:11Z

storage/innobase/handler/ha_innodb.cc

+buffer pool belongs to LRU or freelist.
+At the same time buf_flush_LRU_list_batch
+keeps up to BUF_LRU_MIN_LEN in LRU.
+In order to avoid deadlock baclkoff requires buffer pool


ihanick · 2017-03-29T19:26:00Z

http://jenkins.percona.com/job/mysql-5.7-param/764/
made a split:

one test case to test multiple buffer pools and fallback to legacy. Multiple buffer pools are allowed in 5.7 if innodb_buffer_pool_size > 1G, minimal number of buffer pool instances to cause fallback: 1026M/19M = 54.
second test case handles buffer pool resize from 20MB to 19MB (with one instance) - resize disallowed if algorithm is backoff, no actual buffer pool resize happening
fixed a typo in comment

laurynas-biveinis · 2017-03-30T02:17:21Z

mysql-test/suite/innodb/t/percona_bug1651657.test

@@ -0,0 +1,13 @@
+--source include/have_innodb.inc


Please add --source include/big_test.inc

laurynas-biveinis · 2017-03-30T02:18:25Z

mysql-test/suite/innodb/t/percona_bug1651657_resize.test

+
+set @old_innodb_buffer_pool_size = @@innodb_buffer_pool_size;
+
+# Test buffer pool shink with backoff algorithm.


ihanick · 2017-03-30T12:20:20Z

http://jenkins.percona.com/view/5.7/job/mysql-5.7-param/767/

added big for 1GB BP test
fixed typo in test comment

laurynas-biveinis reviewed Dec 27, 2016

View reviewed changes

Fix #1651657, failback from backup for legacy for small BP instance size

4d70ecc

ihanick force-pushed the 5.7-ps-bug1651657 branch 2 times, most recently from c5a33ba to 8103a23 Compare March 21, 2017 12:53

laurynas-biveinis suggested changes Mar 22, 2017

View reviewed changes

ihanick force-pushed the 5.7-ps-bug1651657 branch 5 times, most recently from 16c79f5 to a4f31d4 Compare March 28, 2017 10:23

laurynas-biveinis suggested changes Mar 29, 2017

View reviewed changes

ihanick force-pushed the 5.7-ps-bug1651657 branch from a4f31d4 to 5c636d3 Compare March 29, 2017 10:41

laurynas-biveinis suggested changes Mar 30, 2017

View reviewed changes

Merge branch '5.6-ps-bug1651657' into 5.7-ps-bug1651657

19bc3e4

ihanick force-pushed the 5.7-ps-bug1651657 branch from 5c636d3 to 19bc3e4 Compare March 30, 2017 04:57

laurynas-biveinis approved these changes Mar 31, 2017

View reviewed changes

ihanick merged commit 2574fba into percona:5.7 Mar 31, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #1651657, failback from backup for legacy for small BP instance size - 5.7 #1263

Fix #1651657, failback from backup for legacy for small BP instance size - 5.7 #1263

ihanick commented Dec 25, 2016

laurynas-biveinis left a comment

ihanick commented Mar 21, 2017

laurynas-biveinis commented Mar 22, 2017

laurynas-biveinis Mar 22, 2017

laurynas-biveinis Mar 22, 2017

laurynas-biveinis Mar 22, 2017

laurynas-biveinis Mar 22, 2017

ihanick commented Mar 28, 2017

laurynas-biveinis left a comment

laurynas-biveinis Mar 29, 2017

ihanick commented Mar 29, 2017

laurynas-biveinis Mar 30, 2017

laurynas-biveinis Mar 30, 2017

ihanick commented Mar 30, 2017


		set @old_innodb_buffer_pool_size = @@innodb_buffer_pool_size;

		# Test buffer pool shink with backoff algorithm.

Fix #1651657, failback from backup for legacy for small BP instance size - 5.7 #1263

Fix #1651657, failback from backup for legacy for small BP instance size - 5.7 #1263

Conversation

ihanick commented Dec 25, 2016

laurynas-biveinis left a comment

Choose a reason for hiding this comment

ihanick commented Mar 21, 2017

laurynas-biveinis commented Mar 22, 2017

laurynas-biveinis Mar 22, 2017

Choose a reason for hiding this comment

laurynas-biveinis Mar 22, 2017

Choose a reason for hiding this comment

laurynas-biveinis Mar 22, 2017

Choose a reason for hiding this comment

laurynas-biveinis Mar 22, 2017

Choose a reason for hiding this comment

ihanick commented Mar 28, 2017

laurynas-biveinis left a comment

Choose a reason for hiding this comment

laurynas-biveinis Mar 29, 2017

Choose a reason for hiding this comment

ihanick commented Mar 29, 2017

laurynas-biveinis Mar 30, 2017

Choose a reason for hiding this comment

laurynas-biveinis Mar 30, 2017

Choose a reason for hiding this comment

ihanick commented Mar 30, 2017