Avoid "The total number of locks exceeds the lock table size" error

[stig]

I was experiencing this error when running the reports:prune task:

Preparing to delete from resource_statuses
2015-11-16 15:18:16: Deleting 108787 orphaned records from resource_statuses
rake aborted!   99% |####################################################################################################################  | ETA:  00:00:00
ActiveRecord::StatementInvalid: Mysql::Error: The total number of locks exceeds the lock table size: delete from resource_statuses where report_id not in (select id from reports) limit 1000

Tasks: TOP => reports:prune:orphaned
(See full trace by running task with --trace)

Every time it gets almost to the end, then failing on the last batch. Doesn't matter if there's millions of rows to be deleted or a couple thousand. Logging in and attempting to run the delete by hand yields the same result. Reducing the limit to 100 makes the query execute instantly, several times, until there's less than 100 rows to delete again, when it breaks. Same for setting the limit to 10. I eventually manage to delete all the rows by reducing the limit to 1, and running it repeatedly. However, running the reports:prune task again immediately still fails. (This table grows fast!)

Preparing to delete from resource_statuses
2015-11-16 15:21:51: Deleting 2806 orphaned records from resource_statuses
rake aborted!   71% |###################################################################################                                   | ETA:  00:00:00
ActiveRecord::StatementInvalid: Mysql::Error: The total number of locks exceeds the lock table size: delete from resource_statuses where report_id not in (select id from reports) limit 1000

Tasks: TOP => reports:prune:orphaned
(See full trace by running task with --trace)

I tried to increase the innodb_buffer_pool_size ten-fold, as described here but to no avail. Attempting to delete by hand reveals:

mysql> select count(*), count(distinct report_id) from resource_statuses where report_id not in (select  id  from reports ) ;
+----------+---------------------------+
| count(*) | count(distinct report_id) |
+----------+---------------------------+
|      806 |                         3 |
+----------+---------------------------+
1 row in set (26.22 sec)

mysql> delete from resource_statuses where report_id not in (select  id  from reports ) limit 800;                                                          Query OK, 800 rows affected (0.08 sec)

mysql> delete from resource_statuses where report_id not in (select  id  from reports ) limit 800;
ERROR 1206 (HY000): The total number of locks exceeds the lock table size
mysql> delete from resource_statuses where report_id not in (select  id  from reports ) ;
ERROR 1206 (HY000): The total number of locks exceeds the lock table size
mysql> select count(*), count(distinct report_id) from resource_statuses where report_id not in (select  id  from reports ) ;
+----------+---------------------------+
| count(*) | count(distinct report_id) |
+----------+---------------------------+
|        6 |                         1 |
+----------+---------------------------+
1 row in set (36.47 sec)

mysql> delete from resource_statuses where report_id not in (select  id  from reports ) limit 6;
Query OK, 6 rows affected (0.06 sec)

It looks like, for this table only that MySQL is really not liking to have a limit that is bigger than the number of rows to be deleted. So, I applied this change: https://github.com/sodabrew/puppet-dashboard/pull/335/files -- and now it succeeds! Sure, it leaves a few orphaned rows on each run, but those will be cleaned up on the next run.

My the way, I'm using mysql_config --version 5.1.73.

[stig]

I admit I have no idea what the hell is going on there. It shouldn't make a difference, but it does.

[stig]

I admit I have no idea what the hell is going on there. It shouldn't make a difference, but it does.

[stig]

Ping! If this fix is not something you're interested in, please just close the PR so it doesn't clog up my list of open PRs any more.

[stig]

Ping! If this fix is not something you're interested in, please just close the PR so it doesn't clog up my list of open PRs any more.

[sodabrew]

I found someone else describing the same problem with the last batch (where the number of matching records is below the limit value): http://stackoverflow.com/a/23751982

This feels like a strange MySQL-ism that may have been fixed in the later 5.x releases?

At any rate, the patch is reasonable, and the number of records that might be left behind isn't much consequence, as large sites may have many tens of thousands of records to prune daily.

Apologies I've not kept up with the PRs here for quite some time.

[sodabrew]

I found someone else describing the same problem with the last batch (where the number of matching records is below the limit value): http://stackoverflow.com/a/23751982

This feels like a strange MySQL-ism that may have been fixed in the later 5.x releases?

At any rate, the patch is reasonable, and the number of records that might be left behind isn't much consequence, as large sites may have many tens of thousands of records to prune daily.

Apologies I've not kept up with the PRs here for quite some time.