Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
FB8-59: Database admission control (facebook#969) (facebook#969)
Summary: JIRA: https://jira.percona.com/browse/FB8-59 Reference patch: facebook@5cbfab0 Reference patch: facebook@0ebaf75 Reference patch: facebook@cf7ce48 Reference patch: facebook@3842a8f Reference patch: facebook@bcbd77a Reference patch: facebook@e6c2b87 Reference patch: facebook@928c2a5 Reference patch: facebook@86587af Reference patch: facebook@7870828 Reference patch: facebook@0151b45 Reference patch: facebook@75de821 Reference patch: facebook@08d4174 Reference patch: facebook@c1e5b4f Reference patch: facebook@08368d4 Reference patch: facebook@a2198e0 Reference patch: facebook@0730342 Reference patch: facebook@d868c1b Reference patch: facebook@645fc5e Reference patch: facebook@0b130f1ebc9 Reference patch: facebook@f49a4a74945 <Comment by Percona developers> This patch ports Database admission control. The multi-tenancy plugin changes are not ported. This code will be eventually moved to new plugin of audit type. There are possible improvements with code. Below are the deferred items (makes it easier for review. Can compare with 5.6 code) 1. Remove MT_RESOURCE_ATTRS. we only need database name (also removes the typedefs) 2. Remove enums enum_multi_tenancy_resource_type & enum_multi_tenancy_return_type 3. Remove the usage of 'multi tenancy' word thd->is_real_trans is not ported from 5.6. Instead an equivalent is used. This is verified in 5.6 and it works well. (5.6 equivalent is: thd->transaction.all.ha_list == NULL;) "!thd->get_transaction()->is_active(Transaction_ctx::SESSION)" is used instead of thd->is_real_trans. The above check seems sufficient but after porting thd->is_real_trans, it should be possible to replace the above check with thd->is_real_trans </end> ------- facebook@5cbfab0 -------- Add two global sys vars max_running_queries and max_waiting_queries. They control maximum number of running queries on a database and maximum waiting queries when max_running_queries limit is crossed on that database. A value of 0 implies no limits are applied for the queries. If max_waiting_queries limit is crossed, then new queries will simply fail. For waiting threads SHOW PROCESSLIST will show the state as 'waiting for admission'. Only dagtabase set at the session level is considered in these checks. Admission checks are by-passed in the following cases 1. Query is run by super user. 2. Query is run by replication threads. 3. No database is set for the session. 4. max_running_queries is 0. Performance testing 1. start MySQL using mtr > mtr --start --mysqld=--default_storage_engine=InnoDB 2. Connect to mysql using > mysql --socket=mysql-test/var/tmp/mysqld.1.sock --user=root 3. Set innodb_flush_log_at_trx_commit=2. mysql> set global.innodb_flush_log_at_trx_commit=2 4. Check rows inserted using > mysqladmin extended-status --user=root -i 1 -r --socket=mysql-test/var/tmp/mysqld.1.sock status | grep "Innodb_rows_inserted" 5. Run mysqlslap using > mysqlslap --auto-generate-sql --number-of-queries=5000000 --concurrency=$concurrency --auto-generate-sql-load-type=write --auto-generate-sql-add-autoincrement --csv=a --user=root --socket=mysql-test/var/tmp/mysqld.1.sock == Innodb_rows_inserted with concurrency=100 == 38106 (max_running_queries=0, max_waiting_queries=0) 38733 (max_running_queries=1000, max_waiting_queries=0) 38365 (max_running_queries=100, max_waiting_queries=0) 40308 (max_running_queries=70, max_waiting_queries=0) 43790 (max_running_queries=50, max_waiting_queries=0) 47321 (max_running_queries=20, max_waiting_queries=0) 49473 (max_running_queries=10, max_waiting_queries=0) 54851 (max_running_queries=5, max_waiting_queries=0) == Innodb_rows_inserted with concurrency=100 == 26516 (max_running_queries=0, max_waiting_queries=0) 44494 (max_running_queries=10, max_waiting_queries=0) 51193 (max_running_queries=5, max_waiting_queries=0) The above numbers show the improvement in quality of service. == Innodb_rows_inserted with concurrency=20 == 47228 (max_running_queries=0, max_waiting_queries=0) 47107 (max_running_queries=100, max_waiting_queries=0) 47082 (max_running_queries=20, max_waiting_queries=0) 47515 (max_running_queries=10, max_waiting_queries=0) * There isn't much difference under low concurrency workloads. * There isn't much difference with non zero max_waiting_queries values greater than max_running_queries. With values lower than max_running_queries, lots of queries simply fail, so the qps in this scenario is of no value. Original Reviewers: tianx, kradhakrishnan, jtolmer, jkedgar ----- facebook@0ebaf75 ----- Added the following status variables: 1. Database_admission_control_aborted_queries gives the total number of queries aborted because of queue overfill. 2. Database_admission_control_running_queries gives the total number of running queries across all the databases. 3. Database_admission_control_waiting_queries gives the total number of waiting queries across all the databases. ----- facebook@cf7ce48 ------ In order to avoid possible mysqld hangs, we need an option to blacklist specific commands in admission control. Example usage is set global.admission_control_filter='BEGIN,COMMIT,SET'; An empty value for admission_control_filter implies no commands are blacklisted in the admission control. ---- facebook@bcbd77a ------ Initial version of admission control applied limits on each query statement in multi query packet. The behavior is modified to the following: A thread enters admission control when executing the first non-filtered command in the multi query packet and exits after executing all the query statements in the multi query packet. This diff also fixes a bug where admission control limits are not initialized during mysqld startup. ----- facebook@928c2a5 -------- Get accurate status variables Hold global admission control locks to get accurate value of running/waiting queries. Initial stats were not accurate causing test failures ------- facebook@86587af ----------- Added SHOW command filter in admission control. ------- facebook@7870828 ------ The original ac_map was a map of unique_ptr<Ac_info>. When accessing the value we had to use a reference to the value. This diff changes that so that we use a shared_ptr<Ac_info> instead so we can just pass it around have have reference counting automatically. ---- facebook@75de821 ------ Fixing the following two issues in admission control when changing database is in a multi-statement query: - If the session hasn't set a database, admission control will allow arbitrary number of multi-query packet to run, even the packet starts with `use [db_name]`. - If the session already has a default database, the admission control will use the limit of the default database, even if we change database in the multi-query packet. This diff fixes the above issues in the following way: - The admission control flag (is_in_ac) is reset when a `use` command is executed. So in the multi-statement query, the subsequent sub-queries are not bypassed. - Admission control uses the `thd->db` directly instead of attribute map, so the attribute map doesn't need to store the session database. - `USE [db_name]` statement is added to `admission_control_filter`. In addition, I also changed the multitenancy plugin interface to use a wrapper object to pass down the connection_attrs_map, query_attrs_map, and session database info. Attribute maps will not be modified. --------- facebook@c1e5b4f -------- Fix resource leaks by admission_control_by_trx - Rollback will need to mark is_real_trans - Query resource needs to be released when THD is terminated. Pull Request resolved: facebook#969 Reviewed By: lloyd Differential Revision: D14567747 (facebook@03e9575) Pulled By: lth fbshipit-source-id: 5c1fdbde73c
- Loading branch information
1 parent
c556804
commit d31f8c3