Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
WL#14591 InnoDB: Make system variable
innodb_open_files
dynamic
- The `innodb_open_files` system variable can now be set with a dynamic SQL procedure `innodb_set_open_files_limit(N)`. If the new value is too low, an error is returned to client with the minimum value presented. If the value is out of bounds or of incorrect type, it will be reported as error also. - `Fil_system::set_open_files_limit` was added to allow changes to the global opened files limit. The `Fil_system::m_max_n_open` is atomic now and extracted to a separate class `fil::detail::Open_files_limit`, instantiated as `Fil_system::m_open_files_limit`. - `Fil_shard::open_file` is refactored: * Now it has a while loop that waits for the file to be ready to be opened and for all right to open the additional file within the limit acquired. It checks all necessary conditions like `prevent_file_open`. * The release of the rights is encapsulated entirely within this method, not like the open_slot in all usages of the `mutex_acquire_and_get_space`. * It is assured that we don't open more than 90% of the files that are not ever added to the LRU - this leaves 10% buffer for the system to continue operation with all regular files and not hang. * All waits in this method to not spin, instead the sleep for 1ms is performed if we can't have the file opened. - `Fil_shard::mutex_acquire_and_get_space` is removed - now one just need to get the space object and call `prepare_file_for_io`. - Removed a TODO comment in the `Fil_shard::file_close_to_free` that related to removed argument in a scenario that is not a problem anymore as there is no open file mutex for some time already. - `Fil_shard::prepare_file_for_io` has the check for over-booked limit removed. - `Fil_shard::write_completed` assumes the temporary tablespaces are not buffered and we don't add them to unflushed list. - `Fil_shard::reserve_open_slot`, Fil_shard::release_open_slot and static Fil_shard::s_open_slot were removed. Now we have CAS-based system of assuring the opened files will not exceed the limit set. - `Fil_system::close_file_in_all_LRU` uses `Fil_system::s_next_shard_to_close_from_LRU` to distribute file closures from LRU to different shard each time it is called. - `Fil_shard::open_file` and `Fil_shard::prepare_file_for_io` have the extend parameter removed. It is not needed after `prevent_file_open` was refactored. - Static `Fil_shard::s_n_spaces_in_lru` is replaced by `Fil_system::s_n_files_not_belonging_in_lru`. - `Fil_shard::s_files_opened_limit_last_message` is introduced to throttle the number of warnings on long file open globally to one per 10 seconds. - `Fil_shard::wait_for_io_to_stop` was removed and inlined into `Fil_shard::open_file()` directly. Also, it did not make any sense before, as - `mutex_acquire_and_get_space()` was bumping `file->in_use` before calling `wait_for_io_to_stop()`, which was waiting for the `stop_ios` to be false, while the rename thread could not close the file as it has `in_use` flag, so the rename flag had to reset the `stop_ios` flag. This behaviour is not changed much. We still don't check in `prepare_file_for_io` if the file is about to be renamed to not starve the thread that waits for the file to be able to be renamed. - `fil_space_t::stop_ios` is renamed to `prevent_file_open` - it was used only when we wanted to prevent tablespace files from being opened, in preparation to rename it. - `Fil_shard::close_file` has the unused param `LRU_close` removed. - `Fil_shard::close_files_in_LRU` has the excessive debug logging removed. - `Fil_shard::file_opened` is partially inlined into `file_open()` and is renamed to `add_to_lru` to match the `remove_from_LRU` counterpart - now it is it's only responsibility and it is used in two places. - `fil_node_t::in_use` is renamed to `is_being_extended` - it was used only to synchronize file size extension. - `OS_FILE_TOO_MANY_OPENED` error code is added with the support for windows and POSIX. - The temporary tablespaces are no longer exempted from the opened files LRU. - Extracted common logic around dynamic SQL procedures into `srv0dynamic_procedures.h` - In `Fil_shard::get_file_size` we do not check for `OS_FILE_TOO_MANY_OPENED` - we have a right to open a file already and if it fails we report an error instead of trying to fix the situation. - Modified `innodb_open_file_hang.test` to additionally open 500 concurrent connections in background. Running them by executing mysql clients in background allow to test more connections than the limit, without hanging the connections like the original mysqltest-based solution did. RB#26120
- Loading branch information
Marcin Babij
committed
Sep 23, 2021
1 parent
ac53fbe
commit b184bd3
Showing
32 changed files
with
1,876 additions
and
1,017 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,91 @@ | ||
SET @start_global_value = @@GLOBAL.innodb_open_files; | ||
SELECT @start_global_value; | ||
@start_global_value | ||
4000 | ||
SHOW STATUS LIKE 'SYSTEM_VARIABLES_ADMIN'; | ||
Variable_name Value | ||
SELECT * FROM performance_schema.global_status WHERE variable_name like 'SYSTEM_VARIABLES_ADMIN'; | ||
VARIABLE_NAME VARIABLE_VALUE | ||
SELECT @@innodb_open_files; | ||
@@innodb_open_files | ||
4000 | ||
CREATE USER 'user1'@'localhost' IDENTIFIED BY '123'; | ||
GRANT SUPER ON *.* to 'user1'@'localhost'; | ||
Warnings: | ||
Warning 1287 The SUPER privilege identifier is deprecated | ||
SHOW GRANTS FOR 'user1'@'localhost'; | ||
Grants for user1@localhost | ||
GRANT SUPER ON *.* TO `user1`@`localhost` | ||
SELECT * FROM performance_schema.global_status WHERE variable_name like 'SYSTEM_VARIABLES_ADMIN'; | ||
VARIABLE_NAME VARIABLE_VALUE | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
4000 | ||
SELECT innodb_set_open_files_limit(10); | ||
innodb_set_open_files_limit(10) | ||
10 | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
10 | ||
DROP USER 'user1'@'localhost'; | ||
CREATE USER 'user1'@'localhost' IDENTIFIED BY '123'; | ||
GRANT SYSTEM_VARIABLES_ADMIN ON *.* to 'user1'@'localhost'; | ||
SHOW GRANTS FOR 'user1'@'localhost'; | ||
Grants for user1@localhost | ||
GRANT USAGE ON *.* TO `user1`@`localhost` | ||
GRANT SYSTEM_VARIABLES_ADMIN ON *.* TO `user1`@`localhost` | ||
SELECT * FROM performance_schema.global_status WHERE variable_name like 'SYSTEM_VARIABLES_ADMIN'; | ||
VARIABLE_NAME VARIABLE_VALUE | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
10 | ||
SELECT innodb_set_open_files_limit(2147483647); | ||
innodb_set_open_files_limit(2147483647) | ||
2147483647 | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
2147483647 | ||
DROP USER 'user1'@'localhost'; | ||
CREATE USER 'user1'@'localhost' IDENTIFIED BY '123'; | ||
GRANT SUPER ON *.* to 'user1'@'localhost'; | ||
Warnings: | ||
Warning 1287 The SUPER privilege identifier is deprecated | ||
GRANT SYSTEM_VARIABLES_ADMIN ON *.* to 'user1'@'localhost'; | ||
SHOW GRANTS FOR 'user1'@'localhost'; | ||
Grants for user1@localhost | ||
GRANT SUPER ON *.* TO `user1`@`localhost` | ||
GRANT SYSTEM_VARIABLES_ADMIN ON *.* TO `user1`@`localhost` | ||
SELECT * FROM performance_schema.global_status WHERE variable_name like 'SYSTEM_VARIABLES_ADMIN'; | ||
VARIABLE_NAME VARIABLE_VALUE | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
2147483647 | ||
SELECT innodb_set_open_files_limit(4000); | ||
innodb_set_open_files_limit(4000) | ||
4000 | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
4000 | ||
DROP USER 'user1'@'localhost'; | ||
CREATE USER 'user1'@'localhost' IDENTIFIED BY '123'; | ||
GRANT CREATE, SELECT, UPDATE, DELETE ON *.* TO 'user1'@'localhost'; | ||
SHOW GRANTS FOR 'user1'@'localhost'; | ||
Grants for user1@localhost | ||
GRANT SELECT, UPDATE, DELETE, CREATE ON *.* TO `user1`@`localhost` | ||
SELECT * FROM performance_schema.global_status WHERE variable_name like 'SYSTEM_VARIABLES_ADMIN'; | ||
VARIABLE_NAME VARIABLE_VALUE | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
4000 | ||
SELECT innodb_set_open_files_limit(6000); | ||
ERROR 42000: Access denied; you need (at least one of) the SUPER or SYSTEM_VARIABLES_ADMIN privilege(s) for this operation | ||
SELECT @@innodb_open_files ; | ||
@@innodb_open_files | ||
4000 | ||
DROP USER 'user1'@'localhost'; | ||
SELECT innodb_set_open_files_limit(@start_global_value); | ||
innodb_set_open_files_limit(@start_global_value) | ||
4000 | ||
SELECT @@GLOBAL.innodb_open_files; | ||
@@GLOBAL.innodb_open_files | ||
4000 |
87 changes: 87 additions & 0 deletions
87
mysql-test/suite/innodb/r/innodb_set_open_files_limit.result
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,87 @@ | ||
CREATE UNDO TABLESPACE undo_009 ADD DATAFILE 'undo_009.ibu'; | ||
CREATE UNDO TABLESPACE undo_008 ADD DATAFILE 'undo_008.ibu'; | ||
CREATE UNDO TABLESPACE undo_007 ADD DATAFILE 'undo_007.ibu'; | ||
CREATE UNDO TABLESPACE undo_006 ADD DATAFILE 'undo_006.ibu'; | ||
CREATE UNDO TABLESPACE undo_005 ADD DATAFILE 'undo_005.ibu'; | ||
CREATE UNDO TABLESPACE undo_004 ADD DATAFILE 'undo_004.ibu'; | ||
CREATE UNDO TABLESPACE undo_003 ADD DATAFILE 'undo_003.ibu'; | ||
CREATE UNDO TABLESPACE undo_002 ADD DATAFILE 'undo_002.ibu'; | ||
CREATE UNDO TABLESPACE undo_001 ADD DATAFILE 'undo_001.ibu'; | ||
CREATE UNDO TABLESPACE undo_000 ADD DATAFILE 'undo_000.ibu'; | ||
SET @start_global_value = @@GLOBAL.innodb_open_files; | ||
SELECT @start_global_value; | ||
@start_global_value | ||
4000 | ||
SELECT innodb_set_open_files_limit(1.1); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid first argument type. | ||
SELECT innodb_set_open_files_limit(1e1); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid first argument type. | ||
SELECT innodb_set_open_files_limit('AUTO'); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid first argument type. | ||
SELECT innodb_set_open_files_limit(); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid number of arguments. | ||
SELECT innodb_set_open_files_limit(1, 2); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid number of arguments. | ||
SELECT innodb_set_open_files_limit(1, 2, 3, 4); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; Invalid number of arguments. | ||
SELECT innodb_set_open_files_limit(NULL); | ||
ERROR HY000: Can't initialize function 'innodb_set_open_files_limit'; First argument must not be null. | ||
select @@global.innodb_open_files; | ||
@@global.innodb_open_files | ||
4000 | ||
SELECT innodb_set_open_files_limit(-3); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be smaller than 10. | ||
SELECT innodb_set_open_files_limit(1); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be smaller than 10. | ||
SELECT innodb_set_open_files_limit(3); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be smaller than 10. | ||
SELECT innodb_set_open_files_limit(9); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be smaller than 10. | ||
SELECT innodb_set_open_files_limit(2147483648); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be larger than 2147483647. | ||
SELECT innodb_set_open_files_limit(12342147483647); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. New limit value can't be larger than 2147483647. | ||
SELECT innodb_set_open_files_limit(10); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. Cannot update innodb_open_files to this value. Not enough files could be closed in last 5 seconds or a number of files that cannot be closed would exceed 90% of the new limit. Consider setting it above 17. | ||
SELECT innodb_set_open_files_limit(16); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. Cannot update innodb_open_files to this value. Not enough files could be closed in last 5 seconds or a number of files that cannot be closed would exceed 90% of the new limit. Consider setting it above 17. | ||
SELECT innodb_set_open_files_limit(17); | ||
innodb_set_open_files_limit(17) | ||
17 | ||
SELECT innodb_set_open_files_limit(16); | ||
ERROR HY000: Incorrect arguments to innodb_set_open_files_limit. Cannot update innodb_open_files to this value. Not enough files could be closed in last 5 seconds or a number of files that cannot be closed would exceed 90% of the new limit. Consider setting it above 17. | ||
SELECT innodb_set_open_files_limit(2147483647); | ||
innodb_set_open_files_limit(2147483647) | ||
2147483647 | ||
SELECT innodb_set_open_files_limit(4000); | ||
innodb_set_open_files_limit(4000) | ||
4000 | ||
SELECT innodb_set_open_files_limit(150); | ||
innodb_set_open_files_limit(150) | ||
150 | ||
ALTER UNDO TABLESPACE undo_009 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_008 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_007 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_006 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_005 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_004 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_003 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_002 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_001 SET INACTIVE; | ||
ALTER UNDO TABLESPACE undo_000 SET INACTIVE; | ||
DROP UNDO TABLESPACE undo_009; | ||
DROP UNDO TABLESPACE undo_008; | ||
DROP UNDO TABLESPACE undo_007; | ||
DROP UNDO TABLESPACE undo_006; | ||
DROP UNDO TABLESPACE undo_005; | ||
DROP UNDO TABLESPACE undo_004; | ||
DROP UNDO TABLESPACE undo_003; | ||
DROP UNDO TABLESPACE undo_002; | ||
DROP UNDO TABLESPACE undo_001; | ||
DROP UNDO TABLESPACE undo_000; | ||
SELECT innodb_set_open_files_limit(@start_global_value); | ||
innodb_set_open_files_limit(@start_global_value) | ||
4000 | ||
SELECT @@GLOBAL.innodb_open_files; | ||
@@GLOBAL.innodb_open_files | ||
4000 |
Oops, something went wrong.