Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nixos/mysql: add settings and configFile option #81940

Merged
merged 1 commit into from Mar 10, 2020

Conversation

@aanderse
Copy link
Contributor

aanderse commented Mar 7, 2020

Motivation for this change
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc @thorstenweber83

Copy link
Member

Infinisil left a comment

Nice! Perfect use case for the RFC.

nixos/modules/services/databases/mysql.nix Outdated Show resolved Hide resolved
nixos/modules/services/databases/mysql.nix Outdated Show resolved Hide resolved
@aanderse aanderse force-pushed the aanderse:mysql-config branch from 0c205ef to 6ae6712 Mar 7, 2020
@aanderse aanderse marked this pull request as ready for review Mar 7, 2020
@aanderse aanderse requested a review from florianjacob Mar 7, 2020
@aanderse aanderse force-pushed the aanderse:mysql-config branch from 6ae6712 to 983f5dd Mar 7, 2020
@Izorkin
Copy link
Contributor

Izorkin commented Mar 7, 2020

Not working this configuration:

    ...
    mysqldump = {
      quick;
      max_allowed_packet = "16M";
    };
    ...

error:

syntax error, unexpected ';', at /home/user/works/nixops/deploy/hetzner-all/pve2-test/mysql.nix:57:45

if use package mariadb.server builded withoutClient = true; error:

2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup[96574]: /nix/store/2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup: line 30: /nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql: No such file or directory

After this update poorly view the current configuration /etc/my.cnf.
Example after:

[client]
default_character_set=utf8mb4

[myisamchk]
key_buffer_size=256M
myisam_sort_buffer_size=256M
read_buffer_size=2M

[mysql]
default_character_set=utf8mb4

[mysqld]
aria_log_dir_path=/var/data/db/mysql/dblogs
aria_sort_buffer_size=2M
back_log=2048
bind_address=0.0.0.0
binlog_cache_size=2M
binlog_format=ROW
character_set_server=utf8mb4
collation_server=utf8mb4_unicode_ci
datadir=/var/data/db/mysql/db
default_storage_engine=InnoDB
expire_logs_days=7
innodb_autoinc_lock_mode=1
innodb_buffer_pool_dump_at_shutdown=ON
innodb_buffer_pool_load_at_startup=ON
innodb_buffer_pool_size=256M
innodb_compression_algorithm=lz4
innodb_compression_default=ON
innodb_data_file_path=ibdata1:12M:autoextend
innodb_data_home_dir=/var/data/db/mysql/db
innodb_defragment=ON
innodb_file_per_table=ON
innodb_flush_log_at_trx_commit=2
innodb_flush_method=O_DIRECT
innodb_log_buffer_size=16M
innodb_log_checksums=ON
innodb_log_compressed_pages=ON
innodb_log_file_size=128M
innodb_log_group_home_dir=/var/data/db/mysql/dblogs
innodb_max_dirty_pages_pct=5
innodb_sort_buffer_size=2M
innodb_temp_data_file_path=ibtmp1:12M:autoextend
innodb_thread_concurrency=16
interactive_timeout=3600
join_buffer_size=256K
key_buffer_size=16M
key_cache_division_limit=70
log_bin=/var/data/db/mysql/dblogs/mysql_bin.log
log_bin_index=/var/data/db/mysql/dblogs/mysql_bin.index
log_error=/var/data/db/logs/mysql_error.log
log_warnings=2
long_query_time=0.500000
max_allowed_packet=32M
max_binlog_cache_size=4G
max_binlog_size=256M
max_connect_errors=10240
max_connections=1024
max_heap_table_size=1024M
max_user_connections=128
myisam_sort_buffer_size=2M
performance_schema=OFF
plugin-load-add=ha_rocksdb.so;ha_tokudb.so
port=3306
query_cache_limit=2M
query_cache_size=32M
query_cache_type=ON
read_buffer_size=2M
read_rnd_buffer_size=2M
relay_log=/var/data/db/mysql/dblogs/mysql_relay_bin.log
rocksdb_datadir=/var/data/db/mysql/db_rocksdb
secure_auth=ON
skip_name_resolve=ON
slow_query_log=ON
slow_query_log_file=/var/data/db/logs/mysql_slow.log
sort_buffer_size=2M
symbolic_links=OFF
sync_binlog=0
table_definition_cache=4096
table_open_cache=2048
thread_cache_size=64
thread_handling=pool-of-threads
thread_pool_size=8
tmp_table_size=1024M
tokudb_data_dir=/var/data/db/mysql/db_tokudb
tokudb_dir_per_db=ON
tokudb_log_dir=/var/data/db/mysql/dblogs
transaction_isolation=READ-COMMITTED
wait_timeout=3600

[mysqld_safe]
collation_server=utf8mb4_unicode_ci
default_character_set=utf8mb4
log_error=/var/data/db/logs/mysql_error.log
malloc-lib=/nix/store/k78fbnb3167dwiryiv12fanaqp9j575g-jemalloc-4.5.0/lib/libjemalloc.so

Before:

[mysqld]
port = 3306
datadir = /var/data/db/mysql/db

### Paths
datadir = /var/data/db/mysql/db
innodb_data_home_dir = /var/data/db/mysql/db
innodb_log_group_home_dir = /var/data/db/mysql/dblogs
aria_log_dir_path = /var/data/db/mysql/dblogs
innodb_data_file_path = ibdata1:12M:autoextend
innodb_temp_data_file_path = ibtmp1:12M:autoextend

### General options
character_set_server = utf8mb4
collation_server = utf8mb4_unicode_ci
default_storage_engine = InnoDB
innodb_autoinc_lock_mode = 1
innodb_defragment = ON
innodb_compression_algorithm = lz4
innodb_compression_default = ON
innodb_file_per_table = ON
innodb_flush_method = O_DIRECT
innodb_flush_log_at_trx_commit = 2
innodb_max_dirty_pages_pct = 5
innodb_buffer_pool_load_at_startup = ON
innodb_buffer_pool_dump_at_shutdown = ON
performance_schema = OFF
symbolic_links = OFF
transaction_isolation = READ-COMMITTED

### Connections
bind_address = 0.0.0.0
back_log = 2048
max_allowed_packet = 32M
max_connect_errors = 10240
max_connections = 1024
max_user_connections = 128
secure_auth = ON
skip_name_resolve = ON

### Threads
innodb_thread_concurrency = 16
thread_cache_size = 64
thread_handling = pool-of-threads
thread_pool_size = 8

### Timeout
wait_timeout = 3600
interactive_timeout = 3600

### Logs
innodb_log_file_size = 128M
innodb_log_checksums = ON
innodb_log_compressed_pages = ON
innodb_log_buffer_size = 16M
log_error = /var/data/db/logs/mysql_error.log
log_warnings = 2
long_query_time = 0.500000
slow_query_log_file = /var/data/db/logs/mysql_slow.log
slow_query_log = ON

### Binlogs
binlog_format = ROW
expire_logs_days = 7
log_bin = /var/data/db/mysql/dblogs/mysql_bin.log
log_bin_index = /var/data/db/mysql/dblogs/mysql_bin.index
relay_log = /var/data/db/mysql/dblogs/mysql_relay_bin.log
binlog_cache_size = 2M
max_binlog_size = 256M
max_binlog_cache_size = 4G
sync_binlog = 0

### Buffers
aria_sort_buffer_size = 2M
innodb_sort_buffer_size = 2M
myisam_sort_buffer_size = 2M
sort_buffer_size = 2M
join_buffer_size = 256K
key_buffer_size = 16M
read_buffer_size = 2M
read_rnd_buffer_size = 2M
innodb_buffer_pool_size = 256M

### Caches
query_cache_type = ON
query_cache_limit = 2M
query_cache_size = 32M
key_cache_division_limit = 70
table_definition_cache = 4096
table_open_cache = 2048
max_heap_table_size = 1024M
tmp_table_size = 1024M

### zfs
innodb_doublewrite = OFF
innodb_compression_algorithm = none
innodb_compression_default = OFF
innodb_use_atomic_writes = OFF
innodb_use_native_aio = OFF

### zfs, checksums
#aria_page_checksum = OFF
#innodb_checksums = OFF
#innodb_checksum_algorithm = NONE
#innodb_log_checksums = OFF
#binlog_checksum = NONE

### RocksDB
plugin-load-add = ha_rocksdb.so
rocksdb_datadir = /var/data/db/mysql/db_rocksdb

### TokuDB
plugin-load-add = ha_tokudb.so
tokudb_dir_per_db = ON
tokudb_data_dir = /var/data/db/mysql/db_tokudb
tokudb_log_dir = /var/data/db/mysql/dblogs

[client]
default_character_set = utf8mb4

[mysql]
default_character_set = utf8mb4

[mysqld_safe]
default_character_set = utf8mb4
collation_server = utf8mb4_unicode_ci
log_error = /var/data/db/logs/mysql_error.log

### TokuDB
malloc-lib = /nix/store/k78fbnb3167dwiryiv12fanaqp9j575g-jemalloc-4.5.0/lib/libjemalloc.so

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer_size = 256M
myisam_sort_buffer_size = 256M
read_buffer_size = 2M

@aanderse
Copy link
Contributor Author

aanderse commented Mar 7, 2020

@Izorkin thank you for your feedback and testing! You are always so reliable 😄

Not working this configuration:

    ...
    mysqldump = {
      quick;
      max_allowed_packet = "16M";
    };
    ...

I'm under the impression this is a shorthand syntax for what should be written as:

mysqldump = {
  quick = true;
  max_allowed_packet = "16M";
};

if use package mariadb.server builded withoutClient = true; error:

2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup[96574]: /nix/store/2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup: line 30: /nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql: No such file or directory

Can you please provide /nix/store/2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup?

@Izorkin it is very true that RFC42 doesn't let you hand craft beautiful configuration files which is disappointing to you. I don't know how to address this. I think you should mention this in NixOS/rfcs#42 for a discussion point. Any comment @Infinisil?

@Infinisil
Copy link
Member

Infinisil commented Mar 7, 2020

Yeah I guess commenting in config files themselves isn't something supported by this approach, but I think that's rather minor compared to the benefits it gives. It's still possible to have comments in the Nix code at least.

@Izorkin
Copy link
Contributor

Izorkin commented Mar 7, 2020

I'm under the impression this is a shorthand syntax for what should be written as:

https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_quick

Can you please provide /nix/store/2mcwly6c6yz8rf70ikz7zjjf278g2gfi-mysql-setup?

#!/nix/store/8gy2bmpz3qawxr3g8hqhfgkf32wb0wfl-bash-4.4-p23/bin/bash -e

# Wait until the MySQL server is available for use
count=0
while [ ! -e /run/mysqld/mysqld.sock ]
do
    if [ $count -eq 30 ]
    then
        echo "Tried 30 times, giving up..."
        exit 1
    fi

    echo "MySQL daemon not yet started. Waiting for 1 second..."
    count=$((count++))
    sleep 1
done

if [ -f /tmp/mysql_init ]
then

# Execute initial script
# using toString to avoid copying the file to nix store if given as path instead of string,
# as it might contain credentials
cat /nix/store/36wz42cljwgx94kba5pgfavrkjw57g61-mysql_init.sql | /nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql -u root -N
    rm /tmp/mysql_init
fi
l /nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql
ls: невозможно получить доступ к '/nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql': No such file or directory

@Izorkin it is very true that RFC42 doesn't let you hand craft beautiful configuration files which is disappointing to you. I don't know how to address this. I think you should mention this in NixOS/rfcs#42 for a discussion point. Any comment @Infinisil?

You can process the block while maintaining the layout of the lines from the configuration?

@aanderse
Copy link
Contributor Author

aanderse commented Mar 7, 2020

@Infinisil I just realized I missed something here... any thoughts on how to deal with this? https://dev.mysql.com/doc/refman/8.0/en/option-files.html#option-file-inclusions

@aanderse
Copy link
Contributor Author

aanderse commented Mar 7, 2020

https://dev.mysql.com/doc/refman/5.7/en/mysqldump.html#option_mysqldump_quick

services.mysql.settings.mysqldump.quick = true; is compatible with this. If you test you should see this works 👍 I will add this to examples for clarification.

l /nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql
ls: невозможно получить доступ к '/nix/store/9zjrpl0blyi7rzlzsd2nx4pjjacgpybx-mariadb-server-10.3.22/bin/mysql': No such file or directory

This appears to be a bug in master, unrelated to this PR. Can you possibly test and confirm?

You can process the block while maintaining the layout of the lines from the configuration?

I don't think this is possible 😞

@aanderse aanderse force-pushed the aanderse:mysql-config branch 2 times, most recently from 1fb9af4 to 0b020a9 Mar 7, 2020
@Izorkin
Copy link
Contributor

Izorkin commented Mar 7, 2020

This appears to be a bug in master, unrelated to this PR. Can you possibly test and confirm?

Yes, this bug in master

diff --git a/nixos/modules/services/databases/mysql.nix b/nixos/modules/services/databases/mysql.nix
index 8d520b82fb5..d1f06e2b328 100644
--- a/nixos/modules/services/databases/mysql.nix
+++ b/nixos/modules/services/databases/mysql.nix
@@ -398,7 +398,7 @@ in
                         # Execute initial script
                         # using toString to avoid copying the file to nix store if given as path instead of string,
                         # as it might contain credentials
-                        cat ${toString cfg.initialScript} | ${mysql}/bin/mysql -u root -N
+                        cat ${toString cfg.initialScript} | ${mariadb.client}/bin/mysql -u root -N
                       ''}

                     rm /tmp/mysql_init

Only this option will not work with MySQL 5.7 and 8.0.

@aanderse
Copy link
Contributor Author

aanderse commented Mar 7, 2020

Only this option will not work with MySQL 5.7 and 8.0.

Thanks for confirming the bug exists in master. If you can file an issue that would be appreciated.

@aanderse aanderse changed the title nixos/mysql: add config and plugin options nixos/mysql: add settings and plugin options Mar 7, 2020
@aanderse aanderse changed the title nixos/mysql: add settings and plugin options nixos/mysql: add settings and plugins options Mar 7, 2020
Copy link
Contributor

florianjacob left a comment

Works for my configuration, and I'm always happy to throw out types.lines literals. 👍

@aanderse aanderse changed the title nixos/mysql: add settings and plugins options nixos/mysql: add settings option Mar 8, 2020
@aanderse aanderse force-pushed the aanderse:mysql-config branch from 0b020a9 to a0bcbd1 Mar 8, 2020
@aanderse aanderse force-pushed the aanderse:mysql-config branch from a0bcbd1 to cce4ce0 Mar 10, 2020
@aanderse aanderse changed the title nixos/mysql: add settings option nixos/mysql: add settings and configFile option Mar 10, 2020
@aanderse aanderse force-pushed the aanderse:mysql-config branch from cce4ce0 to cac1b6d Mar 10, 2020
@aanderse aanderse force-pushed the aanderse:mysql-config branch from cac1b6d to 08028b0 Mar 10, 2020
@aanderse
Copy link
Contributor Author

aanderse commented Mar 10, 2020

Thanks to @Infinisil I believe this is ready for final testing and merging.

nixos/modules/services/databases/mysql.nix Outdated Show resolved Hide resolved
nixos/modules/services/databases/mysql.nix Outdated Show resolved Hide resolved
@aanderse aanderse force-pushed the aanderse:mysql-config branch from 08028b0 to 641b94b Mar 10, 2020
@aanderse
Copy link
Contributor Author

aanderse commented Mar 10, 2020

@Infinisil I just realized I missed something here... any thoughts on how to deal with this? https://dev.mysql.com/doc/refman/8.0/en/option-files.html#option-file-inclusions

I realized that include directives really work against the new declarative/merging nature of a settings option, so I resolved this outstanding issue by including a configFile escape hatch.

Does anyone have any other requests before merging this?

Copy link
Member

Infinisil left a comment

With a fully declarative NixOS config these !include statements can even be replaced with a separate NixOS module that gets importsed, because the new settings option now handles composition :). So it's not entirely unjustified to not support it (though this doesn't work with includes from runtime paths). How it's done now looks good

@Infinisil
Copy link
Member

Infinisil commented Mar 10, 2020

@GrahamcOfBorg test mysql

@Infinisil Infinisil merged commit a12bea5 into NixOS:master Mar 10, 2020
15 checks passed
15 checks passed
Evaluation Performance Report Evaluator Performance Report
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
tests.mysql on aarch64-linux Success
Details
tests.mysql on x86_64-linux Success
Details
@aanderse aanderse deleted the aanderse:mysql-config branch Mar 11, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.