Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding config file functionality #370

Merged
merged 3 commits into from
Aug 3, 2021
Merged

Adding config file functionality #370

merged 3 commits into from
Aug 3, 2021

Conversation

davidducos
Copy link
Member

@davidducos davidducos commented Jun 30, 2021

This PR is adding 2 features:
1- Load the parameters from a file on group section [myloader] and [mydumper]
2- Load the set session variables from [mydumper_variables] and [myloader_variables]

@davidducos davidducos changed the title Adding config file functionallity Adding config file functionality Jun 30, 2021
@davidducos davidducos added this to the Release 0.10.9 milestone Jun 30, 2021
@davidducos
Copy link
Member Author

The variables set in the file will be the latest to load, which means that it will get precedence from the parameters list in the execution line.

@davidducos davidducos merged commit bc7ab57 into master Aug 3, 2021
@druud
Copy link

druud commented Aug 4, 2021

Generally, command line option values overrule matching config-file option values,
unless the options are of a 'strict' nature, for example to enforce policies.
IOW, config-file option values generally re-set defaults.
See also https://dev.mysql.com/doc/refman/8.0/en/option-file-options.html

For group names, maybe support a layered structure, such that [myloader_foo] is a specialization of the [myloader] settings. This also assumes that one can specify the group name suffix in the command.

@davidducos
Copy link
Member Author

I will check how hard it is going to be to change the variable loading order, tbh I didn't think that it would matter.
About the %_variables, they are in a different section to make simpler the code, as it goes through the list of variables and executes them. Actually I was thinking of adding another group like mydumper/myloader_global_variables, so you can set global variables, such as innodb_flush_log_at_trx_commit, sync_binlog, and others that improves the performance and remove them as parameters. This will reduce the lines in the code and give you more options to speed up the import/export tasks.

Take into account that we use the config file, as other tools like Proxysql or Pxc scheduler handler, which keep different groups depending what your configuring. What we are doing differently, is that we allow to keep the config for both tools: mydumper and myloader.

Please let me know if this clarifies the goal and it does or does not make sense to you

@davidducos davidducos deleted the config_file branch August 4, 2021 11:57
@davidducos davidducos linked an issue Aug 24, 2021 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment