Skip to content
This repository has been archived by the owner on May 9, 2023. It is now read-only.

Make possible to create backup from master and start slave from this backup for --test_mode #144

Closed
ShahriyarR opened this issue Oct 26, 2017 · 7 comments
Closed

Make possible to create backup from master and start slave from this backup for --test_mode #144

ShahriyarR opened this issue Oct 26, 2017 · 7 comments
Labels
easy_to_fix enhancement for_beginners help wanted

Comments

@ShahriyarR
Copy link
Owner

This feature is needed for --test_mode.
The idea is, to take backup, then prepare this backup and start slave server if it is started properly + check consistency between master and slave.
ShahriyarR added a commit that referenced this issue Oct 26, 2017
@ShahriyarR
Preliminary fix for issue #144
81d24c4
@ShahriyarR
Copy link
Owner Author

Related issue:
Percona-QA/percona-qa#241

ShahriyarR added a commit that referenced this issue Oct 26, 2017
@ShahriyarR
Preliminary fix for issue #144
ef25485
@ShahriyarR
Copy link
Owner Author 

2017-10-30 11:38:10 ERROR    pt-table-checksum command failed
2017-10-30 11:38:10 ERROR    Diffs cannot be detected because no slaves were found.  Please read the --recursion-method documentation for information.
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
10-30T11:37:49      0      0        0       1       0   0.093 mysql.columns_priv
10-30T11:37:49      0      0        2       1       0   0.124 mysql.db
10-30T11:37:49      0      0        0       1       0   0.088 mysql.event
10-30T11:37:49      0      0        0       1       0   0.090 mysql.func
10-30T11:37:50      0      0       40       4       0   0.413 mysql.help_category
10-30T11:37:50      0      0      615       4       0   0.406 mysql.help_keyword
10-30T11:37:50      0      0     1253       1       0   0.106 mysql.help_relation
10-30T11:37:50      0      0      585       1       0   0.096 mysql.help_topic
10-30T11:37:50      0      0        0       1       0   0.090 mysql.ndb_binlog_index

@ShahriyarR
Copy link
Owner Author

The last thing in this feature is to make slave to work :)

2017-10-30 21:21:43 DEBUG    <pid.PidFile object at 0x7f6f456c0408> closing pidfile: /tmp/MySQL-AutoXtraBackup/autoxtrabackup.pid
Traceback (most recent call last):
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/bin/autoxtrabackup", line 11, in <module>
    load_entry_point('mysql-autoxtrabackup==1.4.9', 'console_scripts', 'autoxtrabackup')()
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/click-6.7-py3.5.egg/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.4.9-py3.5.egg/autoxtrabackup.py", line 169, in all_procedure
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.4.9-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 257, in wipe_backup_prepare_copyback
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.4.9-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 173, in run_change_master
  File "/home/shahriyar.rzaev/virtualenvs/py_3_5_3_autoxtrabackup/lib/python3.5/site-packages/mysql_autoxtrabackup-1.4.9-py3.5.egg/prepare_env_test_mode/runner_test_mode.py", line 110, in check_slave_status
RuntimeError: Slave_IO_Running is not Yes
2017-10-30 21:21:43 DEBUG    <pid.PidFile object at 0x7f6f456c0408> closing pidfile: /tmp/MySQL-AutoXtraBackup/autoxtrabackup.pid

@ShahriyarR
Copy link
Owner Author

I think I should run SET GLOBAL gtid_purged= prior running CHANGE MASTER

ShahriyarR added a commit that referenced this issue Oct 31, 2017
@ShahriyarR
Finalizing the #144 implementation
d51a174
@ShahriyarR
Copy link
Owner Author

After fighting with several issues, Now it is fully implemented.

@ShahriyarR
Copy link
Owner Author 

2017-10-31 14:50:05 DEBUG    Started to make this new server as slave...
2017-10-31 14:50:05 DEBUG    Trying to get basedir path...
2017-10-31 14:50:05 DEBUG    Could get PS basedir path returning...
2017-10-31 14:50:05 DEBUG    Trying to get mysql client connection...
2017-10-31 14:50:05 DEBUG    Could get mysql client
2017-10-31 14:50:05 DEBUG    Trying to get basedir path...
2017-10-31 14:50:05 DEBUG    Could get PS basedir path returning...
2017-10-31 14:50:05 DEBUG    Trying to get mysql client connection...
2017-10-31 14:50:05 DEBUG    Could get mysql client
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 'select @@port'
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "CREATE USER 'repl'@'%' IDENTIFIED BY 'Baku12345'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "GRANT REPLICATION SLAVE ON *.* TO 'repl'@'%'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "select user, host from mysql.user where user like ''"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "select user, host from mysql.user where user like ''"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "drop user ''@'localhost'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "drop user ''@'localhost'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "drop user ''@'qaserver-02.ci.percona.com'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e "drop user ''@'qaserver-02.ci.percona.com'"
2017-10-31 14:50:05 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 'set global gtid_purged="a3b124ec-be49-11e7-b106-002590e9b448:1-42584"'
2017-10-31 14:50:06 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e "CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='repl', MASTER_PASSWORD='Baku12345', MASTER_PORT=15196, MASTER_AUTO_POSITION=1"
2017-10-31 14:50:06 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 'start slave'
2017-10-31 14:50:26 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 'show slave status\G'
2017-10-31 14:50:26 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 'CREATE TABLE `dsns` (
                          `id` int(11) NOT NULL AUTO_INCREMENT,
                          `parent_id` int(11) DEFAULT NULL,
                          `dsn` varchar(255) NOT NULL,
                          PRIMARY KEY (`id`))
                        '
2017-10-31 14:50:26 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/sock0.sock --force test -e 'select @@port'
2017-10-31 14:50:26 DEBUG    Running -> /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/bin/mysql -A -uroot -S/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/socket.sock --force test -e 'insert into dsns(id, parent_id, dsn) values(30, 84, "h=127.0.0.1,u=root,P=58135")'
2017-10-31 14:50:36 DEBUG    Trying to get basedir path...
2017-10-31 14:50:37 DEBUG    Could get PS basedir path returning...
2017-10-31 14:50:37 DEBUG    Trying to get socket file...
2017-10-31 14:51:32 DEBUG    pt-table-checksum succeeded on master

@ShahriyarR ShahriyarR mentioned this issue Oct 31, 2017
Merged
ShahriyarR pushed a commit that referenced this issue Oct 31, 2017
Bugfix issue144 (#146)
17ab839 
* Preliminary fix for issue #144

* Fixing option passing

* Opening the config file in w+ mode

* [ERROR] --gtid-mode=ON or UPGRADE_STEP_1 or UPGRADE_STEP_2 requires --log-bin and --log-slave-updates

* Preliminary fix for issue #144

* Added slave_count option to config generator

* Changd start_mysql_func() to accept start tool as well

* First test run

* TypeError: prepare_start_slave_options() missing 2 required positional arguments: 'basedir' and 'slave_number'

* TypeError: prepare_start_slave_options() missing 1 required positional argument: 'slave_number'

* [ERROR] The socket file path is too long (> 107): /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/node0/slave_socket.sock

* Now trying to fix slave start up problem

* Added few tests and also pt-table-checksum run

* Added assert statement

* Trying to make started server to be slave

* Added unit test test_run_change_master()

* Added returns to method

* Trying to fix sql error

* Trying to fix sql error again

* Trying to escape characters

* Escaping some characters here also

* Checking again for failed thing

* Passing master port for slave's change master statement

* Fixed port checking call

* Found the value of port

* Passing port number to change master statement

* Trying to pass proper create statement

* Passed true values for master server

* Fix the string

* Successfully made the started node as slave

* Added run for pt-table-checksum tool on master

* Trying to run pt-table-checksum on master server

* TypeError: 'int' object is not iterable

* Fixing some dependent tasks

* Passing start slave command to slave server

* Trying to fix this weird situation on returning True from method

* Check if prepare methods also returning something

* returning True from functions

* Changed the structure of run_change_master()

* Raising RuntimeError here

* Added new check_slave_output() static method

* Fixing open issue for stop_node{} file

* Added sleep(10)

* Fixing small things

* Giving some sleep(10)

* Added new debug message

* Added few mor fixes here

* Added drop_blank_mysql_users() static method

* Trying to fix failed test

* Again trying to fix failed test sql syntax error

* Next attempt

* next attempt n2

* seems to be final attempt

* Finalized dropping blank users

* Added final docstrings and other checks

* Changed binlog_format to row

* Added get_gtid_address() static method

* Running gtid_purged

* RuntimeError: Failed to run SQL command -> ERROR 1193 (HY000) at line 1: Unknown system variable 'gitd_purged'

* Increased the sleep time

* Added pytest for check_slave_status()

* Could find the fail reason. it was due to not using lstrip()

* Changed assert to check against None

* Added new tests and create_dsns_table()

* Creating table

* Added pytest test_populate_dsns_table()

* Fix for failed mysql client command

* for the right syntax to use near 's=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-l' at line 1

* Another sql syntax error

* Adding some fix for sql syntax

* Again trying to fix sql syntax issue

* Added --recursion-method

* Added pytest test_run_pt_table_checksum()

* Using localhost instead of ip

* removed p=

* Passing slave port

* changed to 127.0.0.1

* Finalizing the #144 implementation
@ShahriyarR
Copy link
Owner Author

Merged to release_v1.5.0.
Closing.

ShahriyarR pushed a commit that referenced this issue Nov 16, 2017
Release v1.5.0 (#154)
a995d28 
* Bugfix issue144 (#146)

* Preliminary fix for issue #144

* Fixing option passing

* Opening the config file in w+ mode

* [ERROR] --gtid-mode=ON or UPGRADE_STEP_1 or UPGRADE_STEP_2 requires --log-bin and --log-slave-updates

* Preliminary fix for issue #144

* Added slave_count option to config generator

* Changd start_mysql_func() to accept start tool as well

* First test run

* TypeError: prepare_start_slave_options() missing 2 required positional arguments: 'basedir' and 'slave_number'

* TypeError: prepare_start_slave_options() missing 1 required positional argument: 'slave_number'

* [ERROR] The socket file path is too long (> 107): /home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-linux-x86_64-debug/node0/slave_socket.sock

* Now trying to fix slave start up problem

* Added few tests and also pt-table-checksum run

* Added assert statement

* Trying to make started server to be slave

* Added unit test test_run_change_master()

* Added returns to method

* Trying to fix sql error

* Trying to fix sql error again

* Trying to escape characters

* Escaping some characters here also

* Checking again for failed thing

* Passing master port for slave's change master statement

* Fixed port checking call

* Found the value of port

* Passing port number to change master statement

* Trying to pass proper create statement

* Passed true values for master server

* Fix the string

* Successfully made the started node as slave

* Added run for pt-table-checksum tool on master

* Trying to run pt-table-checksum on master server

* TypeError: 'int' object is not iterable

* Fixing some dependent tasks

* Passing start slave command to slave server

* Trying to fix this weird situation on returning True from method

* Check if prepare methods also returning something

* returning True from functions

* Changed the structure of run_change_master()

* Raising RuntimeError here

* Added new check_slave_output() static method

* Fixing open issue for stop_node{} file

* Added sleep(10)

* Fixing small things

* Giving some sleep(10)

* Added new debug message

* Added few mor fixes here

* Added drop_blank_mysql_users() static method

* Trying to fix failed test

* Again trying to fix failed test sql syntax error

* Next attempt

* next attempt n2

* seems to be final attempt

* Finalized dropping blank users

* Added final docstrings and other checks

* Changed binlog_format to row

* Added get_gtid_address() static method

* Running gtid_purged

* RuntimeError: Failed to run SQL command -> ERROR 1193 (HY000) at line 1: Unknown system variable 'gitd_purged'

* Increased the sleep time

* Added pytest for check_slave_status()

* Could find the fail reason. it was due to not using lstrip()

* Changed assert to check against None

* Added new tests and create_dsns_table()

* Creating table

* Added pytest test_populate_dsns_table()

* Fix for failed mysql client command

* for the right syntax to use near 's=/home/shahriyar.rzaev/XB_TEST/server_dir/PS231017-percona-server-5.6.37-82.2-l' at line 1

* Another sql syntax error

* Adding some fix for sql syntax

* Again trying to fix sql syntax issue

* Added --recursion-method

* Added pytest test_run_pt_table_checksum()

* Using localhost instead of ip

* removed p=

* Passing slave port

* changed to 127.0.0.1

* Finalizing the #144 implementation

* Bugfix issue147 (#148)

* Adding log file support

* Adding handlers.RotateFileHandler usage

* Bugfix issue150 (#153)

* Preliminary fix for issue #150 changed CheckEnv class

* Pushing some other changes now.

* Added much more things here

* Finalizing the issue #150

* Preliminary fix for issue #152 (#155)

* Preliminary fix for issue #151 (#156)

* Bugfix issue157 (#159)

* Preliminary fix for issue #157

* Added shutdown call for slave

* Bugfix issue158 (#161)

* Preliminary fix for issue #158

* Trying to fix, failed check

* Finalized the fix for issue #158

* Using try/except to catch exceptions

* Removed redundant logger.error messages

* Printing the RuntimError name

* Bugfix issue162 (#165)

* Preliminary fix for issue #162

* Checking failed thing

* Commenting for test purposes

* removed comments

* Commenting the function

* Uncommented

* Checking something here

* Passing ctx

* Trying to figure out something

* Removed help and version

* Added few other things

* Printed message

* Calling click.help_option()

* Ugly way of implementing --help

* reverted the changes

* Added print_help() wrapper

* Changed print_help

* Added some code

* Now it must be fixed

* changed a bit

* Eliminated options with default values

* using show_default inside option

* Changing some options

* Finalizing the issue fix

* Bugfix issue166 (#167)

* Preliminary fix for issue #166

* changed setup.py

* Added file name

* Changed back

* Changed a bit

* Changed to print() statement

* Using sys.exit()

* Bugfix issue168 (#169)

* Preliminary fix for issue #168

* The fix for partial.py file

* Fixing partial.py issue

* preliminary fix for issue #170 (#173)

* Bugfix issue163 (#174)

* Preliminary fix for issue #163

* TypeError: add_tag() missing 1 required positional argument: 'tag_string'

* Changed the type column

* changed the open mode

* Changed full_backup() and added pytest test

* Changed inc_backup() and added pytest test

* prepared test_inc_backup() test

* Added the tag passing facilities to autoxtrabackup

* Changed the sequence of input for backup_tags.txt

* Removing quotes for status value

* Bugfix issue164 (#175)

* Preliminary fix for issue #164

* Calling from CLI

* Trying to print the output in a good way

* Adding some extra strings

* Trying to fix something

* Hard coding the output

* Adding error message if there is no backup_tags.txt

* Changed the print message a bit

* Bugfix issue171 (#178)

* Preliminary fix for issue #171

* passing basedir path

* Failed with PS 5.6

* Added run_sql_stetement()

* Passing correct SQL

* Trying to figure out the fail reason

* passing --keyring-file-data to xtrabackup

* Added flush tables statement

* Added sleep statement here

* Bugfix issue172 (#180)

* Preliminary fix for issue #172

* Get the output for SQL

* Adding optimize table running

* Disabling the compression support due to bug 1641745

* Bugfix issue176 (#181)

* Preliminary fix for issue #176

* Passing tag to Prepare class from autoxtrabackup

* Fix for issue related to iteration

* Preliminary fix for issue #182 (#183)

* Bugfix issue185 (#186)

* Preliminary fix for issue #185

* Finalizing the fix

* Bugfix issue188 (#189)

* Preliminary fix for issue #188

* Again updating VagrantFile

* Introducing huge changes

* Committing some few changes as well

* decreasing the --parallel=10 to 1 for PS 5.7

* Committing few other changes

* Committing another change

* Fixing config generation

* Fixing path name

* Trying to finalize bug fix for issue #188

* changing to vagrant:vagrant

* Fixing small issue

* Fixing failed sql statement

* Now finished for sure :)

* Added another sleep(5)

* Increased random number range

* Finalized, can be merged to release_v_1.5.0

* Bugfix issue190 (#191)

* Preliminary fix for issue #190

* Fixing missing thing

* Disabling this features

* Preliminary fix for issue #149 (#160)

* Preliminary fix for issue #149

* Final review for v1.5.0 release

* Version bump
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
easy_to_fix enhancement for_beginners help wanted
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ShahriyarR