Feature Request: Option to avoid running CHANGE MASTER TO on the promoted slave #15

Closed
seattlegaucho opened this Issue Oct 6, 2011 · 10 comments

3 participants

@seattlegaucho

Hi there, me again.

I'd like to know if it would be possible to add an option to avoid running the CHANGE MASTER TO sequence on the promoted slave. We have some master-master use cases where it is desirable to be able to resume replication once the failed master comes back online.

We have this setup ( A <-> B, A -> C & B -> D):

+----+            +----+
|  A |  < --- >   |  B |
+----+            +----+
   V                 V
+----+            +----+
|  C |            |  D |
+----+            +----+

Which upon failure would become (A -> C, A -> D and B is offline):

+----+          +----+
|  A |          |  X |
+----+          +----+
   V        \  
+----+          +----+
|  C |          |  D |
+----+          +----+

What we'd like to do, when possible, as we bring B back online is: (A <-> B, A -> C & A -> D)

+----+            +----+
|  A |  < --- >   |  B |
+----+            +----+
   V        \  
+----+          +----+
|  C |          |  D |
+----+          +----+

Right now we can replicate from A -> B, but we need to do a full CHANGE MASTER TO ... on A to start replication back from B. I'd like to be able to add an option to the MHA config file, so that step is not needed. The option should be OFF by default.

Thanks for your efforts on the project.

Gerry
I hope the ASCII art helps to understand what I'm trying to do.

Cheers,
G

@yoshinorim
Owner

Do you mean you want an option for MHA not to execute RESET SLAVE on the new master (promoted slave)? Currently MHA executes RESET SLAVE (ALL) on the new master when all failover procedure succeeds. If MHA skips this process, the new master still replicates from the dead master, so when you recover the crashed master replication may be able to continue (without sync-binlog=1, replication might fail because some necessary binlogs may be lost on the crashed master).
Then it is easy to add. (i.e. adding a new parameter skip_reset_slave in MHA config file)

@seattlegaucho

Hi,
I believe there is a mixup in the middle, but I need to reset my test environment to present the example. It'll be the easiest way. Please give me a couple of days. Thx.

@seattlegaucho

Hi,

Finally got time to reproduce what I'd like.

I start with the following replication topology: 003 <-> 004 | 003 -> 005 | 005 -> 006. MHA agent runs on 006.

The pertinent SHOW SLAVE STATUS lines in 003, 004 and 005 are:
003:
Master_Host: xdc-tst-mysql-004
Master_User: repl
Master_Port: 3306
004:
Master_Host: xdc-tst-mysql-003
Master_User: repl
Master_Port: 3306
005:
Master_Host: xdc-tst-mysql-003
Master_User: repl
Master_Port: 3306

On 003 I force a crash and let MHA do it's magic. Then I bring 003 back and start replication. The resulting replication topology now is: 004 -> 003 | 004 -> 005 | 005 -> 006. The same 3 lines on the 3 servers now look like:
003:
Master_Host: xdc-tst-mysql-004
Master_User: repl
Master_Port: 3306
004:
Master_Host: xdc-tst-mysql-003
Master_User: test
Master_Port: 3306
005:
Master_Host: xdc-tst-mysql-004
Master_User: repl
Master_Port: 3306

The line in bold illustrates the changes in 004 that prevent me from just re-starting the slave to reestablish master-master. I realize this is not the best default behavior, but it's highly desirable in our environment. For example, in a controlled fail over.

I hope this helps to understand what we're looking for.

Cheers,
G

@yoshinorim
Owner

I added a new parameter "skip_reset_slave". If this option is set, MHA does not execute RESET SLAVE on the new master.
Committed code is here: e9212bf
And I've pushed to the main tree. Please test it and give me your feedbacks.

@yoshinorim
Owner

Closing this issue since current implementation has passed all test cases. Please feel free to reopen if you have any issues.

@yoshinorim yoshinorim closed this Nov 7, 2011
@seattlegaucho

I apologize for not getting back to you, but it has been crazy. Thank you very much for your efforts, keep the good work and contributions.
Gerry

@brk0v

In case when we use "skip_reset_slave" and multi-master configuration:

M1 = M2
| \
S1 S2

We can't doing any work on the passive master (M2) (e.g. adding indexes, etc) because we will get conflicts during failover on the step applying diff binary relay logs. Am I right?

@yoshinorim
Owner

Regardless of skip_reset_slave settings, I recommend stopping MHA when doing maintenance(DDL etc) on a passive master. After everything is finished, restarting MHA again. Stopping/starting MHA is possible without stopping MySQL replication.

@brk0v

When MHA starts it execute FLASH LOGS?
If I right understanding how MHA works -- it uses relay's log "end_log_pos" as a Global Transaction ID.
So there are different transactions in the Masters and Slaves logs. And during failover there will be conflicts.

@yoshinorim
Owner

MHA checks master's binary log name and end log pos written in relay logs. MHA uses master's binary log name and end log pos as a global transaction id. All slaves should have consistent information about master's binary log name and end log pos, so as long as you don't update slaves (or don't execute replication-unsafe queries on master) no conflicts will happen. In depth algorithm is described here: http://www.slideshare.net/matsunobu/automated-master-failover (though it's a bit old).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment