Question: why are master/master configs unsupported?

[Roguelazer]

The tool bails out if a master-master pair is detected anywhere in the replication hierarchy (even if it's in a different branch), but I'm not actually sure why? I can see why it might cause issues with --test-on-replica (if the replica were a member of the master/master pair), but are there any other issues to be aware of? Given a topology like the following:

Is there any danger running gh-ost pointed to one of R1/R2/R3?

[shlomi-noach]

The tool allows master-master if executed with

gh-ost --allow-master-master ...

The reason this flag is required is that gh-ost cannot confirm your master-master setup is an active-passive one. For all it knows it may be an active-active one, where both masters accept writes to the migrated table, in which case it cannot reliably determine all changes are propagated to the ghost table upon cutover phase.

If you know for sure your setup is active-passive, you should be on safe grounds.

[Roguelazer]

Excellent, thanks!

Possibly worth adding to the help or a doc file for future curious folks?

James Brown,
currently mobile

On Aug 6, 2016, at 12:40, Shlomi Noach notifications@github.com wrote:

The tool allows master-master if executed with

gh-ost --allow-master-master ...
The reason this flag is required is that gh-ost cannot confirm your master-master setup is an active-passive one. For all it knows it may be an active-active one, where both masters accept writes to the migrated table, in which case it cannot reliably determine all changes are propagated to the ghost table upon cutover phase.

If you know for sure your setup is active-passive, you should be on safe grounds.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[thedrow]

What if we have an active-active multimaster topology? Does it mean we can't use this tool?

[shlomi-noach]

@thedrow I need to wrap my head around this some more; right now I will update the docs to indicate active-active is not supported. Whether it would actually work or not is what I'll be thinking of. I've worked a lot with active-active setups and there was such a variety of corner cases that I'm afraid to be missing something.

[shlomi-noach]

See #148

[shlomi-noach]

Need to fix error message to indicate --allow-master-master as a hint for successful execution

[Roguelazer]

So, one weird thing with passive/active master/master configs is that gh-ost finds the master that's "furthest" away from the current replica, which means that it sometimes applies changes to the passive side of the master/master pair.

It would be nice to either (a) be able to specify --master-address, or (b) have gh-ost look at which database is read_only and apply it on the one that isn't.

Thoughts?

[Roguelazer]

That is to say, given the following configuration:

Where only (A) is writable, if gh-ost is invoked on (D) or (E), it will attempt to do the migration on (C). This proceeds if the user has SUPER (since SUPER users can override read_only unless you're on WebScaleSQL or Percona with the super_read_only patch), but can cause data corruption if there are writes being served on (A), since there won't actually be a READ LOCK acquired on the table on the correct host during the swap.

Obviously there are other issues with this kind of setup, but it seems plausible to either look at read_only or allow the user to specify the master host.

[shlomi-noach]

@Roguelazer I've similarly found the same.

A simple solution would be: if master-master, then force user to specify active master. Operationally though, I'm not sure this is easy on the user. Sometimes the user herself is unaware who the active master is as it is obscured by virtual IP.

• outlining this solution: if gh-ost sees a master-master setup, and the connected server is not one of the two (so, a normal replica), it will bail out.
• --allow-master-master would perhaps change to something like --yes-this-really-is-the-active-master-trust-me.

Detecting read_only is a sensible approach. Operationally again, this can flip throughout the migration anyhow.

Lastly, I really want to wrap my head around whether active-active is valid. It may actually be valid -- I'm unsure as yet but a few scenarios I ran in my mind seem fine.

[Roguelazer]

I do actually like being able to run on a replica instead of the master, though, so it would be kind of a bummer if the only supported configuration for --allow-master-master required me to run gh-ost on the current master.

The biggest problem with active-active seems the be the lock to flip the tables. You'd need to get a lock on both masters before you could safely do the swap, right?

[shlomi-noach]

The one problem I'm positive of right now is that of:

M1 <--> M2 --> R1

such that R1 is the replica, M1 is the master gh-ost chooses to operate on, and while table is locked in M1, M2 still writes to the table.

Placing a lock on both masters -- sounds feasible. I'm going to spend some time thinking about this.