Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
replication: do not relay rows coming from a remote instance back to it
We have a mechanism for restoring rows originating from an instance that suffered a sudden power loss: remote masters resend the isntance's rows received before a certain point in time, defined by remote master vclock at the moment of subscribe. However, this is useful only on initial replication configuraiton, when an instance has just recovered, so that it can receive what it has relayed but haven't synced to disk. In other cases, when an instance is operating normally and master-master replication is configured, the mechanism described above may lead to instance re-applying instance's own rows, coming from a master it has just subscribed to. To fix the problem do not relay rows coming from a remote instance, if the instance has already recovered. Closes #4739 (cherry picked from commit ed2e143)
- Loading branch information
1 parent
1be8a8b
commit 1e37433
Showing
9 changed files
with
145 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,88 @@ | ||
-- test-run result file version 2 | ||
env = require('test_run') | ||
| --- | ||
| ... | ||
test_run = env.new() | ||
| --- | ||
| ... | ||
|
||
SERVERS = {'rebootstrap1', 'rebootstrap2'} | ||
| --- | ||
| ... | ||
test_run:create_cluster(SERVERS, "replication") | ||
| --- | ||
| ... | ||
test_run:wait_fullmesh(SERVERS) | ||
| --- | ||
| ... | ||
|
||
test_run:cmd('switch rebootstrap1') | ||
| --- | ||
| - true | ||
| ... | ||
fiber = require('fiber') | ||
| --- | ||
| ... | ||
-- Stop updating replicaset vclock to simulate a situation, when | ||
-- a row is already relayed to the remote master, but the local | ||
-- vclock update hasn't happened yet. | ||
box.error.injection.set('ERRINJ_RELAY_FASTER_THAN_TX', true) | ||
| --- | ||
| - ok | ||
| ... | ||
lsn = box.info.lsn | ||
| --- | ||
| ... | ||
f = fiber.create(function() box.space._schema:replace{'something'} end) | ||
| --- | ||
| ... | ||
f:status() | ||
| --- | ||
| - suspended | ||
| ... | ||
-- Vclock isn't updated. | ||
box.info.lsn == lsn | ||
| --- | ||
| - true | ||
| ... | ||
|
||
-- Wait until the remote instance gets the row. | ||
test_run:wait_cond(function()\ | ||
return test_run:get_vclock('rebootstrap2')[box.info.id] > lsn\ | ||
end, 10) | ||
| --- | ||
| - true | ||
| ... | ||
|
||
-- Restart the remote instance. This will make the first instance | ||
-- resubscribe without entering orphan mode. | ||
test_run:cmd('restart server rebootstrap2 with wait=False') | ||
| --- | ||
| - true | ||
| ... | ||
test_run:cmd('switch rebootstrap1') | ||
| --- | ||
| - true | ||
| ... | ||
-- Wait until resubscribe is sent | ||
test_run:wait_cond(function()\ | ||
return box.info.replication[2].upstream.status == 'sync'\ | ||
end, 10) | ||
| --- | ||
| - true | ||
| ... | ||
box.error.injection.set('ERRINJ_RELAY_FASTER_THAN_TX', false) | ||
| --- | ||
| - ok | ||
| ... | ||
box.space._schema:get{'something'} | ||
| --- | ||
| - ['something'] | ||
| ... | ||
test_run:cmd('switch default') | ||
| --- | ||
| - true | ||
| ... | ||
test_run:drop_cluster(SERVERS) | ||
| --- | ||
| ... |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,36 @@ | ||
env = require('test_run') | ||
test_run = env.new() | ||
|
||
SERVERS = {'rebootstrap1', 'rebootstrap2'} | ||
test_run:create_cluster(SERVERS, "replication") | ||
test_run:wait_fullmesh(SERVERS) | ||
|
||
test_run:cmd('switch rebootstrap1') | ||
fiber = require('fiber') | ||
-- Stop updating replicaset vclock to simulate a situation, when | ||
-- a row is already relayed to the remote master, but the local | ||
-- vclock update hasn't happened yet. | ||
box.error.injection.set('ERRINJ_RELAY_FASTER_THAN_TX', true) | ||
lsn = box.info.lsn | ||
f = fiber.create(function() box.space._schema:replace{'something'} end) | ||
f:status() | ||
-- Vclock isn't updated. | ||
box.info.lsn == lsn | ||
|
||
-- Wait until the remote instance gets the row. | ||
test_run:wait_cond(function()\ | ||
return test_run:get_vclock('rebootstrap2')[box.info.id] > lsn\ | ||
end, 10) | ||
|
||
-- Restart the remote instance. This will make the first instance | ||
-- resubscribe without entering orphan mode. | ||
test_run:cmd('restart server rebootstrap2 with wait=False') | ||
test_run:cmd('switch rebootstrap1') | ||
-- Wait until resubscribe is sent | ||
test_run:wait_cond(function()\ | ||
return box.info.replication[2].upstream.status == 'sync'\ | ||
end, 10) | ||
box.error.injection.set('ERRINJ_RELAY_FASTER_THAN_TX', false) | ||
box.space._schema:get{'something'} | ||
test_run:cmd('switch default') | ||
test_run:drop_cluster(SERVERS) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters