Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix the race between PromoteReplica and replication manager tick #9859

Merged
merged 6 commits into from
Mar 20, 2022
Merged

Fix the race between PromoteReplica and replication manager tick #9859

merged 6 commits into from
Mar 20, 2022

Conversation

GuptaManan100
Copy link
Member

Description

The order of operations for PromoteReplica before were -

  1. Fix MySQL by resetting replication and fix semi-sync settings
  2. Change Tablet type in the topo server (tablet record)
  3. update the local state of tablet manager to Primary.
  4. Stop replication manager
  5. Start a loop of shard sync which will update the topo server (shard record)
  6. Start replication

So any run of replication manager tick after the first step and before the 4th step caused the issue seen in #9819.

This PR fixes the race between PromoteReplica stopping the replication manager and the replication manager tick running after step 1.

After this PR, the order of operations for PromoteReplica are -

  1. Stop replication manager
  2. Fix MySQL by resetting replication and fix semi-sync settings
  3. Change Tablet type in the topo server (tablet record)
  4. update the local state of tablet manager to Primary.
  5. Start a loop of shard sync which will update the topo server (shard record)
  6. Start replication

Related Issue(s)

Fixes #9819

Checklist

  • Should this PR be backported? No
  • Tests were added or are not required
  • Documentation was added or is not required

Deployment Notes

@GuptaManan100
test: improve test to check that replication manager is stopped befor…
f9881cc 
…e we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
feat: stop the replication manager first in PromoteReplica call
3aaa897 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 requested review from rohit-nayak-ps and removed request for rohit-nayak-ps March 11, 2022 09:50
@darenseagrave darenseagrave mentioned this pull request Mar 15, 2022
Closed
@GuptaManan100
docs: add comments to replication manager setTabletType
a9be7db 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
docs: add comment for SetReplicationSource in wrangler
3349b77 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
Merge remote-tracking branch 'upstream/main' into replication-manager…
72fd087 
…-promote-replica

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100
docs: add comments to SetReplicationSource in reparentutil
9c46153 
Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 force-pushed the replication-manager-promote-replica branch from 4209136 to 9c46153 Compare March 17, 2022 07:38
@GuptaManan100 GuptaManan100 merged commit 30f03c8 into vitessio:main Mar 20, 2022
@GuptaManan100 GuptaManan100 deleted the replication-manager-promote-replica branch March 20, 2022 03:43
GuptaManan100 added a commit to planetscale/vitess that referenced this pull request Apr 12, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (vit…
f4059b7 
…essio#9859)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 mentioned this pull request Apr 12, 2022
Merged
3 tasks
GuptaManan100 added a commit to planetscale/vitess that referenced this pull request Apr 12, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (vit…
c0caa38 
…essio#9859)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 mentioned this pull request Apr 12, 2022
Merged
3 tasks
GuptaManan100 added a commit to planetscale/vitess that referenced this pull request Apr 12, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (vit…
4d9e04d 
…essio#9859)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
@GuptaManan100 GuptaManan100 mentioned this pull request Apr 12, 2022
Merged
3 tasks
GuptaManan100 added a commit that referenced this pull request Apr 12, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (#9859…
cd6bf39 
…) (#10075)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
ameetkotian pushed a commit to tinyspeck/vitess that referenced this pull request Apr 12, 2022
@GuptaManan100 @ameetkotian
Fix the race between PromoteReplica and replication manager tick (vit…
55d326a 
…essio#9859)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
GuptaManan100 added a commit that referenced this pull request Apr 13, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (#9859…
ccbf388 
…) - release 11 (#10077)

* Fix the race between PromoteReplica and replication manager tick (#9859)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: change PRIMARY to MASTER to fix syntax error

Signed-off-by: Manan Gupta <manan@planetscale.com>
GuptaManan100 added a commit that referenced this pull request Apr 14, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (#9859…
e87dfc4 
…) (#10076)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
@ameetkotian ameetkotian mentioned this pull request Apr 19, 2022
Merged
8 tasks
notfelineit pushed a commit to planetscale/vitess that referenced this pull request May 3, 2022
@GuptaManan100
Fix the race between PromoteReplica and replication manager tick (vit…
7c6097a 
…essio#9859) (vitessio#552)

* test: improve test to check that replication manager is stopped before we make any changes in MySQL

Signed-off-by: Manan Gupta <manan@planetscale.com>

* feat: stop the replication manager first in PromoteReplica call

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to replication manager setTabletType

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comment for SetReplicationSource in wrangler

Signed-off-by: Manan Gupta <manan@planetscale.com>

* docs: add comments to SetReplicationSource in reparentutil

Signed-off-by: Manan Gupta <manan@planetscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@deepthi deepthi deepthi approved these changes

@shlomi-noach shlomi-noach Awaiting requested review from shlomi-noach shlomi-noach is a code owner

@rohit-nayak-ps rohit-nayak-ps Awaiting requested review from rohit-nayak-ps rohit-nayak-ps is a code owner

Assignees
No one assigned
Labels
Component: Cluster management Type: Bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Bug Report: race condition with PlannedReparentShard can leave a shard with no PRIMARY
2 participants
@GuptaManan100 @deepthi