Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOCS] add replace yb-master docs #4791

Merged
merged 25 commits into from
Oct 14, 2020
Merged
Changes from 12 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
8c1c525
add replace yb-master docs
ddorian Jun 16, 2020
12f8a53
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
45fe032
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
69432c5
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
9aacb83
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
fdffc1c
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
9dd9a08
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Jun 25, 2020
674174f
Merge branch 'master' of https://github.com/yugabyte/yugabyte-db into…
ddorian Jun 25, 2020
76729c7
Merge branch 'doc-replace-master' of https://github.com/ddorian/yugab…
ddorian Jun 25, 2020
54962e4
added missing note
ddorian Jun 25, 2020
e32b014
add note
ddorian Jun 29, 2020
d950ef4
Merge branch 'master' of https://github.com/yugabyte/yugabyte-db into…
ddorian Sep 9, 2020
34d4435
move note to the top
ddorian Sep 29, 2020
0761470
Merge branch 'master' of https://github.com/yugabyte/yugabyte-db into…
ddorian Sep 29, 2020
ddf04ef
nicer headers
ddorian Sep 29, 2020
67996cc
fix sentence for including all yb-masters
ddorian Sep 29, 2020
96afa87
explain better adding a master later
ddorian Sep 29, 2020
2849ae3
replacement -> standby
ddorian Oct 8, 2020
6568157
use empty string for --master_addresses
ddorian Oct 8, 2020
7d6017b
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
f752eff
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
80c100d
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
ceac034
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
92959e5
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
2e2a24f
Update docs/content/latest/troubleshoot/cluster/replace_master.md
ddorian Oct 13, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
73 changes: 73 additions & 0 deletions docs/content/latest/troubleshoot/cluster/replace_master.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
---
title: Replace a failed YB-Master
headerTitle: Replace a failed YB-Master
linkTitle: Replace a failed YB-Master
description: Steps to replace a failed YB-Master in a YugabyteDB cluster.
aliases:
- /troubleshoot/cluster/replace-master/
- /latest/troubleshoot/cluster/replace-master/
menu:
latest:
identifier: replace-failed-master
parent: troubleshoot-cluster
ddorian marked this conversation as resolved.
Show resolved Hide resolved
weight: 831
isTocNested: true
showAsideToc: true
---

To replace a failed YB-Master server in a YugabyteDB cluster, follow these steps:

For the steps below, the examples use the following scenario:

- The cluster includes three `yb-master` servers: `M1`, `M2`, `M3`.
- YB-Master server `M1` failed and needs to be replaced.
- A new YB-Master server (`M4`) will replace `M1`.
- The default master RPC port is `7100`


1. Start the new (replacement) YB-Master server in standby mode.
ddorian marked this conversation as resolved.
Show resolved Hide resolved

To start `yb-master` in standby mode, set the `--master_addresses` flag to an empty string (`""`).

```sh
./bin/yb-master --master_addresses=""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this might end up being confusing to users, if they already have a bunch of other necessary flags, like placement options, fs_data_dirs, etc

I wouldn't want folks to assume this is all the flags they need to bring the master up, but I'm not sure how we're handling that in other sections?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, now it's actually a bit more confusing, as we're talking about the flag being set to empty string AND the flag not being set, in the same sentence...which for a simple user, could be 2 different things.

What I was thinking was, the command we highlight should either:

  • not include --master_addresses -- and then the messaging above should say just don't provide it
  • include --master_addresses="" -- and the messaging should say set it to empty string

But moreover, the command should maybe also include something like

--fs_data_dirs=<your data directories>
[any other flags you would normally pass to this master process]

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed change with --master_addresses=""

```

2. Add the new YB-Master server into the existing cluster.

To add the new YB-Master server, run the [`yb-admin change_master_config ADD_SERVER`](../../admin/cli/yb-admin/#change-master-config) command.

```sh
./bin/yb-admin -master_addresses M1:7100,M2:7100,M3:7100 change_master_config ADD_SERVER M4 7100
```

3. Remove the old YB-Master server from the cluster.

To remove the failed YB-Master server from the cluster, use the [`yb-admin change_master_config REMOVE_SERVER`](../../admin/yb-admin/#change-master-config`) command.

```sh
./yb-admin -master_addresses M1:7100,M2:7100,M3:7100,M4:7100 change_master_config REMOVE_SERVER M1 7100
```
{{< note title="Note" >}}

Make sure to specify all YB-Master addresses which will prevent the new YB-Master server from unexpectedly becoming the leader.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, the end part of this comment does not make sense...you need to specify M4, not to prevent it from becoming the leader (this command cannot affect that), but instead, to make sure that, if it DID become the leader in the meantime, then yb-admin can find it

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Check change here 67996cc ?


{{< /note >}}
## Validate cluster

Validate that your set of masters is now `M2`, `M3` and `M4` using:
```bash
yb-admin -master_addresses M2:7100,M3:7100,M4:7100 list_all_masters
```

Until [#1542](https://github.com/yugabyte/yugabyte-db/issues/1542) is implemented, the TS will by default only know of
whatever masters are encoded in the `--tserver_master_addrs` flag that they are started with.

If any one of those masters is still part of the active quorum, then they will propagate the new master quorum over via heartbeats.
If, however, none of the current masters are present in the TS flag, then the TS will not be able to join the cluster!

So it is important to make sure to update `--tserver_master_addrs` on every TS to the new set of master addresses, `M2:7100,M3:7100,M4:7100`!

{{< note title="Note" >}}
If the master you wish to replace is already dead (eg: VM was terminated), you might want to first do the `REMOVE` step, then do the `ADD` step.
{{< /note >}}
ddorian marked this conversation as resolved.
Show resolved Hide resolved