Doc updates for planned maintenance events

[jseldess]

• Add FAQ on handling planned node maintenance
• Add guidance on using server.time_until_store_dead for rolling upgrades
• Minor improvements to cockroach quit doc
• Retitle Remove Nodes to Decommission Nodes

Fixes #2038

[cockroach-teamcity]

This change is 

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/7a042fc57de281436af41ec66ebd3d5be9d8fa65/

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/af982d5569136a88a0630adf102b0f9c77da4bde/

[dianasaur323]

lgtm!

[jseldess]

TFTR, @dianasaur323!

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/870d0e95a9d8a7b37fc5740baf750d6f009c6d9d/

[bdarnell]

Review status: 0 of 7 files reviewed at latest revision, 1 unresolved discussion, some commit checks pending.

---

v2.0/upgrade-cockroach-version.md, line 34 at r5 (raw file):

5. By default, if a node stays offline for more than 5 minutes, the cluster will consider it dead and will rebalance its data to other nodes. If you expect any nodes to be offline for longer than 5 minutes, you can prevent the cluster from unnecessarily rebalancing data off the nodes by increasing the `server.time_until_store_dead` [cluster setting](cluster-settings.html) to match the estimated maintenance window.

    For example, let's say you expect nodes to be offline for up to 10 minutes during the rolling upgrade. Before starting, you would change the `server.time_until_store_dead` cluster setting as follows:

Why would you expect this much downtime during a rolling upgrade? You should easily be able to complete your cockroach upgrade within the default 5m window (as long as you're automating things instead of typing them by hand during the upgrade).

In my experience the only thing that might take long enough to warrant an increase in server.time_until_store_dead is a kernel upgrade.

---

Comments from Reviewable

[jseldess]

v2.0/upgrade-cockroach-version.md, line 34 at r5 (raw file):

Previously, bdarnell (Ben Darnell) wrote…

Why would you expect this much downtime during a rolling upgrade? You should easily be able to complete your cockroach upgrade within the default 5m window (as long as you're automating things instead of typing them by hand during the upgrade).

In my experience the only thing that might take long enough to warrant an increase in server.time_until_store_dead is a kernel upgrade.

Downgraded this to a note that states 5 min should be more than enough. Calls out the setting just in case. Is that ok, @bdarnell?

---

Comments from Reviewable

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/90efa5a15b14bff13b594ab95b888bae1f732e5d/

[bdarnell]

Review status: 0 of 7 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

---

v2.0/upgrade-cockroach-version.md, line 34 at r5 (raw file):

Previously, jseldess wrote…

Downgraded this to a note that states 5 min should be more than enough. Calls out the setting just in case. Is that ok, @bdarnell?

I'd probably leave it out completely, but I guess since we're writing these instructions as if you're running commands by hand instead of automating it it may be worth leaving in a mention.

---

Comments from Reviewable

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/d762d8d901dddec2b420d2f5f1e0a7202746bfdc/

[jseldess]

v2.0/upgrade-cockroach-version.md, line 34 at r5 (raw file):
OK, removed.

At the beginning of the "Perform the rolling upgrade" section, we have a tip:

We recommend creating scripts to perform these steps instead of performing them by hand.

But I'd like to get more information from you about how this page could be more focused on automation. Seems to me that manual steps are necessary given that users will use differing tools. But I'd love to create and link to an example script or config for a specific system as well.

---

Comments from Reviewable

[cockroach-teamcity]

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/c2b4566d2d79c951d8e7549dc59c8087b9d165d7/

[bdarnell]

Review status: 0 of 6 files reviewed at latest revision, 1 unresolved discussion, all commit checks successful.

---

v2.0/upgrade-cockroach-version.md, line 34 at r5 (raw file):

Previously, jseldess (Jesse Seldess) wrote…

OK, removed.

At the beginning of the "Perform the rolling upgrade" section, we have a tip:

We recommend creating scripts to perform these steps instead of performing them by hand.

But I'd like to get more information from you about how this page could be more focused on automation. Seems to me that manual steps are necessary given that users will use differing tools. But I'd love to create and link to an example script or config for a specific system as well.

My view is that we have to walk through the manual process here so that people will understand what they have to write in their scripts. Over time we can develop a library of scripts/configs (like we have for kubernetes, etc), and those automated versions will displace these manual steps.

---

Comments from Reviewable