DOC-11185 document fast failover/node without services #3378
Conversation
|
NB, there is a pre-build of these materials, to facilitate review. Note the following locations: What's New (pointers to info on fast failover and quorum arbitration) Fast Failover. Quorum Arbitration Rest API Nodes page (section on serviceless) Services (reference to serviceless) |
| The _Data Service_ must run on at least one node. | ||
| Each node can run any number of services, up to the maximum, which is seven. | ||
| In Couchbase Enterprise Server Version 7.6+, a node can be configured to run _no_ service. | ||
| The _Data Service_ must run on at least one node of the cluster. |
There was a problem hiding this comment.
I'm not sure this is important here but....The first node of the cluster (the one that creates the cluster) must have the data service configured.
| * _Quorum Arbitration_. | ||
| See xref:install:deployment-considerations-lt-3nodes.adoc#quorum-arbitration[Quorum Arbitration]. | ||
|
|
||
| Note, however, that in the creation of an initial, one-node cluster, the node must always be assigned at least the Data Service. |
There was a problem hiding this comment.
Ah, I see it mentioned here.
| @@ -21,3 +21,11 @@ See xref:rest-api:rest-setting-security.adoc[Configure On-the-Wire Security]. | |||
| * Credentials for Couchbase-Server internal users can now be rotated at any time, by means of the REST API. | |||
| See xref:rest-api:rest-rotate-internal-credentials.adoc[Rotate Internal Credentials]. | |||
|
|
|||
| * One or more _Serviceless Nodes_ can now be added to a cluster. | |||
There was a problem hiding this comment.
This should change to Arbiter Node
There was a problem hiding this comment.
(Disclaimer: I've not read the doc proposal in it's entirety yet, this comment is just about naming of this new capability)
I agree with Tony's comments in the email that @stevewatanabe shared that "Arbiter Node" isn't a good description of this feature. Rather than describe a new capability of the server (that we can add nodes running only the cluster manager) it describes one use-case for those nodes (acting as an arbiter). Given that we've identified at least two uses for these types of nodes so far (acting as an arbiter and for preferential orchestrator selection) I don't think that it's a particularly good idea to name them after one use case, rather than the capability that we've added. We don't know what other use cases we might find for these in the future, and "Arbiter Node" might not fit those particularly well.
A couple of points additionally worth highlighting:
- In addition to the faster failover case, the preferential orchestrator selection can additionally be used to take orchestrator burden (the increased CPU/memory required) away from any other node.
- Similarly to the example given here, that any non-KV node can be used to the same extent as a service-less node to achieve faster failover, any node running any service can act as an arbiter to achieve a quorum majority. We've recommened, and I believe seen a few customer use, a backup service node for this in the past.
I'm not a fan of naming things, but IMO we should call these something like "Cluster Manager Node" (perhaps "Cluster Manager Only Node"), and explicitly call out that other service nodes will pick up cluster management activites if such a node is used, and that this is perfectly fine for most use cases.
There was a problem hiding this comment.
I can't come up with a better term off the top of my head (and I haven't fully read the draft doc or the associated PRDs). At my previous company, we called a similar feature "standby nodes."
But, assuming that "Arbiter Node" is mentioned either in the UI, the API, or some message someplace in the product, it's too late to try to find the perfect term. I'll go ahead and update this PR with the approved "arbiter Node" term.
There was a problem hiding this comment.
FWIW this feature is only available in the REST API, it is not in the UI, and it is configured as an absence of services when we set up a node. We have not called this an "arbiter node" anywhere within ns_server.
There was a problem hiding this comment.
I've gone ahead and swapped all instances of serviceless node with arbiter node.
| In case where the Master Services are co-located with a highly subscribed services (in particular, the Data Service), this can result in unnecessary latency in failing over the node and ensuring that data can be served. | ||
|
|
||
| In consequence, the Master Services should ideally _not_ be co-located with any service. | ||
| Such a configuration is possible in Couchbase Enterprise Server 7.6+: one or more _serviceless_ nodes can be added to the cluster, with the intention of ensuring that the Master Services will occupy such a node, and thereby not be co-located with any service. |
There was a problem hiding this comment.
@stevewatanabe please correct me if I'm wrong, but there's no requirement to use an arbiter node to achieve this goal. The cluster will automatically choose any node that isn't running the data service as a preference. The addition of an arbiter node is useful where there are no available nodes in the cluster in the required configuration such as a cluster of all data nodes.
There was a problem hiding this comment.
I've asked @BenHuddleston to chime in as he implemented a weighting system based on the services and their type running on each node. Ian is correct that an arbiter node isn't required....Ben can provide more detailed information.
There was a problem hiding this comment.
@b33f any non-KV node could be used for this. For your interest, the priority order is as follows (higher weight => lower priority, service-less nodes are implicitly 0):
%% Weight values used in mb_master to determine orchestrator placement.
-define(DEFAULT_SERVICE_WEIGHTS, [{kv, 10000},
{index, 1000},
{fts, 1000},
{cbas, 1000},
{n1ql, 100},
{eventing, 100},
{backup, 10}]).
There was a problem hiding this comment.
I adde some more detail here to explain the rankings of services. I didn't put down the actual cores, but just mentioned that each "tier" is 10x it's predecessor. Mainly, I think, it just aswers potential user questions on why a node with more services running was picked over a node running just the KV service. Please let me know if that should be removed, or if I got it wrong.
ggray-cb
changed the title
|
Renamed PR to match outstanding JIRA ticket so I don;t lose this thing again. |
* Made repetitive doc into a partial. * Replaced some repirition with a link.
|
I will be merging this tomorrow (March 14th) unless there are objections. |
|
No new comments from me, thanks Gary. |
No description provided.