Introduce CCR getting started guide #35434
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,320 @@ | ||
| [role="xpack"] | ||
| [testenv="platinum"] | ||
| [[ccr-gs]] | ||
| == Getting Started with Cross-Cluster Replication | ||
|
|
||
| beta[] | ||
|
|
||
| This getting-started guide for {ccr} shows you how to: | ||
|
|
||
| * <<ccr-gs-remote-cluster,Connect a local cluster to a remote cluster>> | ||
| * <<ccr-gs-leader-index,Create a leader index>> in a remote cluster that can be | ||
| replicated to a follower index in a local cluster | ||
| * <<ccr-gs-follower-index,Create a follower index>> that replicates a leader | ||
| index in a remote cluster | ||
| * <<ccr-gs-auto-follow,Automatically create follower indices>> that replicate | ||
| new indices in a remote cluster | ||
|
|
||
| [float] | ||
| [[ccr-gs-before-you-begin]] | ||
| === Before you begin | ||
| . {stack-gs}/get-started-elastic-stack.html#install-elasticsearch[Install {es}] | ||
| your on local and remote clusters. | ||
|
|
||
| . Obtain a license that includes the {ccr} features. See | ||
| {stack-ov}/license-management.html[License Management]. | ||
|
|
||
| . If {security} is enabled in your local and remote cluster, you need a user | ||
| that has appropriate authority to perform the steps in this tutorial. | ||
| + | ||
| -- | ||
| [[ccr-gs-security]] | ||
| The {ccr} features use cluster privileges and built-in roles to make it easier | ||
| to control which users have authority to manage {ccr}. | ||
|
|
||
| By default, you can perform all of the steps in this getting-started guide by | ||
| using the built-in `elastic` user. However, a password must be set for this user | ||
| before the user can do anything. For information about how to set that password, | ||
| see {stack-ov}/security-getting-started.html[Getting started with security]. | ||
|
|
||
| If you are performing these steps in a production environment, take extra care | ||
| because the `elastic` user has the `superuser` role and you could inadvertently | ||
| make significant changes. You can alternately assign the appropriate privileges | ||
| to a user ID of your choice. | ||
|
|
||
| On the remote cluster containing the leader index, a user will need the | ||
| `read_ccr` cluster privilege, and `monitor` and `read` privileges on the leader | ||
| index. | ||
|
|
||
| [source,yml] | ||
| -------------------------------------------------- | ||
| ccr_user: | ||
| cluster: | ||
| - read_ccr | ||
|
There was a problem hiding this comment. Should this privilege be added to the list here?: https://www.elastic.co/guide/en/elastic-stack-overview/master/security-privileges.html |
||
| indices: | ||
| - names: [ 'leader-index' ] | ||
| privileges: | ||
| - monitor | ||
| - read | ||
| -------------------------------------------------- | ||
|
|
||
| On the local cluster containing the follower index, the same user will need the | ||
| `manage_ccr` cluster privilege, and `monitor`, `read`, `write` and | ||
| `manage_follow_index` privileges on the follower index. | ||
|
|
||
| [source,yml] | ||
| -------------------------------------------------- | ||
| ccr_user: | ||
| cluster: | ||
| - manage_ccr | ||
|
There was a problem hiding this comment. I don't see this manage_ccr privilege documented yet either: https://www.elastic.co/guide/en/elastic-stack-overview/master/security-privileges.html There was a problem hiding this comment. Yes, we should add that there. |
||
| indices: | ||
| - names: [ 'follower-index' ] | ||
| privileges: | ||
| - monitor | ||
| - read | ||
| - write | ||
| - manage_follow_index | ||
| -------------------------------------------------- | ||
| -- | ||
|
There was a problem hiding this comment. I think you'll also need manage cluster privileges to change the cluster settings. |
||
|
|
||
| [float] | ||
| [[ccr-gs-remote-cluster]] | ||
| === Connecting to a remote cluster | ||
|
|
||
| The {ccr} features require | ||
| {ref}/modules-remote-clusters.html[connecting your local cluster to a remote cluster]. | ||
| In the example for this getting-started guide, we will connect our local cluster | ||
| to a remote cluster with the cluster alias `leader`. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /_cluster/settings | ||
| { | ||
| "persistent" : { | ||
| "cluster" : { | ||
| "remote" : { | ||
| "leader" : { | ||
| "seeds" : [ | ||
| "127.0.0.1:9300" <1> | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[setup:host] | ||
| // TEST[s/127.0.0.1:9300/\${transport_host}/] | ||
| <1> Specifies the hostname and transport port of a seed node in the remote | ||
| cluster. | ||
|
|
||
| You can verify that the local cluster is successfully connected to the remote | ||
| cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| GET /_remote/info | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| The API will respond by showing that the local cluster is connected to the | ||
| remote cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "leader" : { | ||
| "seeds" : [ | ||
| "127.0.0.1:9300" | ||
| ], | ||
| "connected" : true, <1> | ||
| "num_nodes_connected" : 1, <2> | ||
| "max_connections_per_cluster" : 3, | ||
| "initial_connect_timeout" : "30s", | ||
| "skip_unavailable" : false | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
| // TEST[s/127.0.0.1:9300/$body.leader.seeds.0/] | ||
| // TEST[s/"connected" : true/"connected" : $body.leader.connected/] | ||
| // TEST[s/"num_nodes_connected" : 1/"num_nodes_connected" : $body.leader.num_nodes_connected/] | ||
| <1> This shows the local cluster is connected to the remote cluster with cluster | ||
| alias `leader` | ||
| <2> This shows the number of nodes in the remote cluster the local cluster is | ||
| connected to. | ||
|
|
||
| [float] | ||
| [[ccr-gs-leader-index]] | ||
| === Creating a leader index | ||
|
|
||
| Leader indices require special index settings to ensure that the operations that | ||
| need to be replicated by the follower from the leader are available when the | ||
| follower requests them from the leader. These settings are used to enable soft | ||
| deletes on the leader index, and control how many soft deletes are retained. A | ||
| soft delete occurs whenever a document is deleted or updated. Soft deletes can | ||
| only be enabled on new indices created on or after {es} 6.5.0. In the following | ||
| example, we will create a leader index in the remote cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /server-metrics | ||
| { | ||
| "settings" : { | ||
| "index" : { | ||
| "number_of_shards" : 1, | ||
| "number_of_replicas" : 0, | ||
| "soft_deletes" : { | ||
| "enabled" : true, <1> | ||
| "retention" : { | ||
| "operations" : 1024 <2> | ||
| } | ||
| } | ||
| } | ||
| }, | ||
| "mappings" : { | ||
| "metric" : { | ||
| "properties" : { | ||
| "@timestamp" : { | ||
| "type" : "date" | ||
| }, | ||
| "accept" : { | ||
| "type" : "long" | ||
| }, | ||
| "deny" : { | ||
| "type" : "long" | ||
| }, | ||
| "host" : { | ||
| "type" : "keyword" | ||
| }, | ||
| "response" : { | ||
| "type" : "float" | ||
| }, | ||
| "service" : { | ||
| "type" : "keyword" | ||
| }, | ||
| "total" : { | ||
| "type" : "long" | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
| <1> Enables soft deletes on the leader index. | ||
| <2> Sets that up to 1024 soft deletes will be retained. | ||
|
|
||
| [float] | ||
| [[ccr-gs-follower-index]] | ||
| === Creating a follower index | ||
|
|
||
| Follower indices are created with the {ref}/ccr-put-follow.html[create follower API]. | ||
| When creating a follower index, you will reference the | ||
| <<ccr-gs-remote-cluster,remote cluster>> that you connected your | ||
| local cluster to, and the <<ccr-gs-leader-index,leader index>> that | ||
| you created in the remote cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /server-metrics-copy/_ccr/follow | ||
| { | ||
| "remote_cluster" : "leader", | ||
| "leader_index" : "server-metrics" | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "follow_index_created" : true, | ||
| "follow_index_shards_acked" : true, | ||
| "index_following_started" : true | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| Now when you index documents into your leader index, you will see these | ||
| documents are replicated by the follower index from the leader index. You can | ||
| inspect the status of replication using the | ||
| {ref}/ccr-get-follow-stats[get follower stats API]. | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| POST /server-metrics-copy/_ccr/pause_follow | ||
|
|
||
| POST /server-metrics-copy/_close | ||
|
|
||
| POST /server-metrics-copy/_ccr/unfollow | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [float] | ||
| [[ccr-gs-auto-follow]] | ||
| === Automatically create follower indices | ||
|
|
||
| The auto-follow feature in {ccr} helps for time-series use-cases where new | ||
| indices that you want to follow are periodically created in the remote cluster | ||
| (e.g., daily Beats indices). Auto-following is configured using the | ||
| {ref}/ccr-put-auto-follow-pattern.html[create auto-follow pattern API]. With an | ||
| auto-follow pattern, you reference the <<ccr-gs-remote-cluster,remote cluster>> | ||
| that you connected your local cluster to, and a collection of patterns that | ||
| match the indices that you want to be automatically followed when they are | ||
| created in the remote cluster. | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| PUT /_ccr/auto_follow/beats | ||
| { | ||
| "remote_cluster" : "leader", | ||
| "leader_index_patterns" : | ||
| [ | ||
| "metricbeat-*", <1> | ||
| "packetbeat-*" <2> | ||
| ], | ||
| "follow_index_pattern" : "{{leader_index}}-copy" <3> | ||
| } | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
| <1> Automatically follow newly-created Metricbeat indices. | ||
| <2> Automatically follow newly-created Packetbeat indices. | ||
| <3> The name of the follower index is derived from the name of the leader index | ||
| by adding the suffix `-copy` to the name of the leader index. | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| { | ||
| "acknowledged" : true | ||
| } | ||
| -------------------------------------------------- | ||
| // TESTRESPONSE | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| ////////////////////////// | ||
|
|
||
| [source,js] | ||
| -------------------------------------------------- | ||
| DELETE /_ccr/auto_follow/beats | ||
| -------------------------------------------------- | ||
| // CONSOLE | ||
| // TEST[continued] | ||
|
|
||
| ////////////////////////// | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -9,7 +9,7 @@ | |
| beta[] | ||
|
|
||
| * <<ccr-overview, Overview>> | ||
| * <<ccr-gs,Getting Started>> | ||
|
|
||
| -- | ||
|
|
||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It would be better for SEO purposes to leave "getting-started" spelled out in this anchor. The subsections can use the abbreviation. Should probably also use the {ccr} attribute in the header.