Fix renaming data streams with CCR replication #88875

dakrone · 2022-07-27T22:00:10Z

This commit fixes the situation where a user wants to use CCR to replicate indices that are part of
a data stream while renaming the data stream. For example, assume a user has an auto-follow request
that looks like this:

PUT /_ccr/auto_follow/my-auto-follow-pattern
{
  "remote_cluster" : "other-cluster",
  "leader_index_patterns" : ["logs-*"],
  "follow_index_pattern" : "{{leader_index}}_copy"
}

And then the data stream logs-mysql-error was created, creating the backing index
.ds-logs-mysql-error-2022-07-29-000001.

Prior to this commit, replicating this data stream means that the backing index would be renamed to
.ds-logs-mysql-error-2022-07-29-000001_copy and the data stream would not be renamed. This
caused a check to trip in TransportPutLifecycleAction asserting that a backing index was not
renamed for a data stream during following.

After this commit, there are a couple of changes:

First, the data stream will also be renamed. This means that the logs-mysql-error becomes
logs-mysql-error_copy when created on the follower cluster. Because of the way that CCR works,
this means we need to support renaming a data stream for a regular "create follower" request, so a
new parameter has been added: data_stream_name. It works like this:

PUT /mynewindex/_ccr/follow
{
  "remote_cluster": "other-cluster",
  "leader_index": "myotherindex",
  "data_stream_name": "new_ds"
}

Second, the backing index for a data stream must be renamed in a way that does not break the parsing
of a data stream backing pattern, whereas previously the index
.ds-logs-mysql-error-2022-07-29-000001 would be renamed to
.ds-logs-mysql-error-2022-07-29-000001_copy (an illegal name since it doesn't end with the
rollover digit), after this commit it will be renamed to
.ds-logs-mysql-error_copy-2022-07-29-000001 to match the renamed data stream. This means that for
the given follow_index_pattern of {{leader_index}}_copy the index changes look like:

Leader Cluster	Follower Cluster
`logs-mysql-error` (data stream)	`logs-mysql-error_copy` (data stream)
`.ds-logs-mysql-error-2022-07-29-000001`	`.ds-logs-mysql-error_copy-2022-07-29-000001`

Which internally means the auto-follow request turned into the create follower request of:

PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow
{
  "remote_cluster": "other-cluster",
  "leader_index": ".ds-logs-mysql-error-2022-07-29-000001",
  "data_stream_name": "logs-mysql-error_copy"
}

Relates to #84940 (cherry-picked the commit for a test)
Relates to #61993 (where data stream support was first introduced for CCR)
Resolves #81751

dakrone · 2022-07-28T22:11:10Z

@elasticsearchmachine test this please

Relates to elastic#81751

This commit fixes the situation where a user wants to use CCR to replicate indices that are part of a data stream while renaming the data stream. For example, assume a user has an auto-follow request that looks like this: ``` PUT /_ccr/auto_follow/my-auto-follow-pattern { "remote_cluster" : "other-cluster", "leader_index_patterns" : ["logs-*"], "follow_index_pattern" : "{{leader_index}}_copy" } ``` And then the data stream `logs-mysql-error` was created, creating the backing index `.ds-logs-mysql-error-2022-07-29-000001`. Prior to this commit, replicating this data stream means that the backing index would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` and the data stream would *not* be renamed. This caused a check to trip in `TransportPutLifecycleAction` asserting that a backing index was not renamed for a data stream during following. After this commit, there are a couple of changes: First, the data stream will also be renamed. This means that the `logs-mysql-error` becomes `logs-mysql-error_copy` when created on the follower cluster. Because of the way that CCR works, this means we need to support renaming a data stream for a regular "create follower" request, so a new parameter has been added: `data_stream_name`. It works like this: ``` PUT /mynewindex/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": "myotherindex", "data_stream_name": "new_ds" } ``` Second, the backing index for a data stream must be renamed in a way that does not break the parsing of a data stream backing pattern, whereas previously the index `.ds-logs-mysql-error-2022-07-29-000001` would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` (an illegal name since it doesn't end with the rollover digit), after this commit it will be renamed to `.ds-logs-mysql-error_copy-2022-07-29-000001` to match the renamed data stream. This means that for the given `follow_index_pattern` of `{{leader_index}}_copy` the index changes look like: | Leader Cluster | Follower Cluster | |--------------|-----------| | `logs-mysql-error` (data stream) | `logs-mysql-error_copy` (data stream) | | `.ds-logs-mysql-error-2022-07-29-000001` | `.ds-logs-mysql-error_copy-2022-07-29-000001` | Which internally means the auto-follow request turned into the create follower request of: ``` PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": ".ds-logs-mysql-error-2022-07-29-000001", "data_stream_name": "logs-mysql-error_copy" } ``` Relates to elastic#84940 (cherry-picked the commit for a test) Relates to elastic#61993 (where data stream support was first introduced for CCR) Resolves elastic#81751

elasticsearchmachine · 2022-07-29T17:01:23Z

Pinging @elastic/es-distributed (Team:Distributed)

elasticsearchmachine · 2022-07-29T17:01:23Z

Hi @dakrone, I've created a changelog YAML for you.

elasticsearchmachine · 2022-07-29T17:01:23Z

Pinging @elastic/es-data-management (Team:Data Management)

dakrone · 2022-07-29T19:36:30Z

@elasticmachine run elasticsearch-ci/packaging-tests-unix-sample

jbaiera

LGTM!

gmarouli

LGTM, I had some ideas/concerns shared, but I see that I am probably my understanding of the data stream is not complete :) . I would be happy to discuss those but I do not want to block this PR for this.

gmarouli · 2022-08-01T09:01:27Z

docs/reference/ccr/apis/follow/put-follow.asciidoc

@@ -76,6 +76,26 @@ referenced leader index. When this API returns, the follower index exists, and
  (Required, string) The <<remote-clusters,remote cluster>> containing
  the leader index.

+[[ccr-put-follow-request-body-data_stream_name]]`data_stream_name`::


Hm, if I understand correctly, CCR works on index level. With this API what we do is to follow a single leader index that belongs to a data stream and we provide also a data stream name. I do not see from this doc, any guidance on naming the follower index or the data stream of the follower index so it could be anything right?

Furthermore, I was wondering, if CCR is defined on an index level, this means that after the leader index will be rolled over the leader data stream would not be followed anymore. Right?

So, I was wondering. Is this API maybe too "free" and error prone, users might put invalid names and potentially create weird bugs?

Should we instead, verify the name of the follower index that it follows the data stream pattern and extract the data stream name from it?

So, if a user tries to follow an data stream backing index, for example, .ds-logs-mysql-default_copy-2022-01-01-000001 we will infer logs-mysql-default_copy as the data stream name. While in the case they try to create a follower index like .ds-logs-mysql-default-2022-01-01-000001_copy, we will fail and let them now that this is an invalid name.

Should we instead, verify the name of the follower index that it follows the data stream pattern and extract the data stream name from it?

It's worse than that, a data stream can have any arbitrary backing index name, because we have the migrate to data stream API that converts an alias to a data stream. So a data stream named logs-mysql-info could have backing indices named myfirstindex and other-index following no pattern. Rather than enforce that there is a pattern with the backing index names, I opted to go with a check for something that looks like our regular pattern and if that doesn't work, then we treat it as an "I don't know" pattern and just do the regular name replacement.

If for some reason we have something that looks like a good pattern name, but is not, then we throw an exception and stop the auto-following. I'll add a test for this specific scenario (in e3001d5).

Furthermore, I was wondering, if CCR is defined on an index level, this means that after the leader index will be rolled over the leader data stream would not be followed anymore. Right?

If the leader is rolled over then the auto-follow pattern would pick up the next index in the data stream (presumably <thing>-000002), the data stream is not really "followed" because CCR doesn't really follow data streams, but the next index would be replicated, only the data stream would already exist because the first index would have created it already.

Things are a little different in ILM, because ILM auto-injects some unfollow actions around rollover, but regardless the naming thing would be the same.

...ck/plugin/ccr/src/main/java/org/elasticsearch/xpack/ccr/action/TransportPutFollowAction.java

gmarouli · 2022-08-01T09:53:40Z

.../plugin/ccr/src/test/java/org/elasticsearch/xpack/ccr/action/AutoFollowCoordinatorTests.java

+            assertThat(request.getRemoteCluster(), equalTo("remote"));
+            assertThat(request.getFollowerIndex(), equalTo("my-backing-index_copy"));
+            assertThat(request.getLeaderIndex(), equalTo("my-backing-index"));
+            assertThat(request.getDataStreamName(), equalTo("logs-foo-bar_copy"));


This test is nicely representing the concern I raised earlier. I didn't know that it was possible and it's hard for me to assess the consequences.

Co-authored-by: Mary Gouseti <mgouseti@gmail.com>

dakrone · 2022-08-01T15:19:38Z

💚 All backports created successfully

Status	Branch	Result
✅	8.4

Questions ?

Please refer to the Backport tool documentation

This commit fixes the situation where a user wants to use CCR to replicate indices that are part of a data stream while renaming the data stream. For example, assume a user has an auto-follow request that looks like this: ``` PUT /_ccr/auto_follow/my-auto-follow-pattern { "remote_cluster" : "other-cluster", "leader_index_patterns" : ["logs-*"], "follow_index_pattern" : "{{leader_index}}_copy" } ``` And then the data stream `logs-mysql-error` was created, creating the backing index `.ds-logs-mysql-error-2022-07-29-000001`. Prior to this commit, replicating this data stream means that the backing index would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` and the data stream would *not* be renamed. This caused a check to trip in `TransportPutLifecycleAction` asserting that a backing index was not renamed for a data stream during following. After this commit, there are a couple of changes: First, the data stream will also be renamed. This means that the `logs-mysql-error` becomes `logs-mysql-error_copy` when created on the follower cluster. Because of the way that CCR works, this means we need to support renaming a data stream for a regular "create follower" request, so a new parameter has been added: `data_stream_name`. It works like this: ``` PUT /mynewindex/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": "myotherindex", "data_stream_name": "new_ds" } ``` Second, the backing index for a data stream must be renamed in a way that does not break the parsing of a data stream backing pattern, whereas previously the index `.ds-logs-mysql-error-2022-07-29-000001` would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` (an illegal name since it doesn't end with the rollover digit), after this commit it will be renamed to `.ds-logs-mysql-error_copy-2022-07-29-000001` to match the renamed data stream. This means that for the given `follow_index_pattern` of `{{leader_index}}_copy` the index changes look like: | Leader Cluster | Follower Cluster | |--------------|-----------| | `logs-mysql-error` (data stream) | `logs-mysql-error_copy` (data stream) | | `.ds-logs-mysql-error-2022-07-29-000001` | `.ds-logs-mysql-error_copy-2022-07-29-000001` | Which internally means the auto-follow request turned into the create follower request of: ``` PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": ".ds-logs-mysql-error-2022-07-29-000001", "data_stream_name": "logs-mysql-error_copy" } ``` Relates to elastic#84940 (cherry-picked the commit for a test) Relates to elastic#61993 (where data stream support was first introduced for CCR) Resolves elastic#81751 (cherry picked from commit 3420be0)

This disabled backwards compatibility tests so the CCR bugfix can be backported, since it includes some version-specific serialization.

Updates versions for backport (elastic#88995) of original fix (elastic#88875) Relates to elastic#88997

This commit fixes the situation where a user wants to use CCR to replicate indices that are part of a data stream while renaming the data stream. For example, assume a user has an auto-follow request that looks like this: ``` PUT /_ccr/auto_follow/my-auto-follow-pattern { "remote_cluster" : "other-cluster", "leader_index_patterns" : ["logs-*"], "follow_index_pattern" : "{{leader_index}}_copy" } ``` And then the data stream `logs-mysql-error` was created, creating the backing index `.ds-logs-mysql-error-2022-07-29-000001`. Prior to this commit, replicating this data stream means that the backing index would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` and the data stream would *not* be renamed. This caused a check to trip in `TransportPutLifecycleAction` asserting that a backing index was not renamed for a data stream during following. After this commit, there are a couple of changes: First, the data stream will also be renamed. This means that the `logs-mysql-error` becomes `logs-mysql-error_copy` when created on the follower cluster. Because of the way that CCR works, this means we need to support renaming a data stream for a regular "create follower" request, so a new parameter has been added: `data_stream_name`. It works like this: ``` PUT /mynewindex/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": "myotherindex", "data_stream_name": "new_ds" } ``` Second, the backing index for a data stream must be renamed in a way that does not break the parsing of a data stream backing pattern, whereas previously the index `.ds-logs-mysql-error-2022-07-29-000001` would be renamed to `.ds-logs-mysql-error-2022-07-29-000001_copy` (an illegal name since it doesn't end with the rollover digit), after this commit it will be renamed to `.ds-logs-mysql-error_copy-2022-07-29-000001` to match the renamed data stream. This means that for the given `follow_index_pattern` of `{{leader_index}}_copy` the index changes look like: | Leader Cluster | Follower Cluster | |--------------|-----------| | `logs-mysql-error` (data stream) | `logs-mysql-error_copy` (data stream) | | `.ds-logs-mysql-error-2022-07-29-000001` | `.ds-logs-mysql-error_copy-2022-07-29-000001` | Which internally means the auto-follow request turned into the create follower request of: ``` PUT /.ds-logs-mysql-error_copy-2022-07-29-000001/_ccr/follow { "remote_cluster": "other-cluster", "leader_index": ".ds-logs-mysql-error-2022-07-29-000001", "data_stream_name": "logs-mysql-error_copy" } ``` Relates to #84940 (cherry-picked the commit for a test) Relates to #61993 (where data stream support was first introduced for CCR) Resolves #81751

…#88998) Updates versions for backport (#88995) of original fix (#88875) Relates to #88997

dakrone marked this pull request as draft July 27, 2022 22:00

elasticsearchmachine added the v8.5.0 label Jul 27, 2022

dakrone force-pushed the fix-ccr-ds-rename branch 2 times, most recently from 840d570 to 32e82d7 Compare July 28, 2022 17:51

martijnvg and others added 2 commits July 29, 2022 10:45

Add failing test for data stream renaming through CCR

801decd

Relates to elastic#81751

dakrone force-pushed the fix-ccr-ds-rename branch from 15b9fb4 to e1d83ca Compare July 29, 2022 16:59

dakrone changed the title ~~WIP For CCR Fix~~ Fix renaming data streams with CCR replication Jul 29, 2022

dakrone marked this pull request as ready for review July 29, 2022 17:00

elasticsearchmachine added the needs:triage Requires assignment of a team area label label Jul 29, 2022

dakrone added >bug :Distributed/CCR Issues around the Cross Cluster State Replication features :Data Management/Data streams Data streams and their lifecycles v8.4.0 and removed needs:triage Requires assignment of a team area label labels Jul 29, 2022

elasticsearchmachine added Team:Data Management Meta label for data/management team Team:Distributed Meta label for distributed team labels Jul 29, 2022

Update docs/changelog/88875.yaml

9ee7adf

dakrone added :Distributed/CCR Issues around the Cross Cluster State Replication features and removed :Distributed/CCR Issues around the Cross Cluster State Replication features labels Jul 29, 2022

Fix changelog

1d3a7f5

dakrone mentioned this pull request Jul 29, 2022

Address CCR auto follow bug when auto following data streams #84940

Closed

gmarouli self-requested a review July 29, 2022 18:00

jbaiera approved these changes Jul 29, 2022

View reviewed changes

gmarouli approved these changes Aug 1, 2022

View reviewed changes

dakrone and others added 4 commits August 1, 2022 08:09

Add a test for an unparseable data stream backing index

e3001d5

Merge remote-tracking branch 'origin/main' into fix-ccr-ds-rename

108eb5e

Fix a typo in the docs

2789600

Co-authored-by: Mary Gouseti <mgouseti@gmail.com>

Whoops, typo in test

00bdaba

dakrone merged commit 3420be0 into elastic:main Aug 1, 2022

dakrone mentioned this pull request Aug 1, 2022

[8.4] Fix renaming data streams with CCR replication (#88875) #88995

Merged

dakrone added a commit to dakrone/elasticsearch that referenced this pull request Aug 1, 2022

Disable BWC tests for backport of elastic#88875

fc753b8

This disabled backwards compatibility tests so the CCR bugfix can be backported, since it includes some version-specific serialization.

dakrone added a commit to dakrone/elasticsearch that referenced this pull request Aug 1, 2022

Update version serialization for CCR backport

a6b45b7

Updates versions for backport (elastic#88995) of original fix (elastic#88875) Relates to elastic#88997

dakrone mentioned this pull request Aug 1, 2022

Update version serialization for CCR backport and re-enable BWC tests #88998

Merged

dakrone added a commit that referenced this pull request Aug 1, 2022

Disable BWC tests for backport of #88875 (#88997)

6121a8a

elasticsearchmachine pushed a commit that referenced this pull request Aug 1, 2022

Update version serialization for CCR backport and re-enable BWC tests (…

857fe2e

…#88998) Updates versions for backport (#88995) of original fix (#88875) Relates to #88997

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix renaming data streams with CCR replication #88875

Fix renaming data streams with CCR replication #88875

dakrone commented Jul 27, 2022 •

edited

dakrone commented Jul 28, 2022

elasticsearchmachine commented Jul 29, 2022

elasticsearchmachine commented Jul 29, 2022

elasticsearchmachine commented Jul 29, 2022

dakrone commented Jul 29, 2022

jbaiera left a comment

gmarouli left a comment

gmarouli Aug 1, 2022

dakrone Aug 1, 2022

gmarouli Aug 1, 2022

dakrone commented Aug 1, 2022

Fix renaming data streams with CCR replication #88875

Fix renaming data streams with CCR replication #88875

Conversation

dakrone commented Jul 27, 2022 • edited

dakrone commented Jul 28, 2022

elasticsearchmachine commented Jul 29, 2022

elasticsearchmachine commented Jul 29, 2022

elasticsearchmachine commented Jul 29, 2022

dakrone commented Jul 29, 2022

jbaiera left a comment

Choose a reason for hiding this comment

gmarouli left a comment

Choose a reason for hiding this comment

gmarouli Aug 1, 2022

Choose a reason for hiding this comment

dakrone Aug 1, 2022

Choose a reason for hiding this comment

gmarouli Aug 1, 2022

Choose a reason for hiding this comment

dakrone commented Aug 1, 2022

💚 All backports created successfully

Questions ?

dakrone commented Jul 27, 2022 •

edited