[BUG] Failed dashboards indices for autofollow with all indices pattern

[defesteban]

What is the bug?
It is forbidden to replicate indices starting from . in OpenSearch, but Dashboards indices start from . and are not system. So, there is no ability to run successful (without failed_indices) autofollow for all indices (*) with installed dashboards.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Install Dashboards along with OpenSearch on primary and secondary clusters.
2. Run autofollow replication on the secondary cluster

curl -XPOST -k -H 'Content-Type: application/json' 'https://localhost:9200/_plugins/_replication/_autofollow' -d '
{
  "leader_alias": "leader-cluster",
  "pattern": "*",
  "name": "replication",
  "use_roles": {
    "leader_cluster_role": "all_access",
    "follower_cluster_role": "all_access"
  }
}'

3. Login to dashboards on primary cluster with admin username and make some actions: create own dashboard, make a backup, etc.
4. Check indices on primary cluster (GET _cat/indices?v):

health status index                                                                     uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   test                                                                      1BUqHYGmRPCW0AO-ix8tuA   1   1          2            0      7.4kb          3.7kb
green  open   .kibana_92668751_admin_1                                                  I7fIY7F7StmRgh9ZaQhREw   1   1          8            1     60.8kb         30.4kb
green  open   .kibana_1                                                                 mvJuKuevQViCoZS33_r2yg   1   1          0            0       416b           208b
green  open   .opendistro_security                                                      qg36BdInQQS5ACC4qu3ROQ   1   2         10            3    203.9kb         67.9kb
green  open   .opensearch-notifications-config                                          HHDun-gLSNWY3UyyPTZytw   1   2          0            0       624b           208b

5. Check autofollow statistics on secondary cluster (GET _plugins/_replication/autofollow_stats):

{
  "num_success_start_replication" : 1,
  "num_failed_start_replication" : 2,
  "num_failed_leader_calls" : 0,
  "failed_indices" : [
    ".opensearch-notifications-config",
    ".kibana_92668751_admin_1"
  ],
  "autofollow_stats" : [
    {
      "name" : "replication",
      "pattern" : "*",
      "num_success_start_replication" : 1,
      "num_failed_start_replication" : 2,
      "num_failed_leader_calls" : 0,
      "failed_indices" : [
        ".opensearch-notifications-config",
        ".kibana_92668751_admin_1"
      ],
      "last_execution_time" : 1684501855228
    }
  ]
}

6. Check OpenSearch logs on secondary cluster:

[2023-05-19T15:44:28,651][INFO ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Auto follow starting replication from leader-cluster:.opensearch-notifications-config -> .opensearch-notifications-config
[2023-05-19T15:44:28,651][WARN ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Failed to start replication for leader-cluster:.opensearch-notifications-config -> .opensearch-notifications-config.
org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: Value .opensearch-notifications-config must not start with '.';2: Value .opensearch-notifications-config must not start with '.';
 at org.opensearch.replication.action.index.ReplicateIndexRequest.validate(ReplicateIndexRequest.kt:94) ~[?:?]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:177) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invokeSuspend(Coroutines.kt:89) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at org.opensearch.replication.util.CoroutinesKt.suspendExecute(Coroutines.kt:87) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt.suspendExecute$default(Coroutines.kt:84) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplication(AutoFollowTask.kt:202) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplicationJobs(AutoFollowTask.kt:175) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.access$startReplicationJobs(AutoFollowTask.kt:49) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask$startReplicationJobs$1.invokeSuspend(AutoFollowTask.kt) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
 at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
 at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.4.1.jar:2.4.1]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
 at java.lang.Thread.run(Thread.java:833) [?:?]

[2023-05-19T15:44:58,661][INFO ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Auto follow starting replication from leader-cluster:.kibana_92668751_admin_1 -> .kibana_92668751_admin_1
[2023-05-19T15:44:58,661][WARN ][o.o.r.t.a.AutoFollowTask ] [opensearch-1][leader-cluster] Failed to start replication for leader-cluster:.kibana_92668751_admin_1 -> .kibana_92668751_admin_1.
org.opensearch.action.ActionRequestValidationException: Validation Failed: 1: Value .kibana_92668751_admin_1 must not start with '.';2: Value .kibana_92668751_admin_1 must not start with '.';
 at org.opensearch.replication.action.index.ReplicateIndexRequest.validate(ReplicateIndexRequest.kt:94) ~[?:?]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:177) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:461) ~[opensearch-2.4.1.jar:2.4.1]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invokeSuspend(Coroutines.kt:89) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt$suspendExecute$4.invoke(Coroutines.kt) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:89) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.BuildersKt.withContext(Unknown Source) ~[kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at org.opensearch.replication.util.CoroutinesKt.suspendExecute(Coroutines.kt:87) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.util.CoroutinesKt.suspendExecute$default(Coroutines.kt:84) ~[opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplication(AutoFollowTask.kt:202) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.startReplicationJobs(AutoFollowTask.kt:175) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask.access$startReplicationJobs(AutoFollowTask.kt:49) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at org.opensearch.replication.task.autofollow.AutoFollowTask$startReplicationJobs$1.invokeSuspend(AutoFollowTask.kt) [opensearch-cross-cluster-replication-2.4.1.0.jar:2.4.1.0]
 at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
 at kotlinx.coroutines.UndispatchedCoroutine.afterResume(CoroutineContext.kt:147) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlinx.coroutines.AbstractCoroutine.resumeWith(AbstractCoroutine.kt:102) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:46) [kotlin-stdlib-1.6.0.jar:1.6.0-release-798(1.6.0)]
 at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106) [kotlinx-coroutines-core-jvm-1.6.0.jar:?]
 at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:747) [opensearch-2.4.1.jar:2.4.1]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
 at java.lang.Thread.run(Thread.java:833) [?:?]

What is the expected behavior?
Skip indices starting from . symbol, because they are already forbidden.

What is your host/environment?

• OpenSearch Version - 2.4.1
• Kubernetes Version - 1.23.6

[soosinha]

Can you confirm what is your requirement:

• Do you want to replicate the dashboard indices as well or
• Do you want that replication should not happen for these indices but the validation exception should not be printed either ?

[defesteban]

OpenSearch has a lot of indices that start from .. As I understand, they are all system and do not cause any problems with replication. All Dashboards indices I've seen are prefixed with .. Does it mean they are system? If they aren't, why do they start with .?
Also, is it okay to replicate Dashboards indices? Could the Dashboards start with replicated indices successfully?

For now we don't need to replicate Dashboards indices, but it would be good to have solution according to OpenSearch conception.