Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Removing remote cluster that is in use by auto-follow pattern causes crash #36667

Closed
jen-huang opened this issue Dec 15, 2018 · 2 comments
Closed

Removing remote cluster that is in use by auto-follow pattern causes crash #36667

jen-huang opened this issue Dec 15, 2018 · 2 comments
Assignees
@martijnvg
Labels
blocker :Distributed/CCR Issues around the Cross Cluster State Replication features

Comments

@jen-huang
Copy link

I have a remote cluster which is in use by a CCR auto-follow pattern:

GET _remote/info:

{
  "prod2" : {
    "seeds" : [
      "127.0.0.1:9400"
    ],
    "connected" : true,
    "num_nodes_connected" : 1,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

GET _ccr/auto_follow:

{
  "patterns" : [
    {
      "name" : "pattern1",
      "pattern" : {
        "remote_cluster" : "prod2",
        "leader_index_patterns" : [
          "test*"
        ],
        "follow_index_pattern" : "{{leader_index}}"
      }
    }
  ]
}

When I remove the remote cluster, ES throws repeated errors until it crashes with ERROR ES exited with code 126:

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "prod2": {
          "seeds" : null,
          "skip_unavailable": null
        }
      }
    }
  }
}

   │ info [o.e.x.c.a.AutoFollowCoordinator] [Jens-MacBook-Pro.local] failure occurred while fetching cluster state for auto follow pattern [pattern1]
   │      java.lang.IllegalArgumentException: unknown cluster alias [prod2]
   │      	at org.elasticsearch.transport.RemoteClusterService.getRemoteClusterClient(RemoteClusterService.java:495) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.client.node.NodeClient.getRemoteClusterClient(NodeClient.java:123) ~[elasticsearch-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch
   │ info .xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda$getRemoteClusterState$0(AutoFollowCoordinator.java:165) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.CcrLicenseChecker.checkRemoteClusterLicenseAndFetchClusterState(CcrLicenseChecker.java:178) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.getRemoteClusterState(AutoFollowCoordinator.java:161) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.start(AutoFollowCoordinator.java:275) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.finalise(AutoFollowCoordinator.java:407) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$AutoFollower.lambda$start$1(AutoFollowCoordinator.java:289) [x-pack-ccr-7.0.0-SNAPSHOT.jar:7.0.0-SNAPSHOT]
   │      	at org.elasticsearch.xpack.ccr.action.AutoFollowCoordinator$2.lambda
@cjcenizal cjcenizal added blocker :Distributed/CCR Issues around the Cross Cluster State Replication features labels Dec 15, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@cjcenizal
Copy link
Contributor

CC @jasontedor

@martijnvg martijnvg self-assigned this Dec 15, 2018
@jasontedor jasontedor mentioned this issue Dec 15, 2018
Merged
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Dec 16, 2018
@martijnvg
Make CCR resilient against missingremote cluster connections
12f7952 
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behaviour similar how CCR
deals with authorization errors.

Closes elastic#36667
Closes elastic#36255
@martijnvg martijnvg mentioned this issue Dec 16, 2018
Merged
martijnvg added a commit that referenced this issue Dec 24, 2018
@martijnvg
Make CCR resilient against missing remote cluster connections (#36682)
4fb62fc 
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behavior similar how CCR
deals with authorization errors.

Closes #36667
Closes #36255
martijnvg added a commit that referenced this issue Dec 24, 2018
@martijnvg
Make CCR resilient against missing remote cluster connections (#36682)
db4b2ff 
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behavior similar how CCR
deals with authorization errors.

Closes #36667
Closes #36255
martijnvg added a commit that referenced this issue Dec 24, 2018
@martijnvg
Make CCR resilient against missing remote cluster connections (#36682)
46dfbb3 
Both index following and auto following should be resilient against missing remote connections.
This happens in the case that they get accidentally removed by a user. When this happens
auto following and index following will retry to continue instead of failing with unrecoverable exceptions.

Both the put follow and put auto follow APIs validate whether the
remote cluster connection. The logic added in this change only exists
in case during the lifetime of a follower index or auto follow pattern
the remote connection gets removed. This retry behavior similar how CCR
deals with authorization errors.

Closes #36667
Closes #36255
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@martijnvg martijnvg

Labels
blocker :Distributed/CCR Issues around the Cross Cluster State Replication features
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@martijnvg @cjcenizal @jen-huang @elasticmachine