Remote cross cluster search stops working...

[asatsi]

Elasticsearch version (bin/elasticsearch --version):
$ bin/elasticsearch --version
Version: 6.2.2, Build: 10b1edd/2018-02-16T19:01:30.685723Z, JVM: 1.8.0_161

Plugins installed:
No external plugins installed.

JVM version (java -version):
"1.8.0_161"

OS version (uname -a if on a Unix-like system):
Linux myhostname 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Dec 28 14:23:39 EST 2017 x86_64 x86_64 x86_64 GNU/Linux

Description of the problem including expected versus actual behavior:
We have two elasticsearch clusters(each 16 nodes, 4 nodes per machine, in all 4 machines per cluster) configured across 8 machines. They are joined together by a third cluster configured as a cross elasticsearch cluster.

$ curl -s -XGET http://192.168.97.15:9400/_cluster/settings?pretty
{
"persistent" : {
"search" : {
"remote" : {
"prod1_cluster" : {
"seeds" : [
"192.168.97.15:9301",
"192.168.97.16:9301",
"192.168.97.26:9301",
"192.168.97.27:9301"
]
},
"prod2_cluster" : {
"seeds" : [
"192.168.181.15:9300",
"192.168.181.16:9300",
"192.168.181.26:9300",
"192.168.181.27:9300"
]
}
}
}
},
"transient" : { }
}

I can create the indices in kibana for two of these clusters initially. However after sometime the search for prod2_cluster (prod2_cluster:myindex_on_prod2) times out. Search for prod1_cluster always work. Also noticed that even though the _cluster/settings give a nice response. _remote/info errors out as below:

$ curl -s -XGET "http://192.168.97.15:9400/_remote/info?pretty"
{
"error" : {
"root_cause" : [
{
"type" : "node_disconnected_exception",
"reason" : "[node-es4-1][192.168.181.27:9300][cluster:monitor/nodes/info] disconnected"
}
],
"type" : "node_disconnected_exception",
"reason" : "[node-es4-1][192.168.181.27:9300][cluster:monitor/nodes/info] disconnected"
},
"status" : 500
}

All ports are opened and there is no connectivity issue.

Steps to reproduce:
Setup total three clusters. Two clusters are the actual clusters holding data and a third cluster which is used to "join" these two as part of a cross cluster search.

[elasticmachine]

Pinging @elastic/es-search-aggs

[javanna]

hi @asatsi did you find any related errors in the logs? We would need to know what the reason for the disconnection is. Hopefully you can find that in the logs and provide its stacktrace. Thanks!

[asatsi]

Hi @javanna ,

I do see many occurrences of the below log. I have snipped specific portion in the attached file.

error.log

[asatsi]

bump! Could somebody help out?

[javanna]

hi @asatsi from your logs, it seems like there are no nodes available that belong to one of the remote clusters. Note that the connection between the clusters uses the transport port (default 9300). Maybe there is some connection problem?

[asatsi]

Thanks @javanna for reviewing the logs. There is apparently no issue with the connectivity as the firewall ports have been specifically opened for all the required ports. I also observed that whenever the "proxy cross cluster" is restarted, the remote connection behaves good. In Kibana I can see logs/dashboards from both the remote clusters. But after sometime, one of the remote cluster starts giving timeouts in Kibana.

[athanatos64]

I'm having the same problem as well. I though it maybe related to x-pack security since only a few users have the permissions to search remote clusters. If I'm not logged in and constantly searching remote indices, the connection timeouts.

[asatsi]

I had to to restart my cross elastic server whenever this happens. And it happens very frequently. I do not have shield or x-pack installed.

…

Sent from my iPhone On 22-May-2018, at 7:45 PM, athanatos64 <notifications@github.com<mailto:notifications@github.com>> wrote: I'm having the same problem as well. I though it maybe related to x-pack security since only a few users have the permissions to search remote clusters. If I'm not logged in and constantly searching remote indices, the connection timeouts. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#30247 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKJ8LWfTWjuaJMCsyyJCY2UhMhveZ0W3ks5t1B2RgaJpZM4TsOnH>.

[athanatos64]

I usually have to do GET /_remote/info from Console and wait until I get connection restored back to search the remote cluster. Good to know that it's not X-pack related.

[asatsi]

For now I am using two different kibana to point to the two clusters.

[danfisk]

What is the default time out for 'transient:' ? I have this working but do not use any kind of proxy server. Also I setup the seeds as persistent: In this way they are active until the next time the cluster is restarted. If you want them to always stay active, place the seeds in he elasticsearch.yml

[luxiaoxun]

@asatsi
Try "skip_unavailable": "true" to see if this configuration helped.

[javanna]

See #34405 , I believe the outlined solutions will help with this issue. Stay tuned for a fix. I am not sure that skip_unavailable helps as in some cases we are not quick enough detecting network disconnections and due to firewalls dropping connections in-between, we may have CCS searches hanging which is a problem regardless of the skip_unavailable setting value.

[javanna]

I am closing this issue in favor of #34405 . From 6.6 it will be possible to enable transport pings directed to remote clusters. We are also working on other improvements that should allow to better detect network issues and react, which are listed in the linked issue.