Do not use elasticsearch-cloud-kubernetes plugin

[pavolloffay]

Can we remove https://github.com/fabric8io/elasticsearch-cloud-kubernetes plugin for unicast discovery?

For example https://github.com/pires/kubernetes-elasticsearch-cluster is using Zen discovery https://www.elastic.co/guide/en/elasticsearch/reference/5.6/modules-discovery-zen.html

There is https://github.com/pires/kubernetes-elasticsearch-cluster/blob/master/es-discovery-svc.yaml which points to master nodes (there is no es-master service). Then all nodes define discovery.zen.ping.unicast.hosts pointing to this service.

Some links:

• https://github.com/pires/kubernetes-elasticsearch-cluster
• https://github.com/pires/docker-elasticsearch-kubernetes
• https://github.com/pires/docker-elasticsearch

[lukas-vlcek]

Some discussion about this topic has been happening here as well: openshift/origin-aggregated-logging#986

[pavolloffay]

I have done some testing with https://github.com/pires/kubernetes-elasticsearch-cluster which does not use elasticsearch-cloud-kubernetes plugin.

I have been doing

• scaling up
• scaling down
• manually deleting master or data nodes

http://localhost:9200/_cat/nodes and http://localhost:9200/_cluster/health?pretty always showed the correct number and list of nodes.

For example:

172.17.0.5 26 69 4 1.29 1.32 1.00 mi - elasticsearch-master-3467712756-5lsdf
172.17.0.6 29 69 4 1.29 1.32 1.00 di - elasticsearch-1
172.17.0.2 27 69 4 1.29 1.32 1.00 mi - elasticsearch-master-3467712756-9rz78
172.17.0.4 31 69 4 1.29 1.32 1.00 di - elasticsearch-0
172.17.0.3 28 69 4 1.29 1.32 1.00 mi * elasticsearch-master-3467712756-jz7vc

When elected master node was deleted the urls http://localhost:9200/_cat/nodes and http://localhost:9200/_cluster/health?pretty where not responding until new master was elected. For example url http://localhost:9200/_nodes/_local?pretty always responded fine.

@lukas-vlcek do you know any other use cases which should be tested to verify that the plugin is not needed?

[pavolloffay]

I have done the same testing with #22 and the behavior was the same. Even master election was much quicker.

[pavolloffay]

interesting comments from pires/kubernetes-elasticsearch-cluster#69, implemented in pires/docker-elasticsearch-kubernetes@d1907da

headless service + round robin DNS
Elasticsearch is able to use a round robin DNS assisted discovery. It's possible to use a headless service DNS A record as the discovery.zen.ping.unicast.hosts parameter. This doesn't need 3rd party discovery plugin anymore.

[ruromero]

@pavolloffay can you forward-port this feature to master?

[pavolloffay]

yes will do

[pavolloffay]

A documentation on headless services:
https://kubernetes.io/docs/concepts/services-networking/service/#headless-services
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#services

I think we should also set clusterIP: None in elasticsearch-cluster service

[pavolloffay]

@wozniakjan hi, if you have some time your feedback would be appreciated.

[wozniakjan]

@pavolloffay it is definitely worth exploring. Some time ago I tried to compare two approaches

1. mentioned elasticsearch-cloud-kubernetes plugin
2. current approach in kubernetes

and described why I think it may be more robust to have the plugin in openshift/origin-aggregated-logging#986 as @lukas-vlcek already mentioned. I am not entirely sure how headless service works with readiness and liveness probes, and even less sure if we need the probes with ES after all. Maybe an init container could serve the purpose we need.

It is usually the broken clusters where we get to see shortcomings of certain solutions. I think @portante has a lot of experience with debugging broken clusters, perhaps he has an idea how to stress test this

[pavolloffay]

@ruromero could you please reopen?

I would like to keep it open, I am planning to do more tests

[pavolloffay]

Closing fabric8-es-plugin also advises to use headless service https://github.com/fabric8io/elasticsearch-cloud-kubernetes#kubernetes-cloud-plugin-for-elasticsearch