Kibana reported that ES is not working in EC2

[neubineshkuttan]

Hi,
I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all.

I also get multiple errors on x-pack services with the same errors on license information.

What else is needed to check in order to get it working?

[synhershko]

Please provide the configuration you used to deploy this

[AAber]

Hi,
Got the same issue, the problem is that the discovery-ec2 plugin is not working.

As a temp fix you can remove the ec2 discovery:
network.host: _ec2:privateIpv4_,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
zen.hosts_provider: ec2
ec2.groups: sg-XXXXXXXXXXXXXXXX
ec2.host_type: private_ip
ec2.tag.Cluster: default-staging-es-cluster
ec2.availability_zones: eu-west-1a,eu-west-1b,eu-west-1c
ec2.protocol: http # no need in HTTPS for internal AWS calls

And setup simple discovery:
network.host: _site_,localhost
discovery.zen.ping.unicast.hosts: ["ip-172-30-2-22.eu-west-1.compute.internal", "ip-172-30-0-139.eu-west-1.compute.internal", "ip-172-30-1-64.eu-west-1.compute.internal", "ip-172-30-1-66.eu-west-1.compute.internal", "ip-172-30-0-217.eu-west-1.compute.internal"]

Let me know what configuration files are needed to debug the issue / ec2 discovery, and I will post.

[neubineshkuttan]

Thanks for the info. I will check this. Any permanent solution for this problem?

[neubineshkuttan]

Please provide the configuration you used to deploy this.
network.host: site,localhost
discovery.zen.ping.unicast.hosts: ["ip-172-30-2-22.eu-west-1.compute.internal", "ip-172-30-0-139.eu-west-1.compute.internal", "ip-172-30-1-64.eu-west-1.compute.internal", "ip-172-30-1-66.eu-west-1.compute.internal", "ip-172-30-0-217.eu-west-1.compute.internal"]

@synhershko - The issue got resolved with the above change. Is this an expected manual change?

please refer the config details.

bootstrap.memory_lock: true
node.name: ${HOSTNAME}

action.destructive_requires_name: true
indices.fielddata.cache.size: 1% # default is unbounded
cluster.name: ELKCluster

discovery.zen.minimum_master_nodes: 2

only data nodes should have ingest and http capabilities

node.master: true
node.data: false
node.ingest: false
http.enabled: false
xpack.security.enabled: false
xpack.monitoring.enabled: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

network.host: ec2:privateIpv4,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
zen.hosts_provider: ec2
ec2.groups: sg-xxxxxxxxxxxxxxxxx
ec2.host_type: private_ip
ec2.tag.Cluster: default-ELKCluster
ec2.availability_zones: us-east-1a,us-east-1b,us-east-1c,us-east-1e
ec2.protocol: http # no need in HTTPS for internal AWS calls

[kgosietsileM]

Hi,
I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all.

I also get multiple errors on x-pack services with the same errors on license information.

What else is needed to check in order to get it working?

Hi, @neubineshkuttan

I also experienced the same issue, I found that the discovery plugin defaults to ec2.us-east-1.amazonaws.com as the EC2 endpoint if
the EC2 endpoint for discovery is not explicitly set.

elastic/elasticsearch#27464

My ES Cluster was up and running after I explicitly set the region for my es cluster in my elasticseach.yml configuration. Please see the example below.

    zen.hosts_provider: ec2
    ec2.groups: ${security_groups}
    ec2.endpoint: ec2.eu-west-1.amazonaws.com
    ec2.host_type: private_ip
    ec2.tag.Cluster: ${es_environment}
    ec2.availability_zones: ${availability_zones}
    ec2.protocol: http # no need in HTTPS for internal AWS calls```

[AAber]

Super cool!!
I updated my config and now it works like a charm.

Thx.

[synhershko]

I'm thinking we should add this to user_data.sh:

ec2.endpoint: ec2.${aws_region}.amazonaws.com

where aws_region gets the value from var.aws_region

thoughts?

[synhershko]

Also, this issue is already open as #50

[neubineshkuttan]

@synhershko , If ec2.endpoint: ec2.${aws_region}.amazonaws.com is included as of user_data.sh then the "commented on Sep 21" will be resolved?

[neubineshkuttan]

Hi,
I've deployed the TF in my AWS environment and when I logged in to Kibana I found out that ES is not working at all.
I also get multiple errors on x-pack services with the same errors on license information.
What else is needed to check in order to get it working?

Hi, @neubineshkuttan

I also experienced the same issue, I found that the discovery plugin defaults to ec2.us-east-1.amazonaws.com as the EC2 endpoint if
the EC2 endpoint for discovery is not explicitly set.

elastic/elasticsearch#27464

My ES Cluster was up and running after I explicitly set the region for my es cluster in my elasticseach.yml configuration. Please see the example below.

    zen.hosts_provider: ec2
    ec2.groups: ${security_groups}
    ec2.endpoint: ec2.eu-west-1.amazonaws.com
    ec2.host_type: private_ip
    ec2.tag.Cluster: ${es_environment}
    ec2.availability_zones: ${availability_zones}
    ec2.protocol: http # no need in HTTPS for internal AWS calls```
``

Hi @synhershko , @kgosietsileM , @AAber

After adding the endpoint, the issue still exist. please refer the below configs.

network.host: ec2:privateIpv4,localhost
plugin.mandatory: discovery-ec2
cloud.node.auto_attributes: true
cluster.routing.allocation.awareness.attributes: aws_availability_zone
discovery:
zen.hosts_provider: ec2
ec2.groups: sg-XXXXXXXXXXXXXXXXX
ec2.endpoint: ec2.us-west-2.amazonaws.com
ec2.host_type: private_ip
ec2.tag.Cluster: xxxxxxxxxxxxxxxxxxxxxxx
ec2.availability_zones: us-west-2a,us-west-2b,us-west-2c
ec2.protocol: http # no need in HTTPS for internal AWS calls

Few other observations,

1. Elasticsearch service still fails.
2. Error in log file as below ---

---

[ERROR][o.e.b.Bootstrap ] Exception
java.lang.IllegalStateException: missing mandatory plugins [discovery-ec2], found plugins []
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:177) ~[elasticsearch-6.4.2.jar:6.4.2]

[yibinlin]

Maybe related: in packer this step seemed failing in master branch:

amazon-ebs: Provisioning with shell script: install-cloud-plugin.sh
amazon-ebs: grep: /var/lib/dhcp/dhclient.eth0.leases: No such file or directory

[BenAHammond]

This worked for @neubineshkuttan and I. Thank you for the fix.

[tsimonyan]

It seems issue was in cloud plugin not being installed. Should be fixed in master version.