Elasticsearch won't start in Kubernetes

[drewr]

Trying to run Elasticsearch 6.3.1 on GKE results in this warning, which is seemingly enough to prevent the container from starting?

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Command:

kubectl run elasticsearch --image docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.1 --port 9200

The pod looks like:

    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Sat, 14 Jul 2018 13:08:01 -0500
      Finished:     Sat, 14 Jul 2018 13:08:04 -0500

[dliappis]

@drewr It's not likely that this is the reason the container terminates; this is just a warning due to the deprecation of CMS, since java-9[1], and it doesn't cause failures elsewhere eg. with normal docker run etc.

I quickly tried your command in my minikube environment and indeed starting failed, but checking the logs shows a different reason:

ERROR: [1] bootstrap checks failed                                                                
[1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]  

Then I used the env var to bypass the bootstrap checks:

kubectl run elasticsearch --image docker.elastic.co/elasticsearch/elasticsearch-oss:6.3.1 --port 9200 --env "discovery.type=single-node"

Retrieved the name of the pod with:

kubectl get pods --all-namespaces

Checked the logs:

kubectl logs elasticsearch-7d879c56f5-vjssd

still showing the warning about CMS deprecation and the bootstrap check vm.max_map_count value, but the latter not terminating elasticsearch

Finally to expose the port locally:

kubectl expose deployment elasticsearch --type=NodePort

And retrieve the URL with:

$ minikube service elasticsearch --url
http://192.168.99.100:30828

And hitting this gave me our familiar:

$ curl http://192.168.99.100:30828
{
  "name" : "oXS6l8T",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "RmsNXQN6Tiad0c0WbRVBKA",
  "version" : {
    "number" : "6.3.1",
    "build_flavor" : "oss",
    "build_type" : "tar",
    "build_hash" : "eb782d0",
    "build_date" : "2018-06-29T21:59:26.107521Z",
    "build_snapshot" : false,
    "lucene_version" : "7.3.1",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}

[1] http://openjdk.java.net/jeps/291

[jarpy]

I got the same behaviour by memory starving the system. The only log message was the UseConcMarkSweepGC deprecation warning. I suspect that message is emitted very early, before the JVM has allocated the heap and (in my case) triggered the OOM killer.

[drewr]

Thanks @dliappis! Experimented a bit more and stumbled on this at the perfect kubectl get pod moment:

elasticsearch-7d879c56f5-gkspv   0/1       OOMKilled   1          1m

which confirms @jarpy suspicion. I had relied on the GKE interface which didn't tell me much.

Using kubectl run it works. I tried to run it directly as a pod before, which must have different semantics than a deployment, and wasn't picking up my environment changes. I'll try that next.

[drewr]

The next hurdle was that I was trying to use a PersistentVolume and running into AccessDeniedException, because I hadn't provided securityContext with the proper config. I've done that now and have a fully functioning, single-node lightweight ES instance. Full config here.