Skip to content
This repository has been archived by the owner on Nov 7, 2018. It is now read-only.

containers not booting. #88

Closed
marcofranssen opened this issue Apr 7, 2017 · 6 comments
Closed

containers not booting. #88

marcofranssen opened this issue Apr 7, 2017 · 6 comments

Comments

@marcofranssen
Copy link

marcofranssen commented Apr 7, 2017 •
edited

Hi, I came accross your deployments for elasticsearch. I have been using following deployment on our own coreos kubernetes cluster running on vcloud.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: es-master
  namespace: dev-vnext
  labels:
    vnext-infra: elasticsearch
    role: master
spec:
  replicas: 3
  template:
    metadata:
      labels:
        vnext-infra: elasticsearch
        role: master
      annotations:
        pod.beta.kubernetes.io/init-containers: '[
            {
              "name": "sysctl",
              "image": "busybox",
              "imagePullPolicy": "IfNotPresent",
              "command": ["sysctl", "-w", "vm.max_map_count=262144"],
              "securityContext": {
                "privileged": true
              }
            }
          ]'
    spec:
      containers:
      - name: es-master
        image: quay.io/pires/docker-elasticsearch-kubernetes:5.3.0
        imagePullPolicy: Always
        securityContext:
          privileged: false
          capabilities:
            add:
              - IPC_LOCK
              - SYS_RESOURCE
        resources:
          requests:
            memory: 1Gi
          limits:
            memory: 1Gi
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        - name: CLUSTER_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: NUMBER_OF_MASTERS
          value: "2"
        - name: NODE_MASTER
          value: "true"
        - name: NODE_INGEST
          value: "false"
        - name: HTTP_ENABLE
          value: "false"
        - name: ES_JAVA_OPTS
          value: -Xms512m -Xmx512m
        ports:
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - mountPath: /data
          name: esdata
      volumes:
      - name: esdata
        emptyDir: {}

Unfortunatily it gives following error. I have only been trying the master deployment so far.

2017-04-07T13:52:54.550941455Z [2017-04-07T13:52:54,550][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [aggs-matrix-stats]
2017-04-07T13:52:54.550992375Z [2017-04-07T13:52:54,550][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [ingest-common]
2017-04-07T13:52:54.550998737Z [2017-04-07T13:52:54,550][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [lang-expression]
2017-04-07T13:52:54.551235577Z [2017-04-07T13:52:54,550][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [lang-groovy]
2017-04-07T13:52:54.551248907Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [lang-mustache]
2017-04-07T13:52:54.551443727Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [lang-painless]
2017-04-07T13:52:54.551453404Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [percolator]
2017-04-07T13:52:54.551689677Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [reindex]
2017-04-07T13:52:54.551713190Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [transport-netty3]
2017-04-07T13:52:54.551960178Z [2017-04-07T13:52:54,551][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] loaded module [transport-netty4]
2017-04-07T13:52:54.552645842Z [2017-04-07T13:52:54,552][INFO ][o.e.p.PluginsService     ] [es-master-22305778-5t845] no plugins loaded
2017-04-07T13:52:57.369364122Z [2017-04-07T13:52:57,368][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] initialized
2017-04-07T13:52:57.369799190Z [2017-04-07T13:52:57,369][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] starting ...
2017-04-07T13:52:57.627392969Z [2017-04-07T13:52:57,619][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [es-master-22305778-5t845] uncaught exception in thread [main]
2017-04-07T13:52:57.627454583Z org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: No up-and-running site-local (private) addresses found, got [name:lo (lo), name:eth0 (eth0)]
2017-04-07T13:52:57.627463083Z 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627490930Z 	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627496900Z 	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627501608Z 	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627506702Z 	at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627511291Z 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627515611Z 	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627520084Z Caused by: java.lang.IllegalArgumentException: No up-and-running site-local (private) addresses found, got [name:lo (lo), name:eth0 (eth0)]
2017-04-07T13:52:57.627524699Z 	at org.elasticsearch.common.network.NetworkUtils.getSiteLocalAddresses(NetworkUtils.java:187) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627529242Z 	at org.elasticsearch.common.network.NetworkService.resolveInternal(NetworkService.java:246) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627533728Z 	at org.elasticsearch.common.network.NetworkService.resolveInetAddresses(NetworkService.java:220) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627538373Z 	at org.elasticsearch.common.network.NetworkService.resolveBindHostAddresses(NetworkService.java:130) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627542671Z 	at org.elasticsearch.transport.TcpTransport.bindServer(TcpTransport.java:678) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627546944Z 	at org.elasticsearch.transport.netty4.Netty4Transport.doStart(Netty4Transport.java:183) ~[?:?]
2017-04-07T13:52:57.627551175Z 	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627555712Z 	at org.elasticsearch.transport.TransportService.doStart(TransportService.java:196) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627560016Z 	at org.elasticsearch.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:69) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627564325Z 	at org.elasticsearch.node.Node.start(Node.java:610) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627568482Z 	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:287) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627572688Z 	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:370) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627576859Z 	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.0.jar:5.3.0]
2017-04-07T13:52:57.627581337Z 	... 6 more
2017-04-07T13:52:58.465120868Z [2017-04-07T13:52:58,463][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] stopping ...
2017-04-07T13:52:58.471444180Z [2017-04-07T13:52:58,470][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] stopped
2017-04-07T13:52:58.471710951Z [2017-04-07T13:52:58,471][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] closing ...
2017-04-07T13:52:58.489035493Z [2017-04-07T13:52:58,488][INFO ][o.e.n.Node               ] [es-master-22305778-5t845] closed
@marcofranssen
Copy link
Author

Also not that I requested 1Gi memory and also put the limit on 1Gi. If I put this too a lower value the contianers crash with a OOM error.

Please let me know how to solve the issues. I would love to contribute my findings back to this project.

@pires
Copy link
Owner

pires commented Apr 7, 2017 •
edited

The problem is the way you do network.

Caused by: java.lang.IllegalArgumentException: No up-and-running site-local (private) addresses found, got [name:lo (lo), name:eth0 (eth0)]

Set NETWORK_HOST environment variable according to your setup.

@pires pires closed this as completed Apr 7, 2017
@pires
Copy link
Owner

pires commented Apr 7, 2017

If I put this too a lower value the contianers crash with a OOM error

Did you set ES_JAVA_OPTS?

@marcofranssen
Copy link
Author

Yes I did use ES_JAVA_OPTS.

So it set both to the same value, this causes a OOM error
if I set the resources for the container to 520 for example and the ES_JAVA_OPTS to 512 I still get this OOM error

@marcofranssen
Copy link
Author

marcofranssen commented Apr 7, 2017 •
edited

Thanks.... I configured it to be....

NETWORK_HOST: _eth0:ipv4 which resolved the network issue.

I also notice the pods are using 786.914 Mi memory although I set the ES_JAVA_OPTS.

ES_JAVA_OPTS=-Xms512m -Xmx512m

@pires
Copy link
Owner

pires commented Apr 7, 2017 •
edited

Maximum heap size should be no more than 50% of the total available RAM for the container. Remember that Lucene also consumes memory, etc.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@marcofranssen @pires