This repository has been archived by the owner. It is now read-only.

unable to discover other nodes - caught exception while handling client on http traffic #47

Closed
elipapa opened this Issue Oct 1, 2016 · 4 comments

Comments

Projects
None yet
2 participants
@elipapa

elipapa commented Oct 1, 2016

While spinning a deployment on GCE kubernetes 1.4, with three equivalent nodes (master=true, data=true), it appears that nodes fail to synchronize and agree on master role.

[2016-10-01 01:44:14,470][WARN ][http.netty               ] [Sally Floyd] Caught exception while handling client http traffic, closing connection [id: 0xbf51eae3, /10.0.1.14:56154 => /10.0.0.10:9200]
java.lang.IllegalArgumentException: invalid version format: INTERNAL:DISCOVERY/ZEN/UNICASTᄇ￐^OPENTARGETS-PRODJACK O'LANTERNDJD5LDRZT0US5XZK1S2SVW   10.0.1.14   10.0.1.14
    at org.jboss.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:94)
    at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
    at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-10-01 01:44:15,043][WARN ][http.netty               ] [Sally Floyd] Caught exception while handling client http traffic, closing connection [id: 0xfe74a744, /10.0.2.9:60868 => /10.0.0.10:9200]
java.lang.IllegalArgumentException: invalid version format: INTERNAL:DISCOVERY/ZEN/UNICASTᄇ￐^OPENTARGETS-PRODANTIPHON THE OVERSEERMGXOUTS4SQK4SSOJ5B84610.0.2.10.0.2.9
    at org.jboss.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:94)
    at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
    at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
@pires

This comment has been minimized.

Contributor

pires commented Oct 1, 2016

Please, share this node's configuration.

@elipapa

This comment has been minimized.

elipapa commented Oct 1, 2016

Thanks for looking into this!

  1. ES docker file forked from https://github.com/pires/docker-elasticsearch-kubernetes with a few more plugins installed and built on docker hub.

  2. kubectl create -f es-svc.yml which reads:

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch
  labels:
    component: elasticsearch
spec:
  type: LoadBalancer
  selector:
    component: elasticsearch
  ports:
  - name: http
    port: 9200
    protocol: TCP
  - name: transport
    port: 9300
    protocol: TCP
  1. followed by kubectl create -f es-node.yml which reads:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: es
  labels:
    component: elasticsearch
spec:
  replicas: 3
  template:
    metadata:
      labels:
        component: elasticsearch
    spec:
      containers:
      - name: es-node
        securityContext:
          privileged: true
          capabilities:
            add:
              - IPC_LOCK
        image: opentargets/docker-elasticsearch-kubernetes
        imagePullPolicy: Always
        resources:
          requests:
            memory: "12Gi"
            cpu: "1"
        env:
        - name: NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: "CLUSTER_NAME"
          value: "opentargets-prod"
        - name: "DISCOVERY_SERVICE"
          value: "elasticsearch"
        - name: NODE_MASTER
          value: "true"
        - name: NODE_DATA
          value: "true"
        - name: HTTP_ENABLE
          value: "true"
        - name: ES_HEAP_SIZE
          value: "8g"
        ports:
        - containerPort: 9200
          name: http
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: storage
          mountPath: /data
      volumes:
          - emptyDir:
              medium: ""
            name: "storage"
  1. kubectl get svc,po, deployment
NAME                  CLUSTER-IP     EXTERNAL-IP      PORT(S)             AGE
elasticsearch         10.3.253.192   104.155.14.192   9200/TCP,9300/TCP   7h
kubernetes            10.3.240.1     <none>           443/TCP             2d
NAME                  READY          STATUS           RESTARTS            AGE
es-3802889608-im2re   1/1            Running          0                   7h
es-3802889608-was2z   1/1            Running          0                   7h
es-3802889608-ye3r0   1/1            Running          0                   7h
NAME                  DESIRED        CURRENT          UP-TO-DATE          AVAILABLE   AGE
es                    3              3                3                   3           7h

  1. kubectl describe po
Name:       es-3802889608-im2re
Namespace:  default
Node:       gke-cluster-dev-es-d086e645-mn2j/10.132.0.6
Start Time: Sat, 01 Oct 2016 02:43:59 +0100
Labels:     component=elasticsearch
        pod-template-hash=3802889608
Status:     Running
IP:     10.0.2.9
Controllers:    ReplicaSet/es-3802889608
Containers:
  es-node:
    Container ID:   docker://7b3aec53e177b84d46f75b27a4e9e744ffbd6c026ae9e824616bb05dccd91977
    Image:      opentargets/docker-elasticsearch-kubernetes
    Image ID:       docker://sha256:fe9668e963a6b15ca5eefb3a16662c3169ebf61132765485a3b6fb382920f8cc
    Ports:      9200/TCP, 9300/TCP
    Requests:
      cpu:      1
      memory:       12Gi
    State:      Running
      Started:      Sat, 01 Oct 2016 02:44:01 +0100
    Ready:      True
    Restart Count:  0
    Environment Variables:
      NAMESPACE:        default (v1:metadata.namespace)
      CLUSTER_NAME:     opentargets-prod
      DISCOVERY_SERVICE:    elasticsearch
      NODE_MASTER:      true
      NODE_DATA:        true
      HTTP_ENABLE:      true
      ES_HEAP_SIZE:     8g
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-ldkkm:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-ldkkm
QoS Tier:   Burstable
No events.

Name:       es-3802889608-was2z
Namespace:  default
Node:       gke-cluster-dev-es-d086e645-4qrv/10.132.0.7
Start Time: Sat, 01 Oct 2016 02:43:59 +0100
Labels:     component=elasticsearch
        pod-template-hash=3802889608
Status:     Running
IP:     10.0.1.14
Controllers:    ReplicaSet/es-3802889608
Containers:
  es-node:
    Container ID:   docker://fec9445fc176917315abd0641afa3e6c4000f7a030139c5a6ed5fc75cd549a55
    Image:      opentargets/docker-elasticsearch-kubernetes
    Image ID:       docker://sha256:fe9668e963a6b15ca5eefb3a16662c3169ebf61132765485a3b6fb382920f8cc
    Ports:      9200/TCP, 9300/TCP
    Requests:
      cpu:      1
      memory:       12Gi
    State:      Running
      Started:      Sat, 01 Oct 2016 02:44:01 +0100
    Ready:      True
    Restart Count:  0
    Environment Variables:
      NAMESPACE:        default (v1:metadata.namespace)
      CLUSTER_NAME:     opentargets-prod
      DISCOVERY_SERVICE:    elasticsearch
      NODE_MASTER:      true
      NODE_DATA:        true
      HTTP_ENABLE:      true
      ES_HEAP_SIZE:     8g
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-ldkkm:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-ldkkm
QoS Tier:   Burstable
No events.

Name:       es-3802889608-ye3r0
Namespace:  default
Node:       gke-cluster-dev-es-d086e645-9mdn/10.132.0.2
Start Time: Sat, 01 Oct 2016 02:43:59 +0100
Labels:     component=elasticsearch
        pod-template-hash=3802889608
Status:     Running
IP:     10.0.0.10
Controllers:    ReplicaSet/es-3802889608
Containers:
  es-node:
    Container ID:   docker://7b614c1c17ce824f6fe037e7efac6c7e4f8e6622b181c51a18f43532dca7d639
    Image:      opentargets/docker-elasticsearch-kubernetes
    Image ID:       docker://sha256:fe9668e963a6b15ca5eefb3a16662c3169ebf61132765485a3b6fb382920f8cc
    Ports:      9200/TCP, 9300/TCP
    Requests:
      cpu:      1
      memory:       12Gi
    State:      Running
      Started:      Sat, 01 Oct 2016 02:44:01 +0100
    Ready:      True
    Restart Count:  0
    Environment Variables:
      NAMESPACE:        default (v1:metadata.namespace)
      CLUSTER_NAME:     opentargets-prod
      DISCOVERY_SERVICE:    elasticsearch
      NODE_MASTER:      true
      NODE_DATA:        true
      HTTP_ENABLE:      true
      ES_HEAP_SIZE:     8g
Conditions:
  Type      Status
  Initialized   True 
  Ready     True 
  PodScheduled  True 
Volumes:
  storage:
    Type:   EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium: 
  default-token-ldkkm:
    Type:   Secret (a volume populated by a Secret)
    SecretName: default-token-ldkkm
QoS Tier:   Burstable
No events.


logs from each node:

2016-10-01T01:44:05.121490450Z [2016-10-01 01:44:05,120][INFO ][node                     ] [Jack O'Lantern] version[2.4.0], pid[13], build[ce9f0c7/2016-08-29T09:14:17Z]
2016-10-01T01:44:05.121574964Z [2016-10-01 01:44:05,121][INFO ][node                     ] [Jack O'Lantern] initializing ...
2016-10-01T01:44:06.000757697Z [2016-10-01 01:44:06,000][INFO ][plugins                  ] [Jack O'Lantern] modules [reindex, lang-expression, lang-groovy], plugins [license, cloud-aws, marvel-agent, cloud-kubernetes, lang-python], sites []
2016-10-01T01:44:06.024595191Z [2016-10-01 01:44:06,024][INFO ][env                      ] [Jack O'Lantern] using [1] data paths, mounts [[/data (/dev/sda1)]], net usable_space [91.7gb], net total_space [94.2gb], spins? [possibly], types [ext4]
2016-10-01T01:44:06.024851029Z [2016-10-01 01:44:06,024][INFO ][env                      ] [Jack O'Lantern] heap size [7.9gb], compressed ordinary object pointers [true]
2016-10-01T01:44:10.509751262Z [2016-10-01 01:44:10,509][INFO ][node                     ] [Jack O'Lantern] initialized
2016-10-01T01:44:10.509880931Z [2016-10-01 01:44:10,509][INFO ][node                     ] [Jack O'Lantern] starting ...
2016-10-01T01:44:10.603529248Z [2016-10-01 01:44:10,603][INFO ][transport                ] [Jack O'Lantern] publish_address {10.0.1.14:9300}, bound_addresses {10.0.1.14:9300}
2016-10-01T01:44:10.610112503Z [2016-10-01 01:44:10,610][INFO ][discovery                ] [Jack O'Lantern] opentargets-prod/DJD5LDRzT0uS5XZK1s2Svw
2016-10-01T01:44:11.392670925Z [2016-10-01 01:44:11,392][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:11.392941332Z [2016-10-01 01:44:11,392][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:11.393176345Z [2016-10-01 01:44:11,393][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
2016-10-01T01:44:11.393304616Z [2016-10-01 01:44:11,393][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
2016-10-01T01:44:12.921639946Z [2016-10-01 01:44:12,920][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:12.921752941Z [2016-10-01 01:44:12,921][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:12.921875090Z [2016-10-01 01:44:12,921][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
2016-10-01T01:44:12.921967939Z [2016-10-01 01:44:12,921][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
2016-10-01T01:44:14.430379519Z [2016-10-01 01:44:14,430][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:14.430449273Z [2016-10-01 01:44:14,430][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:14.430648300Z [2016-10-01 01:44:14,430][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
2016-10-01T01:44:14.430872409Z [2016-10-01 01:44:14,430][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Jack O'Lantern] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
2016-10-01T01:44:14.483877544Z [2016-10-01 01:44:14,483][INFO ][cluster.service          ] [Jack O'Lantern] new_master {Jack O'Lantern}{DJD5LDRzT0uS5XZK1s2Svw}{10.0.1.14}{10.0.1.14:9300}{max_local_storage_nodes=1, master=true}, reason: zen-disco-join(elected_as_master, [0] joins received)
2016-10-01T01:44:14.499376291Z [2016-10-01 01:44:14,499][INFO ][http                     ] [Jack O'Lantern] publish_address {10.0.1.14:9200}, bound_addresses {10.0.1.14:9200}
2016-10-01T01:44:14.499548239Z [2016-10-01 01:44:14,499][INFO ][node                     ] [Jack O'Lantern] started
2016-10-01T01:44:14.581938803Z [2016-10-01 01:44:14,581][INFO ][gateway                  ] [Jack O'Lantern] recovered [0] indices into cluster_state
2016-10-01T01:44:14.900404877Z [2016-10-01 01:44:14,900][INFO ][license.plugin.core      ] [Jack O'Lantern] license [89fa40ea-b515-4ff0-9b3f-205bb6e0a56d] - valid
2016-10-01T01:44:14.904366884Z [2016-10-01 01:44:14,904][ERROR][license.plugin.core      ] [Jack O'Lantern] 
2016-10-01T01:44:14.904380806Z #
2016-10-01T01:44:14.904385886Z # License will expire on [Monday, October 31, 2016]. If you have a new license, please update it.
2016-10-01T01:44:14.904390927Z # Otherwise, please reach out to your support contact.
2016-10-01T01:44:14.904394971Z # 
2016-10-01T01:44:14.904398678Z # Commercial plugins operate with reduced functionality on license expiration:
2016-10-01T01:44:14.904402961Z # - marvel
2016-10-01T01:44:14.904406630Z #  - The agent will stop collecting cluster and indices metrics

and

2016-10-01T01:44:05.169616236Z [2016-10-01 01:44:05,169][INFO ][node                     ] [Antiphon the Overseer] version[2.4.0], pid[13], build[ce9f0c7/2016-08-29T09:14:17Z]
2016-10-01T01:44:05.169688316Z [2016-10-01 01:44:05,169][INFO ][node                     ] [Antiphon the Overseer] initializing ...
2016-10-01T01:44:05.964313458Z [2016-10-01 01:44:05,964][INFO ][plugins                  ] [Antiphon the Overseer] modules [reindex, lang-expression, lang-groovy], plugins [license, cloud-aws, marvel-agent, cloud-kubernetes, lang-python], sites []
2016-10-01T01:44:05.984541958Z [2016-10-01 01:44:05,984][INFO ][env                      ] [Antiphon the Overseer] using [1] data paths, mounts [[/data (/dev/sda1)]], net usable_space [91.8gb], net total_space [94.2gb], spins? [possibly], types [ext4]
2016-10-01T01:44:05.984647215Z [2016-10-01 01:44:05,984][INFO ][env                      ] [Antiphon the Overseer] heap size [7.9gb], compressed ordinary object pointers [true]
2016-10-01T01:44:11.027674521Z [2016-10-01 01:44:11,027][INFO ][node                     ] [Antiphon the Overseer] initialized
2016-10-01T01:44:11.027716968Z [2016-10-01 01:44:11,027][INFO ][node                     ] [Antiphon the Overseer] starting ...
2016-10-01T01:44:11.119092793Z [2016-10-01 01:44:11,118][INFO ][transport                ] [Antiphon the Overseer] publish_address {10.0.2.9:9300}, bound_addresses {10.0.2.9:9300}
2016-10-01T01:44:11.125514855Z [2016-10-01 01:44:11,125][INFO ][discovery                ] [Antiphon the Overseer] opentargets-prod/MGxOutS4SQK4ssOJ5B846Q
2016-10-01T01:44:11.976019280Z [2016-10-01 01:44:11,975][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:11.976267901Z [2016-10-01 01:44:11,976][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:11.976406668Z [2016-10-01 01:44:11,976][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
2016-10-01T01:44:11.976563108Z [2016-10-01 01:44:11,976][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
2016-10-01T01:44:13.506756508Z [2016-10-01 01:44:13,502][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:13.506955532Z [2016-10-01 01:44:13,506][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:13.507140181Z [2016-10-01 01:44:13,507][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
2016-10-01T01:44:13.507297872Z [2016-10-01 01:44:13,507][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
2016-10-01T01:44:15.016498592Z [2016-10-01 01:44:15,016][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9200
2016-10-01T01:44:15.016662521Z [2016-10-01 01:44:15,016][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.0.10, transport_address 10.0.0.10:9300
2016-10-01T01:44:15.016786896Z [2016-10-01 01:44:15,016][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
2016-10-01T01:44:15.016972522Z [2016-10-01 01:44:15,016][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Antiphon the Overseer] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
2016-10-01T01:44:15.064457425Z [2016-10-01 01:44:15,064][INFO ][cluster.service          ] [Antiphon the Overseer] new_master {Antiphon the Overseer}{MGxOutS4SQK4ssOJ5B846Q}{10.0.2.9}{10.0.2.9:9300}{max_local_storage_nodes=1, master=true}, reason: zen-disco-join(elected_as_master, [0] joins received)
2016-10-01T01:44:15.080464636Z [2016-10-01 01:44:15,080][INFO ][http                     ] [Antiphon the Overseer] publish_address {10.0.2.9:9200}, bound_addresses {10.0.2.9:9200}
2016-10-01T01:44:15.080596032Z [2016-10-01 01:44:15,080][INFO ][node                     ] [Antiphon the Overseer] started
2016-10-01T01:44:15.143918403Z [2016-10-01 01:44:15,143][INFO ][gateway                  ] [Antiphon the Overseer] recovered [0] indices into cluster_state
2016-10-01T01:44:15.485594842Z [2016-10-01 01:44:15,485][INFO ][license.plugin.core      ] [Antiphon the Overseer] license [00e9d27d-c98d-4eb3-a42e-a46c67ba077f] - valid
2016-10-01T01:44:15.488429466Z [2016-10-01 01:44:15,488][ERROR][license.plugin.core      ] [Antiphon the Overseer] 
2016-10-01T01:44:15.488445341Z #
2016-10-01T01:44:15.488450909Z # License will expire on [Monday, October 31, 2016]. If you have a new license, please update it.
2016-10-01T01:44:15.488458842Z # Otherwise, please reach out to your support contact.
2016-10-01T01:44:15.488463380Z # 
2016-10-01T01:44:15.488467597Z # Commercial plugins operate with reduced functionality on license expiration:
2016-10-01T01:44:15.488471836Z # - marvel
2016-10-01T01:44:15.488475519Z #  - The agent will stop collecting cluster and indices metrics

and lastly

[2016-10-01 01:44:04,963][INFO ][node                     ] [Sally Floyd] version[2.4.0], pid[12], build[ce9f0c7/2016-08-29T09:14:17Z]
[2016-10-01 01:44:04,963][INFO ][node                     ] [Sally Floyd] initializing ...
[2016-10-01 01:44:05,672][INFO ][plugins                  ] [Sally Floyd] modules [reindex, lang-expression, lang-groovy], plugins [license, cloud-aws, marvel-agent, cloud-kubernetes, lang-python], sites []
[2016-10-01 01:44:05,691][INFO ][env                      ] [Sally Floyd] using [1] data paths, mounts [[/data (/dev/sda1)]], net usable_space [92.2gb], net total_space [94.2gb], spins? [possibly], types [ext4]
[2016-10-01 01:44:05,691][INFO ][env                      ] [Sally Floyd] heap size [7.9gb], compressed ordinary object pointers [true]
[2016-10-01 01:44:10,157][INFO ][node                     ] [Sally Floyd] initialized
[2016-10-01 01:44:10,158][INFO ][node                     ] [Sally Floyd] starting ...
[2016-10-01 01:44:10,248][INFO ][transport                ] [Sally Floyd] publish_address {10.0.0.10:9300}, bound_addresses {10.0.0.10:9300}
[2016-10-01 01:44:10,255][INFO ][discovery                ] [Sally Floyd] opentargets-prod/vJtBKHg8TCm1UiSRLsAlUQ
[2016-10-01 01:44:10,996][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
[2016-10-01 01:44:10,996][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
[2016-10-01 01:44:10,996][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
[2016-10-01 01:44:10,996][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
[2016-10-01 01:44:12,522][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
[2016-10-01 01:44:12,522][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
[2016-10-01 01:44:12,522][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
[2016-10-01 01:44:12,522][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
[2016-10-01 01:44:14,033][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9200
[2016-10-01 01:44:14,034][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.1.14, transport_address 10.0.1.14:9300
[2016-10-01 01:44:14,034][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9200
[2016-10-01 01:44:14,034][INFO ][io.fabric8.elasticsearch.discovery.kubernetes.KubernetesUnicastHostsProvider] [Sally Floyd] adding endpoint /10.0.2.9, transport_address 10.0.2.9:9300
[2016-10-01 01:44:14,051][INFO ][cluster.service          ] [Sally Floyd] new_master {Sally Floyd}{vJtBKHg8TCm1UiSRLsAlUQ}{10.0.0.10}{10.0.0.10:9300}{max_local_storage_nodes=1, master=true}, reason: zen-disco-join(elected_as_master, [0] joins received)
[2016-10-01 01:44:14,066][INFO ][http                     ] [Sally Floyd] publish_address {10.0.0.10:9200}, bound_addresses {10.0.0.10:9200}
[2016-10-01 01:44:14,066][INFO ][node                     ] [Sally Floyd] started
[2016-10-01 01:44:14,119][INFO ][gateway                  ] [Sally Floyd] recovered [0] indices into cluster_state
[2016-10-01 01:44:14,373][INFO ][license.plugin.core      ] [Sally Floyd] license [c99d54bf-54f8-4e54-8713-9890da168bcb] - valid
[2016-10-01 01:44:14,375][ERROR][license.plugin.core      ] [Sally Floyd] 
#
# License will expire on [Monday, October 31, 2016]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
# 
# Commercial plugins operate with reduced functionality on license expiration:
# - marvel
#  - The agent will stop collecting cluster and indices metrics
#  - The agent will stop automatically cleaning indices older than [marvel.history.duration]
[2016-10-01 01:44:14,470][WARN ][http.netty               ] [Sally Floyd] Caught exception while handling client http traffic, closing connection [id: 0xbf51eae3, /10.0.1.14:56154 => /10.0.0.10:9200]
java.lang.IllegalArgumentException: invalid version format: INTERNAL:DISCOVERY/ZEN/UNICAST���������ᄇ￐^��OPENTARGETS-PROD�JACK O'LANTERN�DJD5LDRZT0US5XZK1S2SVW  10.0.1.14   10.0.1.14
    at org.jboss.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:94)
    at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
    at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-10-01 01:44:15,043][WARN ][http.netty               ] [Sally Floyd] Caught exception while handling client http traffic, closing connection [id: 0xfe74a744, /10.0.2.9:60868 => /10.0.0.10:9200]
java.lang.IllegalArgumentException: invalid version format: INTERNAL:DISCOVERY/ZEN/UNICAST���������ᄇ￐^��OPENTARGETS-PROD�ANTIPHON THE OVERSEER�MGXOUTS4SQK4SSOJ5B846Q�10.0.2.9�10.0.2.9
    at org.jboss.netty.handler.codec.http.HttpVersion.<init>(HttpVersion.java:94)
    at org.jboss.netty.handler.codec.http.HttpVersion.valueOf(HttpVersion.java:62)
    at org.jboss.netty.handler.codec.http.HttpRequestDecoder.createMessage(HttpRequestDecoder.java:75)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:191)
    at org.jboss.netty.handler.codec.http.HttpMessageDecoder.decode(HttpMessageDecoder.java:102)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:500)
    at org.jboss.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
    at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
    at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
    at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
    at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
    at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
    at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
    at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
    at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
[2016-10-01 01:44:20,286][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] creating index, cause [auto(bulk api)], templates [.marvel-es-1], shards [1]/[1], mappings [node, shards, _default_, index_stats, index_recovery, cluster_state, cluster_stats, node_stats, indices_stats]
[2016-10-01 01:44:20,446][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-data-1] creating index, cause [auto(bulk api)], templates [.marvel-es-data-1], shards [1]/[1], mappings [node, cluster_info]
[2016-10-01 01:44:20,535][INFO ][cluster.routing.allocation] [Sally Floyd] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.marvel-es-data-1][0], [.marvel-es-1-2016.10.01][0]] ...]).
[2016-10-01 01:44:20,570][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [indices_stats]
[2016-10-01 01:44:20,623][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [cluster_stats]
[2016-10-01 01:44:20,651][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [node_stats]
[2016-10-01 01:44:20,675][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [cluster_state]
[2016-10-01 01:44:20,694][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [node]
[2016-10-01 01:44:30,747][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [indices_stats]
[2016-10-01 01:44:30,772][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [index_stats]
[2016-10-01 01:44:30,788][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [cluster_stats]
[2016-10-01 01:44:30,804][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [shards]
[2016-10-01 01:44:30,817][INFO ][cluster.metadata         ] [Sally Floyd] [.marvel-es-1-2016.10.01] update_mapping [index_recovery]
[2016-10-01 06:04:04,333][DEBUG][action.search            ] [Sally Floyd] [2] Failed to execute fetch phase
RemoteTransportException[[Sally Floyd][10.0.0.10:9300][indices:data/read/search[phase/fetch/id]]]; nested: ScriptException[failed to run inline script [java.lang.Math.class.forName("java.io.BufferedReader").getConstructor(java.io.Reader.class).newInstance(java.lang.Math.class.forName("java.io.InputStreamReader").getConstructor(java.io.InputStream.class).newInstance(java.lang.Math.class.forName("java.lang.Runtime").getRuntime().exec("echo qq952135763").getInputStream())).readLines()] using lang [groovy]]; nested: PrivilegedActionException; nested: ClassNotFoundException[java.io.BufferedReader];
Caused by: ScriptException[failed to run inline script [java.lang.Math.class.forName("java.io.BufferedReader").getConstructor(java.io.Reader.class).newInstance(java.lang.Math.class.forName("java.io.InputStreamReader").getConstructor(java.io.InputStream.class).newInstance(java.lang.Math.class.forName("java.lang.Runtime").getRuntime().exec("echo qq952135763").getInputStream())).readLines()] using lang [groovy]]; nested: PrivilegedActionException; nested: ClassNotFoundException[java.io.BufferedReader];
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:288)
    at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:85)
    at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:188)
    at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:605)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:408)
    at org.elasticsearch.search.action.SearchServiceTransportAction$FetchByIdTransportHandler.messageReceived(SearchServiceTransportAction.java:405)
    at org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
    at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:77)
    at org.elasticsearch.transport.TransportService$4.doRun(TransportService.java:376)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.security.PrivilegedActionException: java.lang.ClassNotFoundException: java.io.BufferedReader
    at java.security.AccessController.doPrivileged(Native Method)
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:278)
    ... 12 more
Caused by: java.lang.ClassNotFoundException: java.io.BufferedReader
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677)
    at groovy.lang.GroovyClassLoader$InnerLoader.loadClass(GroovyClassLoader.java:425)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787)
    at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at java.lang.invoke.MethodHandleImpl$BindCaller$T/1037637747.invoke_V(MethodHandleImpl.java:1258)
    at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)
    at 44e0bdb1792f048619fdca433e4ec27a2d205e98.run(44e0bdb1792f048619fdca433e4ec27a2d205e98:1)
    at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$1.run(GroovyScriptEngineService.java:281)
    ... 14 more

output of kubectl describe svc elasticsearch:


Name:           elasticsearch
Namespace:      default
Labels:         component=elasticsearch
Selector:       component=elasticsearch
Type:           LoadBalancer
IP:         10.3.253.192
LoadBalancer Ingress:   104.155.14.192
Port:           http    9200/TCP
NodePort:       http    32021/TCP
Endpoints:      10.0.0.10:9200,10.0.1.14:9200,10.0.2.9:9200
Port:           transport   9300/TCP
NodePort:       transport   31292/TCP
Endpoints:      10.0.0.10:9300,10.0.1.14:9300,10.0.2.9:9300
Session Affinity:   None
No events.
@pires

This comment has been minimized.

Contributor

pires commented Oct 1, 2016

You are using the same service both for clients and discovery. A node won't be able to connect to the other nodes if you're using a LoadBalancer service. Take a look at the two services detailed in https://github.com/pires/kubernetes-elasticsearch-cluster.

@elipapa

This comment has been minimized.

elipapa commented Oct 1, 2016

i had previously tried the two service approach and couldn't get the nodes to talk to each other.
I have tried again with two separate services and now it appears to work fine.
Thanks!

@elipapa elipapa closed this Oct 1, 2016

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.