ES cluster cannot startup #9

mulloymorrow · 2015-04-25T00:29:15Z

Following the deployment of an app with 2GB of memory usage, ES cannot be resized or deployed in k8s.

Following error when spinning up ES cluster in k8s:

2015-04-25T00:05:01.856141447Z Caused by: com.fasterxml.jackson.databind.JsonMappingException: Numeric value (2147483648) out of range of int
2015-04-25T00:05:01.856141447Z  at [Source: sun.net.www.protocol.http.HttpURLConnection$HttpInputStream@511e7433; line: 2062, column: 35] (through reference chain: io.fabric8.kubernetes.api.model.PodList["items"]->io.fabric8.kubernetes.api.model.Pod["desiredState"]->io.fabric8.kubernetes.api.model.PodState["manifest"]->io.fabric8.kubernetes.api.model.ContainerManifest["containers"]->io.fabric8.kubernetes.api.model.Container["memory"])
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:232)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.JsonMappingException.wrapWithPath(JsonMappingException.java:197)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.wrapAndThrow(BeanDeserializerBase.java:1415)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:244)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:242)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:242)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:242)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:232)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:206)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:25)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:538)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:242)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:118)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.ObjectReader._bind(ObjectReader.java:1232)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.databind.ObjectReader.readValue(ObjectReader.java:676)
2015-04-25T00:05:01.856141447Z  at com.fasterxml.jackson.jaxrs.base.ProviderBase.readFrom(ProviderBase.java:800)
2015-04-25T00:05:01.856141447Z  at org.apache.cxf.jaxrs.utils.JAXRSUtils.readFromMessageBodyReader(JAXRSUtils.java:1322)
2015-04-25T00:05:01.856141447Z  at org.apache.cxf.jaxrs.impl.ResponseImpl.doReadEntity(ResponseImpl.java:369)
2015-04-25T00:05:01.856141447Z  ... 15 more

pires · 2015-04-25T01:19:44Z

@jimmidyson would it be possible to just get the service endpoints instead of retrieving all pods? After all, we're just looking for IP addresses.

Something like:

final Endpoints endpoints = getKubernetes().endpointsForService(this.selector, null);
for (String endpoint : endpoints.getEndpoints()) {
  // instantiate DiscoveryNode
}

I can provide a PR but would like to know if we agree on this.

jimmidyson · 2015-04-25T06:36:19Z

@mulloymorrow - raised fabric8io/kubernetes-model#25 to sort this out. Will try to get a new release out asap. Thanks for the report.

@pires - sounds good to be, much more efficient.

jimmidyson · 2015-04-26T10:24:37Z

@mulloymorrow would you be able to share the manifest causing the issue so I can verify the fix in our tests?

mulloymorrow · 2015-04-26T15:23:01Z

Using fork of @pires manifests,@jimmidyson. https://github.com/porchdotcom/kubernetes-elasticsearch-cluster

jimmidyson · 2015-05-02T21:34:33Z

Will be fixed in next fabric8 release train.

mulloymorrow · 2015-05-12T20:51:34Z

When's the next fabric8 release?

jimmidyson · 2015-05-12T20:55:07Z

Fingers crossed this week.

On 21:51, Tue, 12 May 2015 Mulloy Morrow notifications@github.com wrote:

When's the next fabric8 release?

—
Reply to this email directly or view it on GitHub
#9 (comment)
.

pires · 2015-05-12T21:04:04Z

Please, let me know when it gets cut so I can continue with #11.

pires · 2015-05-16T08:17:56Z

fabric8-2.1.0 has just been tagged, so resuming work.

pires · 2015-06-02T09:57:38Z

So #12 has been merged and should fix this.

@jimmidyson will you cut a new release, please?

pires · 2015-06-03T09:10:20Z

So 1.1.2 is out and just lacks documentation update. @mulloymorrow care to try this out and close if fixed?

jimmidyson · 2015-06-03T09:13:01Z

@pires @mulloymorrow 1.1.2 isn't out yet sorry - had some problems getting this synced to Maven central.

I'm squeezing in service account auth too - will be ready very soon I promise!

pires · 2015-06-03T09:53:39Z

@jimmidyson that should be 1.2.0 then ;)

jimmidyson · 2015-06-03T10:17:38Z

@pires @mulloymorrow OK released 1.2.0, docker image available at fabric8/elasticsearch-k8s:1.5.2 now or wait for central sync (~20 mins) until it's available on Maven central.

Updated documentation, albeit sparsely ;)

pires · 2015-06-03T10:19:07Z

Awesome @jimmidyson. Thanks a lot.

pires · 2015-06-14T18:51:59Z

This can be closed.

mulloymorrow mentioned this issue Apr 25, 2015

Cannot redeploy cluster in k8s pires/kubernetes-elasticsearch-cluster#9

Closed

pires mentioned this issue Apr 25, 2015

Kubernetes API model is broken fabric8io/fabric8#3817

Closed

jimmidyson mentioned this issue Apr 25, 2015

Use longs for int64 values fabric8io/kubernetes-model#25

Closed

pires mentioned this issue May 9, 2015

[WIP] Get endpoints for service instead of pods #11

Closed

mulloymorrow closed this as completed Jun 14, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ES cluster cannot startup #9

ES cluster cannot startup #9

mulloymorrow commented Apr 25, 2015

pires commented Apr 25, 2015

jimmidyson commented Apr 25, 2015

jimmidyson commented Apr 26, 2015

mulloymorrow commented Apr 26, 2015

jimmidyson commented May 2, 2015

mulloymorrow commented May 12, 2015

jimmidyson commented May 12, 2015

pires commented May 12, 2015

pires commented May 16, 2015

pires commented Jun 2, 2015

pires commented Jun 3, 2015

jimmidyson commented Jun 3, 2015

pires commented Jun 3, 2015

jimmidyson commented Jun 3, 2015

pires commented Jun 3, 2015

pires commented Jun 14, 2015

ES cluster cannot startup #9

ES cluster cannot startup #9

Comments

mulloymorrow commented Apr 25, 2015

pires commented Apr 25, 2015

jimmidyson commented Apr 25, 2015

jimmidyson commented Apr 26, 2015

mulloymorrow commented Apr 26, 2015

jimmidyson commented May 2, 2015

mulloymorrow commented May 12, 2015

jimmidyson commented May 12, 2015

pires commented May 12, 2015

pires commented May 16, 2015

pires commented Jun 2, 2015

pires commented Jun 3, 2015

jimmidyson commented Jun 3, 2015

pires commented Jun 3, 2015

jimmidyson commented Jun 3, 2015

pires commented Jun 3, 2015

pires commented Jun 14, 2015