tribe: java.lang.NullPointerException at org.elasticsearch.cluster.routing.RoutingTable$Builder.add(RoutingTable.java:429)

[TinLe]

I get these NPE after running after running for about 10 minutes or less. I did not see these NPEs with ES v1.4.1.

ES v1.4.2 on tribe node and the ES clusters the tribe node is aggregating.

[2015-01-06 04:08:01,591][WARN ][tribe ] [dc1-app1111] failed to process [cluster event from mobile_dc1, zen-disco-receive(from master [[dc1-app2222][TLF_56CDTV2dHXmUf0XFUg][dc1-app2222][inet[/172.24.23.163:9300]]{max_local_storage_nodes=1, master=true, zone=data}])]
java.lang.NullPointerException
at org.elasticsearch.cluster.routing.RoutingTable$Builder.add(RoutingTable.java:429)
at org.elasticsearch.tribe.TribeService$TribeClusterStateListener$1.execute(TribeService.java:265)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:329)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:153)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Here is tribe config:

tribe:
mobile_dc1:
cluster.name: mobile-es_dc1
discovery.zen.ping.unicast.hosts: ["dc1-app2222", "dc1-app3333"]
mobile_dc2:
cluster.name: mobile-es_dc2
discovery.zen.ping.unicast.hosts: ["dc2-app2222", "dc2-app3333"]
mobile_dc3:
cluster.name: mobile-es_dc3
discovery.zen.ping.unicast.hosts: ["dc3-app2222", "dc3-app3333"]
feed_dc1:
cluster.name: dc1-logstash
discovery.zen.ping.unicast.hosts: ["dc1-app1234", "dc1-app1235"]
feed_dc2:
cluster.name: dc2-logstash
discovery.zen.ping.unicast.hosts: ["dc2-app2345", "dc2-app2346"]

[clintongormley]

Hi @TinLe

Is that the whole stack trace, or is there any more that you can give us?

thanks

[TinLe]

That's all I have in my log. I am assuming I need to tweak some flags to
get more verbose logging. Do you happen to know which flags/param I need
to change?

FWIW, these NPEs are still happening regularly. I have to restart my tribe
node, otherwise it stop querrying the cluster that caused the NPEs.

Tin

On Tue, Jan 13, 2015 at 6:47 AM, Clinton Gormley notifications@github.com
wrote:

Hi @TinLe https://github.com/TinLe

Is that the whole stack trace, or is there any more that you can give us?

thanks

—
Reply to this email directly or view it on GitHub
#9174 (comment)
.

[javanna]

This issue might be related to #6411. @TinLe do you have closed indices in some of your tribes?

[TinLe]

Yes, #6411 sound similar. I only keep the last 10 days of indices open,
anything older are closed.

Tin

On Wed, Jan 14, 2015 at 2:07 AM, Luca Cavanna notifications@github.com
wrote:

This issue might be related to #6411
#6411. @TinLe
https://github.com/TinLe do you have closed indices in some of your
tribes?

—
Reply to this email directly or view it on GitHub
#9174 (comment)
.

[clintongormley]

@TinLe Could you try opening all indices (assuming your cluster can handle it) and seeing if the problem goes away?

[javanna]

I am pretty sure this is exactly the same as #6411, I just double checked the line where the exception comes from. Closing as duplicate, fix coming.