elasticsearch output cannot connect if first dns lookup unsuccessful - UnresolvedAddressException

[slaupster]

Logstash 1.4.2 using elasticsearch output transport protocol can not recover from an initial failure to resolve the dns name of the specified elasticsearch host.

Scenario is everything in docker registering services in consul.

1. The elasticsearch host in the logstash.conf is a consul service (so discovered through DNS). Elasticsearch data node starts first and registers the service. A check prevents it being discoverable until its actually listening on its port.
2. Logstash starts up next and fails to connect:

WARN: org.elasticsearch.transport.netty: [logstash-61718d49dc1d-1-2220] exception caught on transport layer [[id: 0xa37a0b12]], closing connection
java.nio.channels.UnresolvedAddressException
at sun.nio.ch.Net.checkAddress(Net.java:158)
at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:704)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.connect(NioClientSocketPipelineSink.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.NioClientSocketPipelineSink.eventSunk(NioClientSocketPipelineSink.java:70)
...

Elasticsearch client then appears to attempt to connect every 5 seconds and fails forever more with same exception.

After a minute the elasticsearch service is discoverable in DNS. The Logstash container can see it with nslookup yet the retry/fail loop goes on.

Restarting the Logstash container it connects first time and all is happy.

I've ruled out networkaddress.cache.negative.ttl. Its as if ElasticSearch client is reusing the same InetSocketAddress again and again, which is in a permanent state of unresolved.

I couldn't find anywhere that indicates this is a known problem although some threads bringing it up were "fixed" by using the IP address instead.

Sorry if this is already known but I've looked high and low! Any assistance much appreciated!

[gkuchta]

For what it's worth, it's not just you. I'm seeing the same behavior. For now I'm going to write a little dependency check to ensure that my ES servers are registered with consul before starting the service.

[slaupster]

Confirmed its still an issue in logstash 1.5 - i.e. with the new es client 1.5.1 (assuming thats where the problem actually is)

[kkorada]

i am also having same issue

[kkorada]

even i tried giving ip address instead of hostname but issue is same
i am usinig elasticsearch 4.X and logstash 1.4

[jordansissel]

This issue was in the Elasticsearch Transport client which this plugin no longer uses.

[jasontedor]

Is this still an issue? The output plugin uses it's own HTTP client now and the issue as reported here seems to be an artifact of having used the transport client previously?

[jasontedor]

Ha, race condition on me asking you here and I guess @dakrone asking you elsewhere and you closing in response to that. 😄