when using a port range on unicast discovery, scanning stops after the first connection

[jordansissel]

Version: ElasticSearch 0.20.6

How to reproduce:

• NodeBuilder.settings.put("discovery.zen.ping.unicast.hosts", "localhost[9300-9400]")
• Have the server listening on 9310
• NodeClient only pings port 9300

Observations:

• tcpdump -nni any 'portrange 9300-9400' - I only see TCP connections to port 9300, not anything in range 9301-9400.

Expected?

Workaround:

• generate my own list of hosts: NodeBuilder.settings.put("discovery.zen.ping.unicast.hosts", "localhost:9300,localhost:9301,localhost:9302...,localhost:9400")

[jordansissel]

The 'host[port1-port2]' stuff is documented here: http://www.elasticsearch.org/guide/reference/modules/discovery/zen/

[kimchy]

we only try the first one because the default is 9300-9400, and we used to try all of them by default (way back), and it was very expensive trying to form the connection on all those ports (assuming nothing was there). So it was changed to try the first one with the assumption that if a range is specified, the first one should be up to answer it.

[dadoonet]

@jordansissel As there is a workaround and that IMHO having more than one node on a single box is very uncommon (more a dev than a prod concern), could we close this issue?

[jordansissel]

This instance was a production issue; where a logstash and elasticsearch nodes colocated on the same server, logstash starts up first occasionally and snags port 9300 and only talks to itself despite elasticsearch being on port 9301 and the port range logstash told via NodeBuilder to connect to as 9300-9400

Still, I've worked around this thusly:
jordansissel/jruby-elasticsearch@4456e18

Basically, implemented the "try all ports" logic in my own code. Works well enough.

[dadoonet]

@jordansissel So this is happenning when you start logstash with an embedded elasticsearch, right?
IMHO, it's safer to separate things. For example, configure logstash to use 9300-9349 ports and Elasticsearch nodes to use 9350-9399 ports.
My 2 cents.

[jordansissel]

Agreed there. For backwards compatibility reasons I can't change the port. I think it's safe to close this given the reported behavior is the intended behavior :)

[jordansissel]

re: logstash with embedded elasticsearch

No, this also occurs, in general, when logstash and elasticsearch are simply on the same server as separate procseses. The result is that the default configurations require startup ordering because both compete for port 9300, and if ES starts on port 9301, logstash searches for ES on ports 9300-9400, finds itself (logstash) on 9300 and keeps trying to ping itself which it ignores.

[jordansissel]

Roughly, just to describe what I've seen - two separate cases:

Case 1

• logstash starts up; activates a no-data elasticsearch node to connect to the cluster so logstash can index things. Uses port 9300 as default.
• elasticsearch starts up. 9300 is in use, so listens on 9301
• logstash's elasticsearch client-node goes into discovery on 9300-9400, finds port 9300 (itself) responds
• logstash pings itself endlessly on port 9300
• elasticsearch never gets contacted

Case 2

• logstash starts up and is configured to listen on port 9900 (randomly chosen to avoid 9300)
• logstash is told elasticsearch can be found in port range 9300-9400
• elasticsearch starts up and is configured to listen on port 9305
• logstash's elasticsearch client-node tries to discover elasticsearch only on port 9300
• elasticsearch never gets contacted because the client-node discovery only talks to 9300 when told to try 9300-9400

In both cases, my work around of setting the unicast discovery list of all port combinations (host:9300, host:9301, ...) works around this.

(Just documenting for completeness and posterity!)

[dadoonet]

Thank you! I was trying to understand and now I think it's clear.
I think you have this issue because you are considering that elasticsearch embedded is a full data node but elasticsearch not embedded is a client node.

IMHO, you should try to use TransportClient when you don't need an embedded node and NodeClient if embedded is set to true. That way, logstash won't start any Node process and won't listen to any http/transport port.