Cluster#start getting Timeout::Error

[jgoggles]

Running Elasticsearch::Extensions::Test::Cluster::Cluster.new.start produces:

Timeout::Error: execution expired from /Users/danw/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/elasticsearch-extensions-0.0.22/lib/elasticsearch/extensions/test/cluster.rb:497:in 'sleep' from /Users/danw/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/elasticsearch-extensions-0.0.22/lib/elasticsearch/extensions/test/cluster.rb:497:in 'block (2 levels) in __wait_for_status' from /Users/danw/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/elasticsearch-extensions-0.0.22/lib/elasticsearch/extensions/test/cluster.rb:489:in 'loop' from /Users/danw/.rbenv/versions/2.2.2/lib/ruby/gems/2.2.0/gems/elasticsearch-extensions-0.0.22/lib/elasticsearch/extensions/test/cluster.rb:489:in 'block in __wait_for_status'

I believe this is because my cluster status is yellow due to some unallocated replicas. However when I set the number of replicas to 0 on my cluster running on port 9200 and ran

Elasticsearch::Extensions::Test::Cluster::Cluster.new(port:9200).start

I receive the same Timeout::Error

Ive also tried setting number_of_nodes: 1, which produces the same result.

It would be nice to be able to start the cluster with the option of skipping wait_for_green, but I would also appreciate some guidance with my current issue.

Also looking at the code it doesnt seem that setting the timeout option upon initialize does anything since wait_for_green calls __wait_for_status('green', 60), unless I'm missing something.

[olivierbuffon]

Don't know if it's related to your issue but regarding the doc, the correct syntax to use is Elasticsearch::Extensions::Test::Cluster.start(port: 9200) (without the new).

[jgoggles]

@olivierbuffon Yes you're right, thanks. I must've made a typo when creating the issue. Still having the same problem when starting the cluster with Elasticsearch::Extensions::Test::Cluster.start, however Elasticsearch::Extensions::Test::Cluster.start(number_of_nodes: 1) has begun working for whatever reason...

Edit: Actually the docs here indicate that Elasticsearch::Extensions::Test::Cluster::Cluster.new.start can be used as well like I originally had (note the extra Cluster). So its a bit confusing on that front.

[sly7-7]

I think that following the pretty big refactor, not all the docs are up to date. I find myself having trouble to correctly start/stop cluster and nodes. I will investigate and try to find out what's wrong

[jgoggles]

Running:
curl -XPUT 'localhost:9250/_settings' -d ' { "index" : { "number_of_replicas" : 0 } }'

will put the cluster into a green state and it wont timeout. However this still doesn't seem like the ideal thing to have to do. However, if the cluster has zero indexes it should start in a green state (and not timeout) without having to do that. This is just what I've been able to gather through my own investigation. This is also my first time implementing Elasticsearch so I am still learning about how it all works underneath and aside from this gem.

I still think that the wait_for_green should be an option passed to start.

Separately, I would like to submit a PR for a couple of different things but I am getting errors when running rake bundle. I suppose I should open a new issue for that...

[olivierbuffon]

I also have some issues related to this step of the starting process.
When I run:

Elasticsearch::Extensions::Test::Cluster.start(port: 9250, number_of_nodes: 1)

The command seems to run fine, PIDs are returned, and I can even see the cluster execution:

$ ps ax | grep elasticsearch
61939   ??  S      0:00.00 sh -c /usr/local/Cellar/elasticsearch/5.0.2/libexec/bin/elasticsearch -E cluster.name=elasticsearch-test-macbook-pro-de-olivier.local -E node.name=node-1 -E http.port=9250 -E path.data=/tmp/elasticsearch_test -E path.logs=/tmp/log/elasticsearch -E cluster.routing.allocation.disk.threshold_enabled=false -E network.host=_local_ -E script.inline=true -E script.stored=true -E node.attr.testattr=test -E path.repo=/tmp -E repositories.url.allowed_urls=http://snapshot.test* -E discovery.zen.minimum_master_nodes=0 -E node.max_local_storage_nodes=1 -E logger.level=DEBUG  > /dev/null\012
61940   ??  S      0:00.94 /usr/bin/java -Xms2g -Xmx2g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/local/Cellar/elasticsearch/5.0.2/libexec -cp /usr/local/Cellar/elasticsearch/5.0.2/libexec/lib/elasticsearch-5.0.2.jar:/usr/local/Cellar/elasticsearch/5.0.2/libexec/lib/* org.elasticsearch.bootstrap.Elasticsearch -E cluster.name=elasticsearch-test-macbook-pro-de-olivier.local -E node.name=node-1 -E http.port=9250 -E path.data=/tmp/elasticsearch_test -E path.logs=/tmp/log/elasticsearch -E cluster.routing.allocation.disk.threshold_enabled=false -E network.host=_local_ -E script.inline=true -E script.stored=true -E node.attr.testattr=test -E path.repo=/tmp -E repositories.url.allowed_urls=http://snapshot.test* -E discovery.zen.minimum_master_nodes=0 -E node.max_local_storage_nodes=1 -E logger.level=DEBUG
61964 s004  R+     0:00.00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn elasticsearch

But at the step of the wait_for_green here is what happened until it Timeout:

Starting 1 Elasticsearch nodes..Using Elasticsearch version [5.0]
/usr/local/Cellar/elasticsearch/5.0.2/libexec/bin/elasticsearch --daemonize -E cluster.name=elasticsearch-test-macbook-pro-de-olivier.local -E node.name=node-1 -E http.port=9250 -E path.data=/tmp/elasticsearch_test -E path.logs=/tmp/log/elasticsearch -E cluster.routing.allocation.disk.threshold_enabled=false -E network.host=_local_ -E script.inline=true -E script.stored=true -E node.attr.testattr=test -E path.repo=/tmp -E repositories.url.allowed_urls=http://snapshot.test* -E discovery.zen.minimum_master_nodes=0 -E node.max_local_storage_nodes=1 -E logger.level=DEBUG > /dev/null
#<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 9250>
.#<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 9250>
.#<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 9250>
.#<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 9250>
.#<Errno::ECONNREFUSED: Connection refused - connect(2) for "localhost" port 9250>

I tried to run the exact same command in another terminal and everything works perfectly.
I don't know if it could be related to my ES version (5.0.2) but for now I'm stuck at this point.

Does anyone have any idea?

[karmi]

Hi all, apologies for the delay with the response. First of all, yes number_of_nodes is correct after the class has been refactored, unfortunately it took time to catch it and has been corrected only recently.

@olivierbuffon & @jgoggles, actually, after the refactor of the class, the correct syntax to use is with the new, previously the class has been a bit of a mess from the API point of view.

UPDATE: Argh, of course the correct syntax is Elasticsearch::Extensions::Test::Cluster.start -- that is a module method which wraps the internal Cluster class, apparently I did so we can keep the same interface as before, but did so many things in between that I forgot it :)

However, if the cluster has zero indexes it should start in a green state (and not timeout) without having to do that.

Yes, absolutely, with no indexes it should be green. Regarding the question of waiting for the green state -- the assumption is that this cluster is used for testing purposes, and therefore everything is wiped upon start.

The main reason for getting a timeout from Errno::ECONNREFUSED is when you would start more than one node with an incorrect value for network.host, so they cannot discover each other. (Multicast being disabled on 5.2).

I see that I have many commits accumulated for the elasticsearch-extensions gem, which are not in the latest release, I should release a new one.

@jgoggles, in the meantime, would it be possible if you run your test against the master or 5.x branch of the gem?

[karmi]

@olivierbuffon, that is strange, I've inspected the arguments which are in your ps output and it looks fine. So when have tried to run that command on the terminal without the --daemonize option, ES started fine?

Can you also run it against the master or 5.x branch?

[karmi]

It's late here, guys, sorry. I think I have figured it out.

ES 5.x has changed the layout of the data directory, and no longer includes the cluster name in the first level of the hierarchy. So the __remove_cluster_data method is no longer correct. I'll fix it tomorrow.

[karmi]

Merged the fix into the master and 5.x branches. The problem was indeed that the data directory has not been removed on start, and since in all my usage of the class I usually do "delete all indices" in teardowns, I haven't run into the issue myself. Sorry the fix took so long, it should work now.