Elasticsearch - No Living Connections

[nexflo]

This has been mentioned before here and there..but im certain this is still a bug or connections arent pooled correctly?

Im doing the following: (it indexes 6.000.000 docs...one by one..(dont wanna use bulk imports..so lets ignore this topic for a second)
I'm even waiting for an index to be complete to call the next one...still at around 16.000 entries...i always get the no more living connections error.

I tried with keepalive: true/false etc. but it seems its a connection pooling issue no?
iprange = array of item
esearch = Elasticsearch client.

function doItem(i){
    process.stdout.write("Importing item:  " + i + "\r");
    var doc = {};
    doc.index = "geoip";
    doc.type = "location";
    doc.body = iprange[i];
    esearch.index(doc, function (err, resp) {
        if(err)
            trace.info(err);

        i++;

        if(i < len) {
            doItem(i);
        }
    });
}

doItem(1);

[spalger]

What is the problem? Is there an error logged by the client or by elasticsearch?

[thecaddy]

I've had this issue as well. If the app does not use the client for a prolonged period of time the I will receive this error and the only solution has been to restart the because it will not reconnect to elasticsearch. keepAlive is defaulted to true.

[nexflo]

Hey @spalger
The elasticsearch client says:

Elasticsearch ERROR: 2015-03-07T16:07:37Z
  Error: Request error, retrying -- connect EADDRNOTAVAIL
      at Log.error (/Users/XXX/Sites/XXX/node_modules/elasticsearch/src/lib/log.js:213:60)
      at checkRespForFailure (/Users/XXX/Sites/XXX/node_modules/elasticsearch/src/lib/transport.js:195:18)
      at HttpConnector.<anonymous> (/Users/XXX/Sites/XXX/node_modules/elasticsearch/src/lib/connectors/http.js:154:7)
      at ClientRequest.bound (/Users/XXX/Sites/XXX/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)
      at ClientRequest.emit (events.js:107:17)
      at Socket.socketErrorListener (_http_client.js:272:9)
      at Socket.emit (events.js:107:17)
      at net.js:451:14
      at process._tickCallback (node.js:355:11)

17:07:37.499 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 
17:07:37.507 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 
17:07:37.511 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 
17:07:37.515 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 
17:07:37.519 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 
17:07:37.524 location.js:399 | { [Error: No Living connections] message: 'No Living connections', [stack]: [Getter/Setter] } 

The elasticsearch server doesnt spit out any error logs.
For me this happens every time somewhere around indexing the 16420 or so item.

(That number is so close to 16384, which is 2^14 .. that i believe it could be that all available client sockets are filled up??? but shouldn't ES client use existing sockets where available? + maxsockets is default 10? + keepAlive true/false doesn't make a difference).

As the data im indexing is the public maxmind geoip...i could also send over the code for you to reproduce this?

[thecaddy]

The error i see:

error: Error: No Living connections
    at sendReqWithConnection (.../node_modules/elasticsearch/src/lib/transport.js:174:15)
    at next (.../node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
    at process._tickCallback (node.js:355:11)

Elasticsearch WARNING: 2015-03-07T00:28:39Z
  Unable to revive connection

[spalger]

@nexflo I would love to play with the code if you don't mind pushing it online somewhere.

[nexflo]

Heya I just extracted a sample here: (i didn't test it..but i guess it should work)

https://gist.github.com/nexflo/fc2a763a408cd31f27cd
(you will need to download the GEO IP database to get the dataset its free)

I guess for testing one can create a simple loop which adds over 20.000 random items into an index as well for sample reasons - guess we would see the timeouts there as well.

[nexflo]

@spalger is there anything else I can do to help you resolve/pin down this issue?

[spalger]

Looking into this now.

[rafacustodio]

I have this issue as well, but in my application is a rabbitmq consumer, and when I use NodeJS Cluster (for more concurrency), at some point start to show to "No living connection" error.

[darylrobbins]

I am seeing the same issue after about 5,000 or so updates (upsert with dynamic Groovy script). I've specific a single address in my 3-node cluster. The connection shouldn't be idle for more than a couple minutes tops while it reads in a big file.

The interesting thing is that the connection eventually recovers after a few minutes. Several batches fail from the error but a few minutes later, it continues successfully with another batch. Then after a while, it starts failing again -- and the cycle repeats.

Before I start getting No living connections, I see the following error:

Elasticsearch ERROR: 2015-03-31T00:27:07Z
  Error: Request error, retrying -- read ECONNRESET
    at Log.error (/app/node_modules/elasticsearch/src/lib/log.js:213:60)
    at checkRespForFailure (/app/node_modules/elasticsearch/src/lib/transport.js:195:18)
    at HttpConnector.<anonymous> (/app/node_modules/elasticsearch/src/lib/connectors/http.js:154:7)
    at ClientRequest.bound (/app/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)
    at ClientRequest.emit (events.js:95:17)
    at Socket.socketErrorListener (http.js:1551:9)
    at Socket.emit (events.js:95:17)
    at net.js:440:14
    at process._tickDomainCallback (node.js:463:13)

[shandyba]

Can confirm this happens to us as well.
The facts: it has been happening for at least half a year already, with regularly updated javascript elasticsearch clients used.
It was the same with node 0.10.x. No change was observed after shift to 0.12.x.
It has been indeed happening in windows x86 and x64 as well as linux x64 node builds.
For some reason we can't catch it our local environments.
On production it happens on those back-end servers which perform lots of sequential background operations with ES (similarly to what is described in this thread). On those production servers which just serve visitors (e.g. use ES for customers-initiated search requests) it is pretty much not observable, even under high load conditions.

[nexflo]

@spalger Any updates on this?
Anything we can do to support?

[ltamrazov]

Also getting this. Seen similar issue in many places, my research thus far has taken me to:

#142 #90 #69 #46

Running:
io.js 1.2.0
elasticsearch 1.4.3
client 4.0.2

My findings thus far:
I doing about 1M queries (not indexing). Initially, it started blowing up around 16K with this message:

Elasticsearch ERROR: 2015-04-16T21:38:41Z Error: Request error, retrying -- connect EADDRNOTAVAIL 127.0.0.1:9200 - Local (0.0.0.0:0) at Log.error (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/log.js:213:60) at checkRespForFailure (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/transport.js:195:18) at HttpConnector.<anonymous> (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/connectors/http.js:154:7) at ClientRequest.wrapper (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/node_modules/lodash/index.js:3189:19) at emitOne (events.js:75:13) at ClientRequest.emit (events.js:150:7) at Socket.socketErrorListener (_http_client.js:249:9) at emitOne (events.js:75:13) at Socket.emit (events.js:150:7) at net.js:432:14 at process._tickCallback (node.js:337:11) Then I've read that

Elasticsearch WARNING: 2015-04-16T21:38:41Z Unable to revive connection: http://localhost:9200/

Elasticsearch WARNING: 2015-04-16T21:38:41Z No living connections

Then I read in documentation: maxSockets —  Maximum number of concurrent requests that can be made to any node. defaults to 10. So I thought maybe thats the issue. Using async i created a queue to control concurrency. I started of doing 2 queries at a time, and at about 16K queries got the same error.

I thought it might help if I increased number of nodes to handle the load, but still getting the same thing.

Following #46 I commented out the agent: this.agent in http.js. At that point, I am able to increase concurrency to 10, which is also the default maxSockets. I can even increase it past that to 20, and then to 50. at 100 concurrent requests, it blows up again with a different error:

Elasticsearch ERROR: 2015-04-16T21:43:48Z Error: Request error, retrying -- connect ECONNRESET 127.0.0.1:9200 at Log.error (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/log.js:213:60) at checkRespForFailure (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/transport.js:195:18) at HttpConnector.<anonymous> (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/connectors/http.js:154:7) at ClientRequest.wrapper (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/node_modules/lodash/index.js:3189:19) at emitOne (events.js:75:13) at ClientRequest.emit (events.js:150:7) at Socket.socketErrorListener (_http_client.js:249:9) at emitOne (events.js:75:13) at Socket.emit (events.js:150:7) at net.js:432:14 at process._tickCallback (node.js:337:11)

This surprised be since I assumed the client would handle concurrency in a similar way, using some sort of back-burner or queue. Is that not the case?

If I remove the queue all together, (but keep agent: this.agent commented out), I again get a different error around 16K:

Elasticsearch ERROR: 2015-04-16T21:11:50Z Error: Request error, retrying -- connect ECONNREFUSED 127.0.0.1:9200 at Log.error (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/log.js:213:60) at checkRespForFailure (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/transport.js:195:18) at HttpConnector.<anonymous> (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/src/lib/connectors/http.js:154:7) at ClientRequest.wrapper (/Users/Levon/Documents/my-dev/test/server/node_modules/elasticsearch/node_modules/lodash/index.js:3189:19) at emitOne (events.js:75:13) at ClientRequest.emit (events.js:150:7) at Socket.socketErrorListener (_http_client.js:249:9) at emitOne (events.js:75:13) at Socket.emit (events.js:150:7) at net.js:432:14 at process._tickCallback (node.js:337:11)

Also tried switching keepAlived: false, this did not have any effect.

Thanks!

[bugs181]

This fix for us was the temporary workaround noted on #142

Client.apis[config.apiVersion].ping.spec.requestTimeout = customDefaultMs;

As far as I can tell, it has solved our issues. I haven't tested any further due to a change in our backend that we have yet to test.