Unable to revive connection

[juliendangers]

Hello,

I keep getting this error, and it never recovers. I'm using Elasticsearch 1.3.2 and elasticsearch-js 2.4.0 (node client)

Elasticsearch WARNING: 2014-09-09T08:09:05Z
  Unable to revive connection: https://elasticsearch.****.net/

Elasticsearch WARNING: 2014-09-09T08:09:05Z
  No living connections

Here is my configuration

{
    apiVersion:'1.3',
    sniffOnConnectionFault:true,
    maxRetries:5,
    keepAlive:true,
    connectionClass:'http'
}

[DavidTanner]

I am also getting this same error. We have tried setting keepAlive to false and the app actually fails faster. I can't find anything in the other issues that fixes this problem.

elasticsearch: 1.5.14
forever-agent: 0.5.2
lodash-node: 2.4.1

node: 0.10.26

I have seen this timeout error just before all of the no living connections errors, but not all the time.

Elasticsearch ERROR: 2014-09-10T19:52:11Z
  Error: Request error, retrying -- connect ETIMEDOUT
      at Log.error (/Users/.../node_modules/elasticsearch/src/lib/log.js:213:60)
      at checkRespForFailure (/Users/.../elasticsearch/src/lib/transport.js:185:18)
      at HttpConnector.<anonymous> (/Users/.../node_modules/elasticsearch/src/lib/connectors/http.js:150:7)
      at ClientRequest.bound (/Users/.../node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)
      at ClientRequest.EventEmitter.emit (events.js:95:17)
      at Socket.socketErrorListener (http.js:1547:9)
      at Socket.EventEmitter.emit (events.js:95:17)
      at net.js:440:14
      at process._tickCallback (node.js:415:13)

[spalger]

@DavidTanner Are you using a similar configuration?

[DavidTanner]

The only configuration I have set is keepAlive, host, and port.

[juliendangers]

here is a bit more information about the error :

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  Unable to revive connection: http://[...]:9200/

Elasticsearch WARNING: 2014-09-12T14:22:16Z
  No living connections

Trace: { [Error: No Living connections] message: 'No Living connections' }
    at /[...]/transfertES.js:60:22
    at respond (/[...]/node_modules/elasticsearch/src/lib/transport.js:252:9)
    at sendReqWithConnection (/[...]/node_modules/elasticsearch/src/lib/transport.js:172:7)
    at next (/[...]/node_modules/elasticsearch/src/lib/connection_pool.js:213:7)
    at process._tickCallback (node.js:419:13)

[juliendangers]

the issue comes from the server.

I've increased the deadTimeout option, and the maxRetries and I haven't seen the error since

[DavidTanner]

I can understand that something happens on the server side to cause the initial error, but is there nothing the library dies to try to reconnect?

With your declaration of it being a server issue, is there any hint you can give us as to what is causing the issue?

[juliendangers]

we have a river plugin on ES (RabbitMQ), and every time I got the Unable to revive connection error, the river timed out too.

Increasing deadTimeout and maxRetries 'solved' it. I have to investigate if it comes from Elasticsearch or from our physical server itself

[juliendangers]

I was wrong, I just ran into the issue again...

[juliendangers]

@DavidTanner do you use a frontend with your ES cluster such as Nginx or HAProxy, or do you directly connect to your node(s) ?

[DavidTanner]

Our instance of ES is not behind any proxies.

[juliendangers]

I ran into a similar issue with the python library...

The issue seems similar to this one : http://elasticsearch-users.115913.n3.nabble.com/Increasing-CLOSE-WAIT-connections-and-HTTP-current-open-metric-td4019752.html

I was reindexing 1 billion documents from one cluster to another, using scan & scroll API. The destination cluster was overflowed by bulk indexing requests (around 90 request/sec with 1k documents in each), and was unable to respond after a few minutes.

I haven't try to connect to the cluster using the Java API yet, I've decided to use the RabbitMQ river instead, and I've no more problems.

[bugs181]

Getting this exact issue on our end as well. What is the fix or work around?

Here's our info:
node -v
v0.10.33

Elasticsearch:
1.4.1

Client library:
2.4.2

Connection:

var client = new ES.Client({
    host: 'localhost:9200',
    log: 'trace'
});

[bugs181]

Another interesting side affect. Whenever this error is thrown in nodejs and we restart node, the error is still there. Restarting elasticsearch via esrestart will usually fix the issue. However, in order to get nodejs to communicate with elastic search again, we have to manually open up an HTTP connection to http://server:9200. Very strange.