Timeout Not Working!

[ttessarolo]

Hi –
I'm trying to set query timeout even from connection using requestTimeout or on per query basis using timeout on client.search() params. But either way is not working: it always use the default time out 30000ms.

Some examples [NOT WORKING: timeout alwais is 30000ms]:

On Connection

var esClient = new elasticsearch.Client(Object.create({
    requestTimeout: 10
}));

var esClient = new elasticsearch.Client(Object.create({
      requestTimeout: "10micros"
}));

Per Request

esClient.search({body: "//something",timeout =  "500micros"});
esClient.search({body: "//something",timeout =  "500ms"});

[delvedor]

Hello!
What are you trying to achieve?
There are two different timeouts, the requestTimeout is client side, while the timeout is for Elasticsearch. From your example, I can't say which one do you need, plus the timeout should not be a part of the body (doc).

Can you upload a full example?
Thanks!

[ttessarolo]

Hey @delvedor –
I was just trying to set a timeout for queries.
I've switched to the new alpha lib @elastic/elasticsearch and now it's working fine (even if the lib still have some problems)

[delvedor]

You can have a client-side timeout, which means that after n milliseconds if you didn't get an answer back the client will throw an error, and you can achieve that with requestTimeout: n. Or, you can set a timeout on Elasticsearch side, with the timeout parameter.
In the first case Elasticsearch will continue to do its job, but on the client-side, the answer will be ignored; in the second case, if the specified timeout doesn't end the search operation, Elasticsearch will send back an error.
Which one are you trying?

I've switched to the new alpha lib @elastic/elasticsearch and now it's working fine (even if the lib still have some problems)

Can you be a little bit more specific?
Thanks!

[ttessarolo]

Sure –
to put it simple now that's the code that is working:

const { Client } = require("@elastic/elasticsearch");
const isNil = require("lodash.isnil");

const _config = {
      node: `http://${host}:${port}`,
      sniffOnStart:
        process.env.ELASTISEARCH_SNIFF_ON_START === "true" ? true : false,
      sniffInterval: isNil(process.env.ELASTISEARCH_SNIFF_ON_START)
        ? 60000
        : Number(process.env.ELASTISEARCH_SNIFF_ON_START),
      requestTimeout: isNil(process.env.ELASTISEARCH_SNIFF_ON_START_QUERY_TIMEOUT)
        ? 30000
        : Number(process.env.ELASTISEARCH_SNIFF_ON_START),
      log: hera.logger
    };

 const esClient = new Client(_config);

Using the const elasticsearch = require("elasticsearch"); package the requestTimeout in config was ignored (or not working). [With the elasticsearch package I was doing Object.create(_config) to "clone" the config obj inside a retry pattern).

[delvedor]

I've switched to the new alpha lib @elastic/elasticsearch and now it's working fine (even if the lib still have some problems)

Can you be a little bit more specific?

What kind of problems are you encountering?

[ttessarolo]

I'm stress testing ES connection with an empty query:

esClient.search({
            query: {
              body: {
                size: 0,
                query: { match: { test: { query: "test" } } }
              }
          }})

I'm using autocannon to test the service (that is part of a microservice app with a rest endpoint):

autocannon -H 'content-type=application/json' -c 50 --forever -m GET http://localhost:9001/go-elastic/

Every run, after 200/250K queries the driver start throwing NoLivingConnectionsError no way to resurrect it.

[delvedor]

Your query is not correct, and the body should be a root key of the object.
Following you can find the code that I wrote to run a stress test, I was getting some error due to the default options of the http agent, and I'll update them to a safer value.

'use strict'

const http = require('http')
const { Client } = require('@elastic/elasticsearch')
const client = new Client({
  node: 'http://localhost:9200',
  agent: { maxSockets: 256 }
})

http.createServer((req, res) => {
  client.search({
    body: {
      size: 0,
      query: { match: { test: { query: 'test' } } }
    }
  }, (e, r) => {
    if (e) console.log(e)
    res.end('ok')
  })
}).listen(3000)

Autocannon:

autocannon -c 100 -d 5 -p 10 --forever localhost:3000

I'm getting consistent results with good performances.

[ttessarolo]

Sorry –
you are right. In my rush trying to synthesise I've reported the wrong code.
My query is actually:

client.search({
    body: {
      size: 0,
      query: { match: { test: { query: 'test' } } }
    }
  }

I've repeated my test with different config conditions I'm working on an in-dev microservices framework that use Restify.

The esClient.search()is inside a try/catchand the catch is:

catch (error) {
   console.log("int:", iteration, error.message);
   if (error instanceof errors.NoLivingConnectionsErrorr) process.exit(1);
}

Thus: every time I got a NoLivingConnectionsErrorr error I quit. Before I log out the number of iterations (stored in a variable that is ++ in every query).

Full Config (for my use case)

// THE ONLY .ENV SETTED IS: ELASTICSEARCH_QUERY_TIMEOUT=200
const _config = {
      node: `http://${host}:${port}`,
      sniffOnStart:
        process.env. ELASTICSEARCH SNIFF_ON_START === "true" ? true : false,
      sniffInterval: isNil(process.env.ELASTICSEARCH_SNIFF_INTERVALL)
        ? 60000
        : Number(process.env. ELASTICSEARCH_SNIFF_INTERVALL),
      requestTimeout: isNil(process.env.ELASTICSEARCH)
        ? 30000
        : Number(process.env. ELASTICSEARCH_QUERY_TIMEOUT)
    };

I've got the following result using autocannon: 267K iteractions before quit.

Solution [for now]

Then (as suggested me) I get rid of the Sniff config + I'added your suggested max_socketsoption:

// THE ONLY .ENV SETTED IS: ELASTICSEARCH_QUERY_TIMEOUT=200
const prefix = "ELASTICSEARCH";
const _config = {
      node: `http://${host}:${port}`,
      // sniffOnStart:
      //   process.env[`${prefix}_SNIFF_ON_START`] === "true" ? true : false,
      // sniffInterval: isNil(process.env[`${prefix}_SNIFF_INTERVALL`])
      //   ? 60000
      //   : Number(process.env[`${prefix}_SNIFF_INTERVALL`]),
      requestTimeout: isNil(process.env[`${prefix}_QUERY_TIMEOUT`])
        ? 30000
        : Number(process.env[`${prefix}_QUERY_TIMEOUT`])
      agent: { maxSockets: 256 }
    };

And the world changed: the test is still going (no quit!), and I bet it will go forever with exceptional performances.

Autocannon Results

Running 10s test @ http://localhost:9001/go-elastic/
50 connections

Stat    2.5% 50%  97.5% 99%   Avg     Stdev   Max
Latency 6 ms 8 ms 12 ms 15 ms 8.49 ms 2.28 ms 40.69 ms

Stat      1%     2.5%   50%    97.5%   Avg    Stdev  Min
Req/Sec   5343   5343   5499   5855    5566   157.05 5341
Bytes/Sec 956 kB 956 kB 984 kB 1.05 MB 996 kB 28 kB  956 kB

Req/Bytes counts sampled once per second.

56k requests in 10s, 9.96 MB read

My Configuration for test

• MacBook Pro (15-inch, 2018)
• Processor: 2,9 GHz Intel Core i9
• Memory: 32 GB 2400 MHz DDR4

I believe sniff is one of the big issue in the alpha so far

Post Scriptum

Very rarely I got the following error in my catch clause: TypeError: Right-hand side of 'instanceof' is not an object

Not every error is an Error.

[delvedor]

And the world changed: the test is still going (no quit!), and I bet it will go forever with exceptional performances

Glad to hear this!

I get rid of the Sniff config

Sniffing is a powerful tool, but it's also a double edge sword.
Internally the client calls _nodes/http to figure out the presence of other nodes, but if you are sitting behind a proxy that is not configured appropriately, or you are using some complex configuration, the sniffing will fail, because the IP returned by the operations are wrong, and the client will mark them as dead.

Very rarely I got the following error in my catch clause: TypeError: Right-hand side of 'instanceof' is not an object
Not every error is an Error.

To be able to debug this I would need more info, you should use a nested try catch and see in which situation this error happens.

[ttessarolo]

I did what you asked like this:

const { Client, errors } = require("@elastic/elasticsearch");

...
/* try to do the query*/.catch (error) {
        try {
          console.log(
            "ERROR",
            typeof error,
            error instanceof errors.NoLivingConnectionsErrorr,
            error
          );
        } catch (error2) {
          console.log("E1", error, "E2", error2);
          process.exit(1);
        }
      }

it prints out (summarised):

E1 { NoLivingConnectionsError: There are not living connections}
E2 TypeError: Right-hand side of 'instanceof' is not an object

Right-hand means that errors.NoLivingConnectionsErrorris not of type Error, but in your code I see:

class ElasticsearchClientError extends Error{}
class NoLivingConnectionsError extends ElasticsearchClientError{}

Then I did:

console.log(
        typeof errors.NoLivingConnectionsErrorr,
        errors.NoLivingConnectionsErrorr
      );

And I've got undefined undefined

Thus the problem is that errors.NoLivingConnectionsErrorr in

const { Client, errors } = require("@elastic/elasticsearch");

is undefined (at least in my implementation).

If I print out errors:

{ 
     ElasticsearchClientError: [Function: ElasticsearchClientError],
     TimeoutError: [Function: TimeoutError],
     ConnectionError: [Function: ConnectionError],
     NoLivingConnectionsError: [Function: NoLivingConnectionsError],
     SerializationError: [Function: SerializationError],
     DeserializationError: [Function: DeserializationError],
     ConfigurationError: [Function: ConfigurationError],
     ResponseError: [Function: ResponseError] 
}

[delvedor]

Can you provide a complete example to reproduce?
I did this, and everything works as expected:

'use strict'

const { errors } = require('@elastic/elasticsearch')

const connErr = new errors.ConnectionError()
console.log(typeof connErr) // object
console.log(connErr instanceof errors.ConnectionError) // true
console.log(connErr instanceof errors.TimeoutError) // false
console.log(connErr instanceof errors.ElasticsearchClientError) // true

[stale]

We understand that this might be important for you, but this issue has been automatically marked as stale because it has not had recent activity either from our end or yours.
It will be closed if no further activity occurs, please write a comment if you would like to keep this going.

Note: in the past months we have built a new client, that has just landed in master. If you want to open an issue or a pr for the legacy client, you should do that in https://github.com/elastic/elasticsearch-js-legacy