More than 10 requests at the same time fail!

[ghost]

Hello. I am trying to make 10 simultaneous POST requests to get some data from ElasticSearch

arrayWithFiveElements.forEach(function (country) {
    needle.post('http://localhost:9200/' + country + '/test_document1/_search?size=1000',
        '{"query": {"match_all": {}}}', function (err, response, body) {
            populateList(err, testData, body);
        });

    needle.post('http://localhost:9200/' + country + '/test_document2/_search?size=1000',
        '{"query": {"match_all": {}}}', function (err, response, body) {
            populateList(err, testData, body);
        });
});

and on the 9th and 10th request I always get this error:

Trace: [ERROR] No hits returned by Elastic Search for D1
    at populateMonthList (/opt/ddd/backend/backend.js:48:21)
    at /opt/ddd/backend/backend.js:74:17
    at error_stop (/opt/ddd/backend/node_modules/needle/lib/needle.js:206:9)
    at ClientRequest.<anonymous> (/opt/delivery-dashboard/backend/node_modules/needle/lib/needle.js:342:7)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at Socket.socketErrorListener (http.js:1547:9)
    at Socket.EventEmitter.emit (events.js:95:17)
    at onwriteError (_stream_writable.js:239:10)
    at onwrite (_stream_writable.js:257:5)
    at WritableState.onwrite (_stream_writable.js:97:5)
{
  "code": "EPIPE",
  "errno": "EPIPE",
  "syscall": "write"
}
Trace: [ERROR] No hits returned by Elastic Search for D2
    at populateRWAMonthList (/opt/ddd/backend/backend.js:59:21)
    at error_stop (/opt/ddd/backend/node_modules/needle/lib/needle.js:206:9)
    at ClientRequest.<anonymous> (/opt/ddd/backend/node_modules/needle/lib/needle.js:342:7)
    at ClientRequest.EventEmitter.emit (events.js:95:17)
    at Socket.socketErrorListener (http.js:1547:9)
    at Socket.EventEmitter.emit (events.js:95:17)
    at onwriteError (_stream_writable.js:239:10)
    at onwrite (_stream_writable.js:257:5)
    at WritableState.onwrite (_stream_writable.js:97:5)
    at fireErrorCallbacks (net.js:437:13)
{
  "code": "EPIPE",
  "errno": "EPIPE",
  "syscall": "write"
}

The other 8 requests are totally fine.

Do you happen to know the solution to this issue?

[bendrucker]

It would be helpful to see how you're orchestrating these calls (also syntax highlight code with GitHub Flavored Markdown please).

[ghost]

Hi. I have updated my original post. On the first POST I get the 1st document and on the second POST the 2nd associated document.

[tomas]

Can you post the full stacktrace of your error?

[ghost]

Hello. I added the console.trace.

[bendrucker]

Is it possible there's a race condition at work and you're actually running up against the per-host maxSockets limit of 5? Are you handling errors and determining success using populateList? Also note that your code as it stands could be more succinctly rewritten as:

needle.post(URL, DATA, populateList);

[ghost]

Hi. As you said I am determining success using populateList. As for the maxSockets, I have no idea. Do you know how I can check that on a RedHat server?

For the part about needle.post(URL, DATA, populateList);, I did it intentionally so I won't pass the response on the parameter but a different parameter. I changed it as you proposed but unfortunately it does not make any difference.

[bendrucker]

It's not a server property. It's something the Node agent proposes but you can override. See the node docs link above.

Re: referencing functions vs. passing the literal in directly, the idea is that they're exactly identical. Just saves you a little verbosity.

I'd consider the way you're aggregate results to be an anti-pattern. You should be using some sort of async pattern to handle that logic (i.e. the aggregate operation should fail if a single member fails). I like promises. They're a more robust abstraction but also require some effort to properly understand. You can also try async for something simpler.

Try running all 10 operations in sequence so only one socket is open at any given time. Based on the stack trace this almost certainly isn't a needle issue.

[ghost]

Unfortunately it will take me some time to rewrite that part so instead I am going to use elasticsearch.js in order to make the calls. Already tried it and it works as expected.

Thank you for the help guys.