Error: getaddrinfo ENOTFOUND in dns.js

[eelcocramer]

I've a node script that downloads video parts based upon the playlist file (m3u8). With node 0.10.6 the script starts downloading ok but, after some time, it starts throwing ENOTFOUND exceptions.

With node 0.8.23 I've no problems at all.

This is the error I'm getting:

Error: getaddrinfo ENOTFOUND
at errnoException (dns.js:37:11)
at Object.onanswer as oncomplete

My OS is OS X 10.8.3.

Thanks,
Eelco

[bnoordhuis]

ENOTFOUND means the upstream DNS server replied that there are no matching records. I'm rather skeptical that's caused by the version of node.js that you're using. Try switching DNS servers. If you want to pursue this further, at the very least post a test case.

[eelcocramer]

I'll try to come up with a test case.

The thing is that the first few hundreds of requests go fine and the subsequent requests are failing. The requests are all for the same server record.

[bnoordhuis]

The point still stands: the ENOTFOUND is what gets returned by the upstream DNS server. If you are querying for the same record over and over again, try caching it.

[eelcocramer]

The OS already caches DNS requests for me. The requests are not actually send over the wire.

I've created a little script that fires dns queries but that just works fine so the issue is probably not in the DNS code like you said. I will investigate further and if I find something that is related the nodejs itself I let you know.

Thanks.

[brianseeders]

For what it is worth, I am currently experiencing and investigating the same issue, though I have not tried 0.8.23.

I am using http.request to make lots of HTTP requests in parallel, to many different hosts. I begin getting this error on every request, after about 1200 requests, and it happens consistently. Restarting the script solves the issue, so it's not DNS server related. I can do 5 requests at a time in parallel, or 40, and it still seems to happen around 1200 requests.

[brianseeders]

I had a hunch that the open files limit was being reached, so I doubled it, and I was able to make double the number of requests before failures started happening.

I then noticed that all of the finished requests were stuck in CLOSE_WAIT for some reason.

See here for disucssion of a similar issue: websockets/ws#180

The current workaround seems to be to forcibly destroy the socket, which is probably not a great solution. To do that, it is simply: request.socket.destroy()

[tjfontaine]

@isaacs has been looking for a reliable way to reproduce that CLOSE_WAIT issue, so if you can provide a smaller test case it would be greatly appreciated.

[eelcocramer]

Thanks for the comments. In my case I get the error after 240 successful request. I've created a gist to reproduce the error: https://gist.github.com/eelcocramer/5626801

The gist has 3 files:

• error-case.js = script that generates the error for me after 240 requests
• success-case.js = different approach to the same functionality but this does not generate the error
• fixed-error-case.js = basically the same script as error-case.js but has the req.socket.close call as suggested by @brianseeders in line 12

Using the scripts above I did not see any big differences between 0.10 and 0.8 versions of NodeJS. Except that 0.8 gives me 4 more successful requests.

[tjwebb]

+1 Also seeing this issue in v0.10.13

[Cominch]

+1 I experience this behavior on a Beaglebone, after the ENOTFOUND in my node application, I can't even ping from the shell any more, giving me a DNS error... Does anyone see a relation to this issue?

[donpdonp]

I'm seeing this too. Lots of successful downloads with at least one ENOTFOUND for the same hostname, either github.com or bower.herokuapp.com. By putting a fixed IP into /etc/hosts, I can work around the problem. It looks suspiciously like bower has some kind of parallel-dns query bug.

[OferE]

+1 - happens a lot. even in 10.0.26

Anyway - just throwing an exception and killing the app is a very very bad behaviour.

One would expect to get such error in an error object of the callback to the function that caused it.
This exception is thrown without any context and without any reasonable way to debug the cause of it.

Please fix this.

[chiragsanghvi]

I'm also getting this error a lot. The minute, total no of requests reach more than 1024, which is the ulimit for my system, I start getting these errors for every 1 of 10 requests for the same hostname. I could bump up the ulimit, but then it doesn't solve the problem.

Have tried to end the request using request.resume() as specified here http://nodejs.org/api/http.html#http_class_http_clientrequest , but still doesn't work.

Any help would be appreciated.

[flesler]

This happens to me consistently when doing even 100 req/s Node version is 0.10.26 (AWS ElasticBeanstalk). Had to work around this by caching IPs on the code by wrapping all calls to http.request() and passing the ip instead of the host name. This worked.

The problem is it also happens with external libraries I can't wrap (aws-sdk) so I tried wrapping dns.lookup(). The error was back after this. Here's the code I used.

var dns = require('dns'), cache = {};
dns._lookup = dns.lookup;
dns.lookup = function(domain, family, done) {
    if (!done) {
        done = family;
        family = null;
    }

    var key = domain+family;
    if (key in cache) {
        var ip = cache[key], 
            ipv = ip.indexOf('.') !== -1 ? 4 : 6;

        return process.nextTick(function() { 
            done(null, ip, ipv);
        });
    }

    dns._lookup(domain, family, function(err, ip, ipv) {
        if (err) return done(err);
        cache[key] = ip;
        done(null, ip, ipv);
    });
};

Please don't dismiss this issue it happens to me consistently when deployed (not locally).

[indutny]

@flesler we are not going to cache it in core. But if you have a test case that is reliable reproducing the problem - we may consider fixing it.

[gx0r]

I can reproduce it with @eelcocramer 's test case if I reduce ulimit:

➜  ~  ulimit -n 1024    
➜  ~  node error-case.js
STATUS: 200
Request count = 777
STATUS: 200
Request count = 778
STATUS: 200
Request count = 779
STATUS: 200
Request count = 780
STATUS: 200
Request count = 781
STATUS: 200
Request count = 782
STATUS: 200
Request count = 783
STATUS: 200
Request count = 784
STATUS: 200
Request count = 785
STATUS: 200
Request count = 786
STATUS: 200
Request count = 787
STATUS: 200
Request count = 788
STATUS: 200
Request count = 789
STATUS: 200
Request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790

Node v0.10.31 on Mac OS X 10.10 public beta 2

OTOH, current node master branch doesn't seem to get the error.

[OferE]

This have to be fixed in core - you are crashing the system without stack trace/any clue on what happened...

[slv]

I get the same error.
node 0.10.26, request 2.27.0,
Amazon linux AMI 2014.03

My application parse the tweets of the Stream API.
At some point:
stream.js: 94, throw er // Unhandled stream error in pipe getaddrinfo ENOTFOUND

I'm sure that error throws even if the file, that request is calling, exists.
this is my code:

              var readStream = request.get(image_url),
                  filename = '/tmp/' + common.uid();

              readStream.pipe(fs.createWriteStream(filename));

              readStream.on('error', function(e) {
                done();
              });

              readStream.on('end', function() { ....

For every tweet there is 2 request.pipe like this.

There is apparently no rules to get this error in my case, it happens 2 or 3 times in a week of tweets'stream.
Sometimes the application is just started, sometimes is running by hours....
At now I'm investigating about the situation in with 2 or more tweets come at the "same" time, but at the moment I'm not able to solve the question.

[gx0r]

Maybe fixed by this 6820054

Feels like a ulimit / running of out file descriptors type situation

[keithics]

I am experiencing this bug on my production server.. same issue if there are a lot of http requests in parallel

[luckydrq]

The OS already caches DNS requests for me. The requests are not actually send over the wire.

@eelcocramer , how do you get this conclution? Is there any tool or some methods? I'm just curious about it. Thanks.

[eelcocramer]

@luckydrq just Google for dnscache to get some pointers. Good luck!

[eladnava]

Also experiencing this problem in production, when executing the same HTTP request every 250ms forever. Every few hours this error is returned, most likely caused by an unreasonable amount of requests to the DNS server, and every once in a while one of them fails (either a random error or the DNS server blocked on purpose).

I think that some basic DNS caching should be implemented in Node.JS to fix this.

Edit: For now, I just ended up installing dnsmasq and have it cache the DNS responses.

I am using AWS Elastic Beanstalk, so I needed this to be configured automatically for my instances, so I created the following .ebextensions:

files:
"/etc/dnsmasq.conf":
content: |
listen-address=127.0.0.1
"/etc/resolv.conf":
content: |
search ec2.internal
nameserver 127.0.0.1
nameserver 172.31.0.2
packages:
yum:
dnsmasq: []

[matiasdecarli]

+1. Any news regarding this?

[suryatech]

Any updates on this ?

[AartiKriplani]

Our production(node 0.10.33) is failing Only on high load, with -
Error: getaddrinfo ENOTFOUND. Hostname: localhost
at errnoException (dns.js:37:11)
at Object.onanswer as oncomplete

Why does it need to query the DNS server for localhost?

[ravi]

@AartiKriplani this doesn't solve the original problem, but if you do not want to query DNS, why not use an IP address: 127.0.0.1.

[AartiKriplani]

The problem is we don't know where and why its making a lookup to localhost, cause we don't have anything configured to point to localhost. Trying to hunt it down but the original problem is that localhost lookup fails which is weird. It should not even reach the DNS server for it.

[ravi]

Ah okay. But re:"localhost" resolution, I am guessing that's just happening in dns.js because that's where any name resolution occurs (whether via DNS or via some other mechanism like the 'hosts' file). Anyway, not much light I can shed on this unfortunately.

[Jeff-Tian]

+1

[Trott]

Regarding the localhost question, as another commenter surmised, it's not actually querying DNS first. dns.js is for all name resolution. So it's checking for an entry in /etc/hosts (or equivalent on Windows) first. So create one there and that should solve the ENOTFOUND problem for lookups on localhost. Or, at least, that has been my experience.

Basically, if your system does not know how to resolve localhost, then Node doesn't know either.

[nickdesaulniers]

Here's the test case to repro, in v4.2.2:

var d = require('dgram');
var u = d.createSocket('udp6');
u.on('message', console.log.bind(console));
var text = new Buffer('hello world');
u.send(text, 0, text.length, 19302, 'stun.l.google.com');
// events.js:141
//      throw er; // Unhandled 'error' event
//      ^

//Error: getaddrinfo ENOTFOUND stun.l.google.com
//    at errnoException (dns.js:26:10)
//    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)

// no issue
var u2 = d.createSocket('udp4');
u2.on('message', console.log.bind(console));
u2.send(text, 0, text.length, 19302, 'stun.l.google.com')

I'm running into this in my RTCPeerConnection implementation while doing Trickle ICE.

[bnoordhuis]

@nickdesaulniers stun.l.google.com doesn't seem to have an AAAA record, only an A record, so unless I'm misunderstanding you (always a possibility) the script seems to be working as expected.

Aside, this repo has been archived. New issues should be filed against https://github.com/nodejs/node/issues.

[nickdesaulniers]

So it looks like Socket.prototype.send is using dns.lookup under the hood, which errors with ENOTFOUND when the OS does not have this info. If send was implemented to use dns.resolve, the error would be ENODATA if the AAAA (for ipv6) or A (for ipv4) records did not exist.

The docs for send say: A string may be supplied for the address parameter, and it will be resolved with DNS.

Ah, sorry I was typing up this response as yours came in.

[nickdesaulniers]

so unless I'm misunderstanding you (always a possibility) the script seems to be working as expected.

Here's the issue:

without knowing which records (AAAA or A) a hostname will resolve to, it's pretty useless to call Socket.prototype.send with a string hostname as opposed to address, since it may fail because ENOTFOUND. So the type of udp socket (upd6 or udp4) you create HAS to be dependent on records of the hostname. So while it's slightly convenient that udp.send let's you pass a string that does resolution, it's ultimately incorrect.

The correct usage would be to:

1. resolve the hostname youself, detect family.
2. create a specific udp socket based on results from step 1.
3. call udp's send method with resolved address.

Or, am I misunderstanding?

[bnoordhuis]

I think I see what you mean. The under-the-hood call to dns.lookup() specifies the query type (A or AAAA) based on the socket family. If you want to go the other way (configure socket family based on the record type), then yes, you would perform the three steps you've outlined.

[nickdesaulniers]

ah ok, thank you for the confirmation

[angorora]

I was setting up a node application using express when I got this error using the test database.The error came about because I had left out the path to the database during connection setup.

[prcongithub]

What is the final conclusion on this?
I have also started facing the same problem on CentOS release 6.4 (Final).
node: v0.10.4
npm: 2.14.3

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: getaddrinfo ENOTFOUND
    at errnoException (dns.js:37:11)
    at Object.onanswer [as oncomplete] (dns.js:124:16)

This happens when I am load testing the app with just 10 requests per second

[tjmcewan]

I'm seeing this on node v7.10.0.

[5amfung]

Is the problem really fixed? I'm still seeing with v7.10.1.

Using this simple script, the ENOTFOUND can be easily reproduced. In my setup, I have set the wait time to 3ms in order to avoid any ENOTFOUND error.

const dns = require('dns');

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function demo() {
for (let i = 0; i < 1000; i++) {
  await sleep(1);
  dns.lookup('somedomain.com', function () {
    console.dir(arguments)
  });
}
}

demo();

[eranimo]

Having the same problem on node 8.3.0