Skip to content
This repository has been archived by the owner. It is now read-only.

Error: getaddrinfo ENOTFOUND in dns.js #5488

Closed
eelcocramer opened this Issue May 16, 2013 · 42 comments

Comments

Projects
None yet
@eelcocramer
Copy link

eelcocramer commented May 16, 2013

I've a node script that downloads video parts based upon the playlist file (m3u8). With node 0.10.6 the script starts downloading ok but, after some time, it starts throwing ENOTFOUND exceptions.

With node 0.8.23 I've no problems at all.

This is the error I'm getting:

Error: getaddrinfo ENOTFOUND
at errnoException (dns.js:37:11)
at Object.onanswer as oncomplete

My OS is OS X 10.8.3.

Thanks,
Eelco

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented May 16, 2013

ENOTFOUND means the upstream DNS server replied that there are no matching records. I'm rather skeptical that's caused by the version of node.js that you're using. Try switching DNS servers. If you want to pursue this further, at the very least post a test case.

@bnoordhuis bnoordhuis closed this May 16, 2013

@eelcocramer

This comment has been minimized.

Copy link
Author

eelcocramer commented May 16, 2013

I'll try to come up with a test case.

The thing is that the first few hundreds of requests go fine and the subsequent requests are failing. The requests are all for the same server record.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented May 16, 2013

The point still stands: the ENOTFOUND is what gets returned by the upstream DNS server. If you are querying for the same record over and over again, try caching it.

@eelcocramer

This comment has been minimized.

Copy link
Author

eelcocramer commented May 16, 2013

The OS already caches DNS requests for me. The requests are not actually send over the wire.

I've created a little script that fires dns queries but that just works fine so the issue is probably not in the DNS code like you said. I will investigate further and if I find something that is related the nodejs itself I let you know.

Thanks.

@brianseeders

This comment has been minimized.

Copy link

brianseeders commented May 20, 2013

For what it is worth, I am currently experiencing and investigating the same issue, though I have not tried 0.8.23.

I am using http.request to make lots of HTTP requests in parallel, to many different hosts. I begin getting this error on every request, after about 1200 requests, and it happens consistently. Restarting the script solves the issue, so it's not DNS server related. I can do 5 requests at a time in parallel, or 40, and it still seems to happen around 1200 requests.

@brianseeders

This comment has been minimized.

Copy link

brianseeders commented May 20, 2013

I had a hunch that the open files limit was being reached, so I doubled it, and I was able to make double the number of requests before failures started happening.

I then noticed that all of the finished requests were stuck in CLOSE_WAIT for some reason.

See here for disucssion of a similar issue: websockets/ws#180

The current workaround seems to be to forcibly destroy the socket, which is probably not a great solution. To do that, it is simply: request.socket.destroy()

@tjfontaine

This comment has been minimized.

Copy link

tjfontaine commented May 20, 2013

@isaacs has been looking for a reliable way to reproduce that CLOSE_WAIT issue, so if you can provide a smaller test case it would be greatly appreciated.

@eelcocramer

This comment has been minimized.

Copy link
Author

eelcocramer commented May 22, 2013

Thanks for the comments. In my case I get the error after 240 successful request. I've created a gist to reproduce the error: https://gist.github.com/eelcocramer/5626801

The gist has 3 files:

Using the scripts above I did not see any big differences between 0.10 and 0.8 versions of NodeJS. Except that 0.8 gives me 4 more successful requests.

@tjwebb

This comment has been minimized.

Copy link

tjwebb commented Aug 23, 2013

+1 Also seeing this issue in v0.10.13

@Cominch

This comment has been minimized.

Copy link

Cominch commented Aug 29, 2013

+1 I experience this behavior on a Beaglebone, after the ENOTFOUND in my node application, I can't even ping from the shell any more, giving me a DNS error... Does anyone see a relation to this issue?

@donpdonp

This comment has been minimized.

Copy link

donpdonp commented Mar 16, 2014

I'm seeing this too. Lots of successful downloads with at least one ENOTFOUND for the same hostname, either github.com or bower.herokuapp.com. By putting a fixed IP into /etc/hosts, I can work around the problem. It looks suspiciously like bower has some kind of parallel-dns query bug.

@OferE

This comment has been minimized.

Copy link

OferE commented Mar 23, 2014

+1 - happens a lot. even in 10.0.26

Anyway - just throwing an exception and killing the app is a very very bad behaviour.

One would expect to get such error in an error object of the callback to the function that caused it.
This exception is thrown without any context and without any reasonable way to debug the cause of it.

Please fix this.

@chiragsanghvi

This comment has been minimized.

Copy link

chiragsanghvi commented May 3, 2014

I'm also getting this error a lot. The minute, total no of requests reach more than 1024, which is the ulimit for my system, I start getting these errors for every 1 of 10 requests for the same hostname. I could bump up the ulimit, but then it doesn't solve the problem.

Have tried to end the request using request.resume() as specified here http://nodejs.org/api/http.html#http_class_http_clientrequest , but still doesn't work.

Any help would be appreciated.

@flesler

This comment has been minimized.

Copy link

flesler commented Sep 8, 2014

This happens to me consistently when doing even 100 req/s Node version is 0.10.26 (AWS ElasticBeanstalk). Had to work around this by caching IPs on the code by wrapping all calls to http.request() and passing the ip instead of the host name. This worked.

The problem is it also happens with external libraries I can't wrap (aws-sdk) so I tried wrapping dns.lookup(). The error was back after this. Here's the code I used.

var dns = require('dns'), cache = {};
dns._lookup = dns.lookup;
dns.lookup = function(domain, family, done) {
    if (!done) {
        done = family;
        family = null;
    }

    var key = domain+family;
    if (key in cache) {
        var ip = cache[key], 
            ipv = ip.indexOf('.') !== -1 ? 4 : 6;

        return process.nextTick(function() { 
            done(null, ip, ipv);
        });
    }

    dns._lookup(domain, family, function(err, ip, ipv) {
        if (err) return done(err);
        cache[key] = ip;
        done(null, ip, ipv);
    });
};

Please don't dismiss this issue it happens to me consistently when deployed (not locally).

@indutny

This comment has been minimized.

Copy link
Member

indutny commented Sep 9, 2014

@flesler we are not going to cache it in core. But if you have a test case that is reliable reproducing the problem - we may consider fixing it.

@llambda

This comment has been minimized.

Copy link

llambda commented Sep 9, 2014

I can reproduce it with @eelcocramer 's test case if I reduce ulimit:

➜  ~  ulimit -n 1024    
➜  ~  node error-case.js
STATUS: 200
Request count = 777
STATUS: 200
Request count = 778
STATUS: 200
Request count = 779
STATUS: 200
Request count = 780
STATUS: 200
Request count = 781
STATUS: 200
Request count = 782
STATUS: 200
Request count = 783
STATUS: 200
Request count = 784
STATUS: 200
Request count = 785
STATUS: 200
Request count = 786
STATUS: 200
Request count = 787
STATUS: 200
Request count = 788
STATUS: 200
Request count = 789
STATUS: 200
Request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790
problem with request: getaddrinfo ENOTFOUND
Last successful request count = 790

Node v0.10.31 on Mac OS X 10.10 public beta 2

OTOH, current node master branch doesn't seem to get the error.

@OferE

This comment has been minimized.

Copy link

OferE commented Sep 9, 2014

This have to be fixed in core - you are crashing the system without stack trace/any clue on what happened...

@slv

This comment has been minimized.

Copy link

slv commented Sep 9, 2014

I get the same error.
node 0.10.26, request 2.27.0,
Amazon linux AMI 2014.03

My application parse the tweets of the Stream API.
At some point:
stream.js: 94, throw er // Unhandled stream error in pipe getaddrinfo ENOTFOUND

I'm sure that error throws even if the file, that request is calling, exists.
this is my code:

              var readStream = request.get(image_url),
                  filename = '/tmp/' + common.uid();

              readStream.pipe(fs.createWriteStream(filename));

              readStream.on('error', function(e) {
                done();
              });

              readStream.on('end', function() { ....

For every tweet there is 2 request.pipe like this.

There is apparently no rules to get this error in my case, it happens 2 or 3 times in a week of tweets'stream.
Sometimes the application is just started, sometimes is running by hours....
At now I'm investigating about the situation in with 2 or more tweets come at the "same" time, but at the moment I'm not able to solve the question.

@llambda

This comment has been minimized.

Copy link

llambda commented Sep 10, 2014

Maybe fixed by this 6820054

Feels like a ulimit / running of out file descriptors type situation

@keithics

This comment has been minimized.

Copy link

keithics commented Sep 18, 2014

I am experiencing this bug on my production server.. same issue if there are a lot of http requests in parallel

@luckydrq

This comment has been minimized.

Copy link

luckydrq commented Sep 22, 2014

The OS already caches DNS requests for me. The requests are not actually send over the wire.

@eelcocramer , how do you get this conclution? Is there any tool or some methods? I'm just curious about it. Thanks.

@eelcocramer

This comment has been minimized.

Copy link
Author

eelcocramer commented Sep 23, 2014

@luckydrq just Google for dnscache to get some pointers. Good luck!

@eladnava

This comment has been minimized.

Copy link

eladnava commented Oct 1, 2014

Also experiencing this problem in production, when executing the same HTTP request every 250ms forever. Every few hours this error is returned, most likely caused by an unreasonable amount of requests to the DNS server, and every once in a while one of them fails (either a random error or the DNS server blocked on purpose).

I think that some basic DNS caching should be implemented in Node.JS to fix this.

Edit: For now, I just ended up installing dnsmasq and have it cache the DNS responses.

I am using AWS Elastic Beanstalk, so I needed this to be configured automatically for my instances, so I created the following .ebextensions:

files:
"/etc/dnsmasq.conf":
content: |
listen-address=127.0.0.1
"/etc/resolv.conf":
content: |
search ec2.internal
nameserver 127.0.0.1
nameserver 172.31.0.2
packages:
yum:
dnsmasq: []

@matiasdecarli

This comment has been minimized.

Copy link

matiasdecarli commented Nov 25, 2014

+1. Any news regarding this?

@suryatech

This comment has been minimized.

Copy link

suryatech commented Feb 26, 2015

Any updates on this ?

@AartiKriplani

This comment has been minimized.

Copy link

AartiKriplani commented Mar 11, 2015

Our production(node 0.10.33) is failing Only on high load, with -
Error: getaddrinfo ENOTFOUND. Hostname: localhost
at errnoException (dns.js:37:11)
at Object.onanswer as oncomplete

Why does it need to query the DNS server for localhost?

@ravi

This comment has been minimized.

Copy link

ravi commented Mar 11, 2015

@AartiKriplani this doesn't solve the original problem, but if you do not want to query DNS, why not use an IP address: 127.0.0.1.

@AartiKriplani

This comment has been minimized.

Copy link

AartiKriplani commented Mar 11, 2015

The problem is we don't know where and why its making a lookup to localhost, cause we don't have anything configured to point to localhost. Trying to hunt it down but the original problem is that localhost lookup fails which is weird. It should not even reach the DNS server for it.

@ravi

This comment has been minimized.

Copy link

ravi commented Mar 11, 2015

Ah okay. But re:"localhost" resolution, I am guessing that's just happening in dns.js because that's where any name resolution occurs (whether via DNS or via some other mechanism like the 'hosts' file). Anyway, not much light I can shed on this unfortunately.

@Jeff-Tian

This comment has been minimized.

Copy link

Jeff-Tian commented Sep 6, 2015

+1

@Trott

This comment has been minimized.

Copy link
Member

Trott commented Sep 6, 2015

Regarding the localhost question, as another commenter surmised, it's not actually querying DNS first. dns.js is for all name resolution. So it's checking for an entry in /etc/hosts (or equivalent on Windows) first. So create one there and that should solve the ENOTFOUND problem for lookups on localhost. Or, at least, that has been my experience.

Basically, if your system does not know how to resolve localhost, then Node doesn't know either.

@jomo jomo referenced this issue Sep 10, 2015

Open

DNS caching #143

@nickdesaulniers

This comment has been minimized.

Copy link

nickdesaulniers commented Dec 28, 2015

Here's the test case to repro, in v4.2.2:

var d = require('dgram');
var u = d.createSocket('udp6');
u.on('message', console.log.bind(console));
var text = new Buffer('hello world');
u.send(text, 0, text.length, 19302, 'stun.l.google.com');
// events.js:141
//      throw er; // Unhandled 'error' event
//      ^

//Error: getaddrinfo ENOTFOUND stun.l.google.com
//    at errnoException (dns.js:26:10)
//    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:77:26)

// no issue
var u2 = d.createSocket('udp4');
u2.on('message', console.log.bind(console));
u2.send(text, 0, text.length, 19302, 'stun.l.google.com')

I'm running into this in my RTCPeerConnection implementation while doing Trickle ICE.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Dec 28, 2015

@nickdesaulniers stun.l.google.com doesn't seem to have an AAAA record, only an A record, so unless I'm misunderstanding you (always a possibility) the script seems to be working as expected.

Aside, this repo has been archived. New issues should be filed against https://github.com/nodejs/node/issues.

@nickdesaulniers

This comment has been minimized.

Copy link

nickdesaulniers commented Dec 28, 2015

So it looks like Socket.prototype.send is using dns.lookup under the hood, which errors with ENOTFOUND when the OS does not have this info. If send was implemented to use dns.resolve, the error would be ENODATA if the AAAA (for ipv6) or A (for ipv4) records did not exist.

The docs for send say: A string may be supplied for the address parameter, and it will be resolved with DNS.

Ah, sorry I was typing up this response as yours came in.

@nickdesaulniers

This comment has been minimized.

Copy link

nickdesaulniers commented Dec 28, 2015

so unless I'm misunderstanding you (always a possibility) the script seems to be working as expected.

Here's the issue:

without knowing which records (AAAA or A) a hostname will resolve to, it's pretty useless to call Socket.prototype.send with a string hostname as opposed to address, since it may fail because ENOTFOUND. So the type of udp socket (upd6 or udp4) you create HAS to be dependent on records of the hostname. So while it's slightly convenient that udp.send let's you pass a string that does resolution, it's ultimately incorrect.

The correct usage would be to:

  1. resolve the hostname youself, detect family.
  2. create a specific udp socket based on results from step 1.
  3. call udp's send method with resolved address.

Or, am I misunderstanding?

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Dec 28, 2015

I think I see what you mean. The under-the-hood call to dns.lookup() specifies the query type (A or AAAA) based on the socket family. If you want to go the other way (configure socket family based on the record type), then yes, you would perform the three steps you've outlined.

@nickdesaulniers

This comment has been minimized.

Copy link

nickdesaulniers commented Dec 28, 2015

ah ok, thank you for the confirmation

@angorora

This comment has been minimized.

Copy link

angorora commented Jan 8, 2016

I was setting up a node application using express when I got this error using the test database.The error came about because I had left out the path to the database during connection setup.

@prcongithub

This comment has been minimized.

Copy link

prcongithub commented Jan 21, 2016

What is the final conclusion on this?
I have also started facing the same problem on CentOS release 6.4 (Final).
node: v0.10.4
npm: 2.14.3

events.js:72
        throw er; // Unhandled 'error' event
              ^
Error: getaddrinfo ENOTFOUND
    at errnoException (dns.js:37:11)
    at Object.onanswer [as oncomplete] (dns.js:124:16)

This happens when I am load testing the app with just 10 requests per second

@tjmcewan

This comment has been minimized.

Copy link

tjmcewan commented Jun 16, 2017

I'm seeing this on node v7.10.0.

@5amfung

This comment has been minimized.

Copy link

5amfung commented Aug 10, 2017

Is the problem really fixed? I'm still seeing with v7.10.1.

Using this simple script, the ENOTFOUND can be easily reproduced. In my setup, I have set the wait time to 3ms in order to avoid any ENOTFOUND error.

const dns = require('dns');

function sleep(ms) {
  return new Promise(resolve => setTimeout(resolve, ms));
}

async function demo() {
for (let i = 0; i < 1000; i++) {
  await sleep(1);
  dns.lookup('somedomain.com', function () {
    console.dir(arguments)
  });
}
}

demo();
@eranimo

This comment has been minimized.

Copy link

eranimo commented Sep 29, 2017

Having the same problem on node 8.3.0

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.