Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

order of getaddrinfo results not respected #6307

Closed
benschulz opened this Issue Apr 20, 2016 · 47 comments

Comments

Projects
None yet
@benschulz
Copy link

benschulz commented Apr 20, 2016

  • Version: v5.10.1
  • Platform: Linux archer 4.4.5-1-ARCH #1 SMP PREEMPT Thu Mar 10 07:38:19 CET 2016 x86_64 GNU/Linux
  • Subsystem: dns/cares_wrap

The results returned by getaddrinfo are reordered, giving preference to ipv4 addresses. The explanation given in #4693 seems unsatisfactory. Not implementing happy eyeballs is a reasonable design decision, of course, but why does that mean ipv4? It should mean taking the first entry. At the moment node is deliberately defying the system configuration.

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Apr 20, 2016

As I understand it, without the reordering, ipv6 addresses could be the first entry, which would break anywhere that doesn't have an ipv6 stack. The current implementation is intentional in order to give preference to ipv4. /cc @bnoordhuis

@addaleax

This comment has been minimized.

Copy link
Member

addaleax commented Apr 20, 2016

Is this still a problem when using the latest v6 release candidate (aka is this fixed by #6021)?

@addaleax addaleax added the dns label Apr 20, 2016

@cjihrig

This comment has been minimized.

Copy link
Contributor

cjihrig commented Apr 20, 2016

The DNS hints aren't related to the reordering.

@cjihrig

This comment has been minimized.

Copy link
Contributor

cjihrig commented Apr 20, 2016

@bnoordhuis would the dns.ADDRCONFIG flag mitigate the problem?

@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Apr 20, 2016

As I understand it, the proper place to configure preferences is via gai.conf.

Snippet from my gai.conf:

# precedence  <mask>   <value>
#
#    Defaults:
#
#precedence  ::1/128       50
#precedence  ::/0          40
#precedence  2002::/16     30
#precedence ::/96          20
#precedence ::ffff:0:0/96  10
#
#    For sites which prefer IPv4 connections change the last line to
#
#precedence ::ffff:0:0/96  100
@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Apr 20, 2016

would the dns.ADDRCONFIG flag mitigate the problem?

Only partially. AI_ADDRCONFIG means 'return AAAA entries only when there is a configured IPv6 interface' but the presence of such an interface doesn't say anything about IPv6 traffic actually being routable. (That's true for IPv4 as well, of course; my point is that AI_ADDRCONFIG isn't a fix per se.)

We made the decision to prefer IPv4 over IPv6 three or four years ago when IPv6 support was much more spotty than it is today. It's something we can reconsider if the majority thinks it's a good idea.

@jasnell

This comment has been minimized.

Copy link
Member

jasnell commented Apr 22, 2016

There are likely two things that can happen here:

  1. We can improve the documentation to indicate why the results are ordered the way they are, and
  2. We can add an option that would tell the impl not to re-order the output. The current default behavior would remain.
@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Apr 25, 2016

This is clearly a violation of the principle of least astonishment and I don't see how adding documentation is going to help. Hardly anyone affected by this will be using the dns module directly. It took me half a day before realizing my issue was with the dns module. Since then I have put a workaround in place and am no longer affected. Still, I think you should slowly phase out this odd logic. IMHO, over time, keeping it will hurt more than it helps.

@igalic

This comment has been minimized.

Copy link

igalic commented Dec 23, 2016

from what i gather, this is possibly the source of my issue with npm install failing (a possible regression of npm/npm#6857)

This time i'm running alpine (hence, muslc) under docker, again, IPv6-only network.

~ # node -e "dns.lookup('registry.npmjs.org', (...args) => console.log(args))"
[ null, '151.101.112.162', 4 ]
~ # 

note that musl implements rfc 3484/6724.

this works fine in ruby, btw:

# pry -r 'open-uri'
[1] pry(main)> open("https://registry.npmjs.org/")
=> #<StringIO:0x0055ecc25d4fd0
 @base_uri=#<URI::HTTPS https://registry.npmjs.org/>,
 @meta=
  {"server"=>"CouchDB/1.5.0 (Erlang OTP/R16B03)",
   "content-type"=>"application/json",
   "cache-control"=>"max-age=300",
   "content-length"=>"194",
   "accept-ranges"=>"bytes",
…etc…

from what i gather, this can be fixed with:

diff --git a/lib/dns.js b/lib/dns.js
index cbb994b8f2..c161a89a66 100644
--- a/lib/dns.js
+++ b/lib/dns.js
@@ -140,7 +140,7 @@ exports.lookup = function lookup(hostname, options, callback) {
     if (all) {
       callback(null, []);
     } else {
-      callback(null, null, family === 6 ? 6 : 4);
+      callback(null, null, family === 4 ? 4 : 6);
     }
     return {};
   }
@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Dec 27, 2016

That fix would default the family to 6 iff the caller isn't asking for all families. However, most of the time the caller should be asking for all families. Even if that was not the case, changing defaults is a mean thing to do to clients.

Generally any caller should ask for all families and then try the results in-order. If the caller were to implement happy eyeballs they would ask for all and concurrently try the per-familiy subsequences in-order.

IMHO the proper way to fix this is to apply #4693. Whether to happy eyeball or not is orthogonal.

@sam-github

This comment has been minimized.

Copy link
Member

sam-github commented Dec 29, 2016

Happy Eyeballs - for those who haven't heard the term

@igalic

This comment has been minimized.

Copy link

igalic commented Feb 3, 2017

every month or two, i try a fresh major release of nodejs, only to find this issue is still there.

(@jasnell commented on 22 Apr 2016) There are likely two things that can happen here:

  • We can improve the documentation to indicate why the results are ordered the way they are, and
  • We can add an option that would tell the impl not to re-order the output. The current default behavior would remain.

i highly agree with @benschulz here:

(@benschulz commented on 25 Apr 2016 • edited) This is clearly a violation of the principle of least astonishment and I don't see how adding documentation is going to help. Hardly anyone affected by this will be using the dns module directly. It took me half a day before realizing my issue was with the dns module. Since then I have put a workaround in place and am no longer affected. Still, I think you should slowly phase out this odd logic. IMHO, over time, keeping it will hurt more than it helps.

and believe that #4693 is the correct way forward — in 7.x+

@Trott

This comment has been minimized.

Copy link
Member

Trott commented Jul 16, 2017

Should this remain open? (I'm guessing the answer is "yes" but would be happy to find out I'm wrong.)

@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Jul 16, 2017

This is your project and you are free to take it in any direction you please. If you think the status quo is good enough or even right, you should close the ticket. All I can tell you is that I had a really rough time with this. But that's just one data point, I imagine there are several more people affected.
Fermi estimate: Two people have raised the issue; if we conservatively posit that .1‒1% of those affected manage to figure out the root cause and find this report, it'd be around 200‒2000 people affected in total.

Hope that helps. :P

Edit: Added missing "affected" qualifier. =)

@gibfahn

This comment has been minimized.

Copy link
Member

gibfahn commented Jul 17, 2017

@bnoordhuis looking at your comments:

#6307 (comment)

We made the decision to prefer IPv4 over IPv6 three or four years ago when IPv6 support was much more spotty than it is today. It's something we can reconsider if the majority thinks it's a good idea.

#4693 (comment)

We don't do happy eyeballs. If you change the order, that would have to change. That's a big can of worms.

I understand that you're against implementing happy eyeballs, but not necessarily against just letting the results come back in their original order. Is your second comment saying that ceasing to modify the order will force us to implement happy eyeballs?

I understand that this may be a massive breaking change, and thus a non-starter, but if it's possible to make that change we can at least consider it (even if it's just an option for now, and doesn't become the default till Node 18.x).

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Jul 17, 2017

Is your second comment saying that ceasing to modify the order will force us to implement happy eyeballs?

Not exactly. Happy eyeballs is a transition mechanism. It's not needed in a world where IPv6 connectivity is good enough. Do we live in that world yet? Maybe, maybe not; not enough data.

@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Jul 17, 2017

Happy eyeballs is a transition mechanism. It's not needed in a world where IPv6 connectivity is good enough.

That's the part of the argument I don't understand. I set my system up to resolve names to IPv6 addresses and, if that yields no results, to IPv4 addresses. What's the issue? (I consider names with invalid AAAA entries an issue with the name, not my system configuration.)

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Jul 17, 2017

A valid AAAA entry does not mean there is a network path to that machine.

That was historically a big barrier to IPv6 adoption and is what happy eyeballs tries to address: try IPv6 and IPv4 in parallel and use whatever manages to establish a connection.

@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Jul 17, 2017

A valid AAAA entry does not mean there is a network path to that machine.

So anyone affected has an IPv4 and an IPv6 address and their ISP is unable to route all IPv6 packets? Couldn't they do the reverse of what I do, i.e. prefer IPv4 over IPv6 at the OS rather than application level?

I appreciate the time taken for the various comments, thank you.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Jul 17, 2017

Yes, but that takes time and effort to set up. We want to avoid a situation where a node.js release breaks the out-of-the-box experience for a large group of users, especially when it might be difficult to debug for them.

@igalic

This comment has been minimized.

Copy link

igalic commented Jul 17, 2017

then, the best way forward, as mentioned numerous times in this issue, would be to implement happy eyes in a future version of node.js

@gibfahn

This comment has been minimized.

Copy link
Member

gibfahn commented Jul 18, 2017

then, the best way forward, as mentioned numerous times in this issue, would be to implement happy eyes in a future version of node.js

I don't see anyone in this thread suggesting implementing happy eyeballs in Node.js

@telmich

This comment has been minimized.

Copy link

telmich commented Aug 8, 2017

Not sure if this is the right bug, but IPv6 support seems to be seriously broken since 0.10... until and including v6.11.2. No version (also not 4.8.2 or 4.8.4) are capable of working in IPv6 only networks (as originally described in bitinn/node-fetch#66). It seems that npm is only trying IPv4, where everything else works.

This breaks various software building on top of node and will cause a variety of problems when bigger vendors like Apple will soon require IPv6 networks to function.

Is there any chance that there will be a proper IPv6 support in nodejs in the near future?

@telmich

This comment has been minimized.

Copy link

telmich commented Aug 8, 2017

I am sorry, I actually meant to refer to npm/npm#6857

@refack

This comment has been minimized.

Copy link
Member

refack commented Aug 9, 2017

@telmich just for clarity, there is a separation of responsibility between node and npm (although there is obviously strong coupling), and node does support other package managers.
My question is, can you make it clear whether node isn't working for you in an IPv6 only environment, or is it npm that fails?
If it is the former, can you open a new issue? Optimally with a clear description of the scenario, and a minimal reproducing code?
If it is the later, a better recourse would be to open an issue in the npm repository, or evaluate an alternative package manager.

@refack

This comment has been minimized.

Copy link
Member

refack commented Aug 9, 2017

@benschulz and @igalic will a configuration mechanism that will reorder/restore-the-order of getaddrinfo result fulfill the requirements of your use case?
Such a mechanism could be an environment variable (like NODE_OPTIONS), a global variable (like require.extensions) or a cli flag (like --zero-fill-buffers)

@refack refack self-assigned this Aug 9, 2017

@igalic

This comment has been minimized.

Copy link

igalic commented Aug 9, 2017

env options or global config vars would work.
cli options would not, cuz you don't know how many layers of software you're working under…

the (often unknown (to an end-user)) layers of indirection are also why it takes so long for people to pup up here after having exhausted all other venues.
when you read thru people's debugging history of this bug it's always,

  • blame the setup
  • verify other software
  • suspect npm (or equivalent)
  • npm devs tell you it's a dns (core module) issue
  • validate issue (wow, it's so obvious now!)

this behaviour of the software is really unintuitive.

but then people finally get here (at least those that persisted thru all these steps) and find a bug where we're arguing whether in a world where we've run out of IPv4 addresses, we should support an IPv6 only setup. for over a year now.

@igalic

This comment has been minimized.

Copy link

igalic commented Aug 9, 2017

@telmich re configuration option: https://tickets.puppetlabs.com/browse/SERVER-393
despite the JVM defaulting to prefer IPv6 for binding in dual-stack setups (see https://docs.oracle.com/javase/8/docs/api/java/net/doc-files/net-properties.html), it doesn't seem to do the same for outgoing connections

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Aug 9, 2017

Implementing happy eyeballs should not be hard and would solve the problem for everyone instantly.

You can do this in your application right now, you don't have to wait for node to do it for you.

Matter of fact, I don't think node should because it's absolute hell to debug when it goes wrong. A cure worse than the disease.

@telmich FWIW, I don't think your issue is this issue. I can testify from first-hand experience that node can work just fine in an IPv6-only environment. If you have a test case of sorts, please file a new issue.

@telmich

This comment has been minimized.

Copy link

telmich commented Aug 9, 2017

@bnoordhuis Well, the authors of NPM seem to disagree with you. If you look at npm/npm#6857 it shows that npm does not work in IPv6 only networks, because of core dns issues which brings me to #6307 - wait, this is here again.

I have the feeling that you are holding up to a theory ("hell to debug") that is not true in practice. I highly recommend to have a look at the code of curl or chromium, both supporting happy eyeballs, both doing it in an easy to debug manner, by printing out the ip when inspecting the connection.

This is exactly what I as a user would expect, whereas nodejs' "use only IPv4" is for all practical reasons stopping to use it in IPv6 environments.

You might also anticipate that IPv6 is actually the new standard that is pushed out aggressively, having grown to 30% of the network traffic in Switzerland.

In the end, it is very easy: you can claim that IPv6 is not worth/interesting/whatever and people who try to use it in newer data centers will learn that nodejs is the IPv6 "bad kid" and basically doesn't work and eventually move on to something else.

Or you anticipate that IPv6 is actually already there and the lack of proper IPv6 support is hitting your users right now (and for almost 3 years, for the matter) and implement a proper solution. Be that happy eyeballs or something else fancy (which I highly disrecommend from seeing that chances of doing it wrong are 99%), is then up to you.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Aug 9, 2017

Well, the authors of NPM seem to disagree with you.

Fine, but the authors of npm don't know node as well as I do.

In the end, it is very easy: you can claim that IPv6 is not worth/interesting/whatever

Have you actually read through this issue? I'm pushing for abandoning the current hack we have.

@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented Aug 9, 2017

I wonder whether the best way forward is an opt-in which can be turned into opt-out with some major version change and ultimately be removed (hopefully).

bnoordhuis added a commit to bnoordhuis/io.js that referenced this issue Aug 10, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

Refs: nodejs#6307
@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Aug 10, 2017

I was under the impression a dns.lookup() option was added some time ago that returns the results as-is but seems I was mistaken. I opened #14731 to add one.

@telmich

This comment has been minimized.

Copy link

telmich commented Aug 10, 2017

Looking at #14731, I wonder if you suggest to use a random address to connect to, @bnoordhuis?
If that is the case, you will have randomly success on IPv6 only networks, instead of always success, compared to happy eyeballs.

Correct me, if I am wrong.

@refack refack referenced this issue Aug 17, 2017

Closed

test, win: fix IPv6 detection on Windows #14865

1 of 2 tasks complete

jasnell added a commit that referenced this issue Aug 23, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

PR-URL: #14731
Ref: #6307
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax added a commit to addaleax/ayo that referenced this issue Aug 25, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

PR-URL: nodejs/node#14731
Ref: nodejs/node#6307
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

addaleax added a commit to ayojs/ayo that referenced this issue Aug 28, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

PR-URL: nodejs/node#14731
Ref: nodejs/node#6307
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added a commit that referenced this issue Sep 10, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

PR-URL: #14731
Ref: #6307
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>

MylesBorins added a commit that referenced this issue Sep 12, 2017

dns: add `verbatim` option to dns.lookup()
When true, results from the DNS resolver are passed on as-is, without
the reshuffling that Node.js otherwise does that puts IPv4 addresses
before IPv6 addresses.

PR-URL: #14731
Ref: #6307
Reviewed-By: Refael Ackermann <refack@gmail.com>
Reviewed-By: Colin Ihrig <cjihrig@gmail.com>
Reviewed-By: James M Snell <jasnell@gmail.com>
@maclover7

This comment has been minimized.

Copy link
Member

maclover7 commented Oct 23, 2017

Should this be closed since #14731 has landed? You can access the results from getaddrinfo directly (no re-ordering) by passing the { verbatim: true } option.

@MylesBorins

This comment has been minimized.

Copy link
Member

MylesBorins commented Oct 23, 2017

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented Oct 23, 2017

Yes, it's fixed now. I'll close.

@bnoordhuis bnoordhuis closed this Oct 23, 2017

@douardda

This comment has been minimized.

Copy link

douardda commented Feb 4, 2018

Not sure if it's related, but I'm using the IPv6 only wifi access point at FOSDEM and "nmp install" failed with

24 http request GET https://registry.npmjs.org/dredd
25 silly fetchPackageMetaData Error: connect ENETUNREACH 151.101.0.162:443 - Local (0.0.0.0:0)

@marka63

This comment has been minimized.

Copy link

marka63 commented May 2, 2018

For the record. RFC 1123 stipulates that addresses should be tried in order and that NETWORK errors should be handled for multi-homed hosts. With the introduction of IPv6 and DNS64 there are lots of multi-homed hosts. Happy Eyeballs is just a mechanism for dealing with a common NETWORK error. It is not and never will be a "Transition Mechanism".

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented May 2, 2018

Happy Eyeballs wouldn't be a thing if IPv6 didn't exist, or had worked great out of the gate; it was brought into this world to get over IPv6's early adoption hurdle. Privately, I call it "a godawful hack", but in polite company, "transition mechanism" seems apt.

addresses should be tried in order

Wouldn't you agree that conflicts with Happy Eyeballs' "first pass the post" principle? Most applications, Node.js included, call getaddrinfo(AF_UNSPEC); they accept any mix of IPv4 and IPv6 addresses.

@marka63

This comment has been minimized.

Copy link

marka63 commented May 3, 2018

Firstly, Happy Eyeballs does in order connection attempts. There are very few cases where a the first connection attempt would have succeeded after the second connection attempt succeeds.

Having multiple TCP connection attempts in flight existed before Happy Eyeballs existed. The only thing Happy Eyeballs did was give the practice a IETF stamp of approval. There was never any requirement for a connection attempt to timeout before starting a new connection attempt to a multi-homed server. A large part of the load balancer business is a direct result of bad multi-home connection attempt handing by TCP clients. But if you want to keep propping up a business model that is wasting 10's of millions of dollars a year rather than do the right thing and support simultaneous connection attempt.

Multi-homed servers are here to stay as are networks where routes to some of the addresses don't work. You can learn from other peoples experiences or continue to use naive connection attempt models.

@telmich

This comment has been minimized.

Copy link

telmich commented May 3, 2018

Guys. It's 2018. It's too late to still DISCUSS on how to fix IPv6 connection establishement issue. Move on, fix it.

@apapirovski

This comment has been minimized.

Copy link
Member

apapirovski commented May 3, 2018

  1. Not all "guys" here.
  2. PRs welcome.
@benschulz

This comment has been minimized.

Copy link
Author

benschulz commented May 3, 2018

I think this is the wrong issue for this discussion. As I said before, whether to do Happy Eyeballs—or some other connection establishing scheme for that matter—is orthogonal.
This issue is about the DNS module. All the DNS module does is resolve names. It should respect the system config when doing so, which is what this issue is trying to address, and I think the right steps are being taken. I hope the opt-in turns opt-out in the near future, but that is a breaking change…

By the way, I appreciate frustration—I "wasted" a day on this issue, after all. But please make an effort to check your attitude before commenting. Especially if you want to see your issue addressed. Snark won't help you, even when you're technically right.

@marka63

This comment has been minimized.

Copy link

marka63 commented May 3, 2018

Well opt-out is what is REQUIRED by RFC 1123 regardless of whether is it breaks stuff. The existing stance is causing people to request than name servers return NODATA responses for A record requests despite there being A records at the name.

@bnoordhuis

This comment has been minimized.

Copy link
Member

bnoordhuis commented May 3, 2018

I hope the opt-in turns opt-out in the near future

Node.js 11, most likely.

@refack refack removed their assignment Oct 12, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.