Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IPv6 publish forwards incorrect ip #17666

Open
jtc0de opened this issue Nov 3, 2015 · 23 comments

Comments

@jtc0de
Copy link

commented Nov 3, 2015

IPv6 publish forwards incorrect ip. It's just the IPv4 of the host, not the IPv6 of the client which it should be.

Sorry for yet another rant, but will you ever start taking IPv6 seriously? I feel like half of my tickets are IPv6, because each day I'm discovering how more atrocious docker's current support is.

People have to use publish because of your other glorious decision, public but random IPv6 address assignments. (who ever thought that was a good idea?) And due to your additional insightful decision to not easily allow optionally specifying a fixed IP, that is not a possible workaround for that either. And now publish is also just a half-baked horrible rushed implementation when using it for IPv6.

Really?

I'll just leave this here because people seem to be forgetting:

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Nov 3, 2015

I realize this won't solve your problems directly, but proper IPv6 support is one of the goals for the next (1.10) release. The foundations for better support are added in the 1.9 release with a massive rewrite of the networking model in lib network (additional improvements coming in 1.10 as well).

Just to explain it's on the radar, but one step at a time. You can check progress (or help with development) in the LibNetwork repository https://github.com/docker/libnetwork, or discuss issues in the #docker-network IRC channel.

/cc @mavenugo @mrjana

@timcoote

This comment has been minimized.

Copy link

commented Nov 4, 2015

fwiw. Despite the apparent design goal of stable IPv6 addresses, it looks like ISPs are intent on breaking the assumption that they will be stable. This will create issues as it means that all programs will need to add another layer to get the name binding right. So unpredictable IP addresses will need to be coped with in the general case. I'd guess that this means either using some local discovery mechanism (Zookeeper? etcd?) or something more global (dns? doa?)

Also, it should be noted that IP addresses (v4 and v6) are not properties of hosts. They are properties of interfaces. Identifying hosts by IP address is difficult, except for special cases.

Facebook has found that it is getting benefit from starting with IPv6 versions of its code (the FB AS's are nearly 100% ipv6, and they've measured 30% better performance for users connected by IPv6 compared to IPv4).

@jtc0de

This comment has been minimized.

Copy link
Author

commented Nov 4, 2015

Well but none of that is an excuse to replace the IP entirely with a v4 variant of the forwarding host. IPv6 addresses aren't reliable? Oh great, then let's break them even more!

@timcoote

This comment has been minimized.

Copy link

commented Nov 4, 2015

indeed. I'll try to read the notes that theJeztah has pointed at.

@tomdee tomdee referenced this issue Nov 6, 2015
0 of 3 tasks complete
@jtc0de

This comment has been minimized.

Copy link
Author

commented Jan 4, 2016

Just a small poke since 1.9 is out and I suppose work on 1.10 has started - it would be really really good if this could be finally addressed. (as written above, it would be instrumental in making the random public IPs problem much less of an issue if this worked properly, and it would make the behavior much more parallel to IPv4/what people are used to)

@schwebke

This comment has been minimized.

Copy link

commented Apr 3, 2016

are there any news regarding IPv6 support for exposed ports? My dockerized webserver with exposed port 80 sees the correct IPv4 client address, but in case of a client connecting over IPv6 just the IPv4 ip of the host (docker version 1.10.3).

@jtc0de

This comment has been minimized.

Copy link
Author

commented Apr 6, 2016

@thaJeztah any news available on this?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Apr 6, 2016

@JonasT Docker 1.10 allows you to specify an IP-address for a container (using the --ip and --ip6 options on docker run and docker network connect - also see https://docs.docker.com/engine/userguide/networking/work-with-networks/, would that solve your case?

@jtc0de

This comment has been minimized.

Copy link
Author

commented Apr 6, 2016

No. For two reasons:

  1. I have many services that need to be reachable under the same host (the microservices movement means splitting up even stuff traditionally operating under one host into many containers). This can't be easily set up without port publishing working. DNS for a single host can't magically point to multiple IPs.
  2. EXPOSE making ports reachable from the internet with IPv6 is a serious flaw and a really dangerous default choice which has the following likely outcome that is unacceptable: it makes it much harder to make supposedly "hidden" internal backends only reachable through a proxy for security reasons which works trivially with IPv4 docker, and it will bite a lot of people who are used to their IPv4 setups and aren't aware in this radical change in behavior who will inavertedly expose sensitive backends to the internet. Therefore I'm both advocating a change of that not happening per default, and I'll use iptables to make other containers except my main gateway unreachable apart from minor exceptions - which means I need to rely on properly working port publishing to work.

So yes, please fix this. Even with --ip and directly reachable EXPOSEd ports (which again as above is a huge problem of its own and IMHO a big mistake to have it work that way per default) it has its uses and you lose a lot of flexibility without it.

Also let me note I think NAT Is a bad idea too. But IPv6 containers could still have sensible iptable defaults to not put them right into the internet without explicit user request, by e.g. simply blocking all TCP/IP connections not initiated by the container, and adding some new option that allows them to be reachable from the internet if desired. And unless you fix port publishing, this can't be addressed easily, since it is often required in a setup where containers aren't all directly reachable from the internet...

@robbertkl

This comment has been minimized.

Copy link

commented Apr 11, 2016

It's not just EXPOSEd ports that are publicly accessible. The container gets a publicly routable IPv6 address, meaning anything that binds to this address (or binds to everything, [::]) will be available to the world, unless additional firewall/filters are in place. Even ports that aren't EXPOSEd.

To me, the issue of publicly accessible IPv6 containers is a different (although related) issue to the current issue: published ports not working / not working correctly, which is now being continued at docker/libnetwork#1099.

@jtc0de

This comment has been minimized.

Copy link
Author

commented Apr 12, 2016

It's not just EXPOSEd ports that are publicly accessible. The container gets a publicly routable IPv6 address, meaning anything that binds to this address (or binds to everything, [::]) will be available to the world, unless additional firewall/filters are in place. Even ports that aren't EXPOSEd.

Ok, I made a ticket for that here: #21951

@robbertkl

This comment has been minimized.

Copy link

commented May 6, 2016

I've created docker-ipv6nat to address both the issue of publicly accessible container ports, and the incorrectly forwarded IP. I know NAT for IPv6 is not "the way to go", but this does fix published ports and other issues with IPv6, compared to IPv4. Until Docker has good, practical alternatives to NAT (which I welcome!), you could give this a try.

@timcoote

This comment has been minimized.

Copy link

commented May 10, 2016

@robbertkl I thought that the recommended approach to IPv6 firewalls was to default to not exposing delegated networks. Yes, all containers will get routeable IP addresses, but they should not be reachable by default. With IPv4, how do containers reach containers on other hosts? I'd expect this to be an emerging pattern for containers on any non-datacentre hosts, so surely it's better not to bake in 'instant legacy'?

@robbertkl

This comment has been minimized.

Copy link

commented May 10, 2016

@timcoote What do you mean by "the recommended approach"? Docker does not manage the IPv6 firewall at all, in contrast to IPv4. This means on most systems (as it was on mine) the IP addresses were routable but not firewalled by default. So you have to do this manually, while on IPv4 this happens automatically because of NAT / exposed ports. Not only is this a nuisance, people will forget to do this and accidentally leave their containers "wide open".

With IPv4, how do containers reach containers on other hosts?

With IPv4, you use linked containers or user-defined networks for inter-container communication. The overlay network driver can be used in multi-host setups. For IPv6 this does not exist. Linking doesn't work for IPv6 as far as I know. If you give each container a publicly routable IPv6, yes, of course they can reach each other from different hosts, but so can the rest of the world. You'll then have to manually "plug these leaks" again by managing the IPv6 firewall on all hosts.

so surely it's better not to bake in 'instant legacy'

Lastly, I think you misunderstood my post, because I'm not trying to get this "baked in". I'm just giving a (perhaps temporary) solution to some of the issues; at the same time showing how IPv6 NAT can deal with them. I welcome any alternatives that address these issues in a "non legacy" way, which can hopefully work in a way that's consistent with how IPv4 works. Yes, IPv6 is coming (eventually), but IPv4 will probably stick around for quite some time. I'd hate to see certain Docker functionality work only for 1 of the protocols (like exposed ports right now) and/or having to all the time think about / support the different protocols when designing container images.

@jtc0de

This comment has been minimized.

Copy link
Author

commented May 10, 2016

The truth is apparently the docker company cares more to put fancy features in rather than fix basic IPv6 support... therefore @robbertkl 's practical temporary workaround is greatly appreciated as far as I'm concerned. However, it's no excuse not to fix this.

@lanrat

This comment has been minimized.

Copy link

commented May 16, 2016

I'm also running into this issue.

When running my web and email servers inside docker (without IPv6 enabled, on an IPv6 connected host) the services see IPv6 client IPs as the docker0 IP.

Simple Demo: https://gist.github.com/lanrat/0b2a952b4bff329fc3b124e3e0d1cf9d

While it looks like a a good solution to the "docker with ipv6" problem is some ways away in the meantime would it be possible to update the docker-proxy to set the correct IPv6 remote address to fix issues like these when running services that deal with IPv6 clients?

@robbertkl

This comment has been minimized.

Copy link

commented May 16, 2016

@lanrat As the name suggests, docker-proxy proxies the connection to the container, meaning it will always show the remote address from docker-proxy itself. In your case it uses 172.17.42.1 since it's connecting to a 172.17.42.x container. It's not possible for the proxy to just "set" the remote address.

The only reason IPv4 does show the correct remote address is because it's actually bypassing docker-proxy completely, because the NAT rules catch it before reaching the proxy. If you would disable NAT on IPv4, you would see the same remote address (172.17.42.1) for IPv4 connections.

@lanrat

This comment has been minimized.

Copy link

commented May 16, 2016

@robbertkl thanks for the clarification.

@jtc0de

This comment has been minimized.

Copy link
Author

commented Aug 4, 2016

This issue would be at least partially resolved with the implementation of IPv6 NAT as requested here: #25407

I would suggest it's not unreasonable to expect people who don't want to use IPv6 NAT will most likely want to write their own iptable rules anyway where this can be addressed, so there's probably not anything more needed except good documentation that proper -p/--publish behavior is limited to IPv4 NAT and IPv6 NAT and for other scenarios, you should set up your own port redirects with iptables as desired.

@MichaelEischer

This comment has been minimized.

Copy link

commented May 20, 2017

Docker (using default settings!) forwarding wrong IPs for IPv6 connections to published ports may even cause security issues:

With IPv4-NAT-only containers could rely on the fact that requests originating from 172.16.0.0/20 are in fact from internal sources. This assumption breaks with IPv6 requests being forwarded over the docker-proxy, as these connections get source IPs like 172.17.0.1 thus looking like internal requests.

I've come across a few containers relying on requests from 172.16.0.0/20 being limited to internal (trusted) sources. Examples include simple mail forwarders for internal use, nginx using the real_ip module.

Other problematic situations include: For docker containers which need to provide ssh access (for example for gitlab), there is no way to block bruteforce attempts over IPv6 since all clients connecting via IPv6 seem to use the same IPv4 address. (Well there is, but blocking everyone is no solution...) Or to put it differently: IP address based rate limiting is not possible at all for requests from IPv6 users.

@jtc0de

This comment has been minimized.

Copy link
Author

commented May 20, 2017

Sadly, I'm pretty sure all those culprits are known to the docker developers. Despite of this, they haven't shown a big interest in implementing this properly so far. (probably since most big docker users use custom iptable rules anyway?)

@thaJeztah are you going to change your position on this any time soon? Or are we just gonna sit here and watch people repeat the glaring, big problems with this over and over with nothing happening.. sorry if I sound super frustrated, but at this point you might as well disable comments and/or close the ticket if nothing is going to be done about this anyway.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented May 20, 2017

@JonasT Contributions welcome.

@jtc0de

This comment has been minimized.

Copy link
Author

commented Sep 16, 2018

For what it's worth, some cool person coded this which seems like it would fix this (or at least do most of the work to make a fix possible): docker/libnetwork#2023 - so far it's still waiting for a merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants
You can’t perform that action at this time.