Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable Userland proxy by default #14856

Open
7 tasks
thaJeztah opened this issue Jul 22, 2015 · 48 comments
Open
7 tasks

Disable Userland proxy by default #14856

thaJeztah opened this issue Jul 22, 2015 · 48 comments

Comments

@thaJeztah
Copy link
Member

@thaJeztah thaJeztah commented Jul 22, 2015

The userland proxy was made optional in #12165 (ported to libnetwork in moby/libnetwork#171),
but still enabled by default because (if I remember correctly) disabling it caused issues on RHEL6 (cf #10676).

Now that we will stop supporting RHEL6 with the coming 1.8 release, I think we can change the default to "disabled". We can still keep the --userland-proxy option around so that users are able to enable it by setting --userland-proxy=true. We can remove the proxy altogether in a future release.

Changing the default might help resolving #11185 (possibly others)

Some issues related to disabling the userland-proxy that we should look into;

  • #21860 userland-proxy doesn't allow host ip communications
  • #22741 overlay networking with userland-proxy disabled prevents port exposure
  • #28589 With --userland-proxy=false, dockerd listens on exposed ports (instead of docker-proxy)
  • #36214 When using userland-proxy=false many iptables entries instead of multiport
  • #37163 Cannot disable userland-proxy and go with hairpin
  • #40518 Cannot communicate with host through gateway IP when userland-proxy: false
  • moby/libnetwork#2423 container not able receive UDP traffic when disabled userland-proxy
@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jul 22, 2015

ping @docker/maintainers @mrjana @mavenugo

@mrjana
Copy link
Contributor

@mrjana mrjana commented Jul 22, 2015

@thaJeztah SGTM. I will push a PR once we get a few more thumbs up from @docker/maintainers.

ping @icecrime as well

@tiborvass
Copy link
Collaborator

@tiborvass tiborvass commented Jul 22, 2015

I personally think @thaJeztah's proposal makes sense. +1 on changing default to false but keeping the flag just in case we have reports of other weird issues.

@LK4D4
Copy link
Contributor

@LK4D4 LK4D4 commented Jul 22, 2015

I think we should check if netlink for Hairpin isn't supported with --false and die then.

@tianon
Copy link
Member

@tianon tianon commented Jul 22, 2015

@icecrime
Copy link
Contributor

@icecrime icecrime commented Jul 22, 2015

I'm all for it, I'm just a bit worried as I don't think so many people are giving --userland-proxy=false a try today (I have it on my host since 1.7.0 without issues), but that's what the freeze period is for.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jul 22, 2015

@icecrime true, but holding off longer won't help there. We've got the flag to give users a way to restore the old behaviour, so even if it results in issues, we can offer users a solution without having to release an update.

@icecrime
Copy link
Contributor

@icecrime icecrime commented Jul 22, 2015

@thaJeztah I agree. Do you want to try this for 1.8.0? I'm assuming we'd need a libnetwork PR to give us HairpinModeSupported() bool.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jul 22, 2015

I think it would be good to have this in 1.8.

@mrjana is already in this discussion and offered to create a PR to accommodate the changes (#14856 (comment))

I'll add it to the milestone (unless @calavera has objections)

@thaJeztah thaJeztah added this to the 1.8.0 milestone Jul 22, 2015
@calavera
Copy link
Contributor

@calavera calavera commented Jul 22, 2015

sounds great!

@chenchun
Copy link
Contributor

@chenchun chenchun commented Jul 24, 2015

I have a question, does libnetwork or docker checks if a random allocated port is currently used by other processes with --userland-proxy=false? I known previously if failing to start userland-proxy process, such as the port is being allocated by other processes on the host, docker will retry to allocate a new port.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Aug 30, 2015

@mrjana looks like there are some issues reported with --userland-proxy=false; #15840 (think there was another one, will add if I remember which one)

@winggundamth
Copy link

@winggundamth winggundamth commented Sep 24, 2015

After I used --userland-proxy=false then I have this problem
#5618

@cpuguy83
Copy link
Member

@cpuguy83 cpuguy83 commented Sep 24, 2015

Thanks @winggundamth, we've actually reverted the change that sets this to default because we saw the same thing on our CI.

I'll re-open this now.

@cpuguy83 cpuguy83 reopened this Sep 24, 2015
@winggundamth
Copy link

@winggundamth winggundamth commented Sep 24, 2015

Thanks @cpuguy83. Hope this can fix soon. In our high load production environment with --userland-proxy=true docker-proxy eat 35% of host memory and continue going up. this really critical for us.

Since I'm not a good coder so I'll be a good tester for this case.

@Soulou
Copy link
Contributor

@Soulou Soulou commented Sep 24, 2015

I want to highlight that #5618 is pretty critical and seems to affect event the most recent kernels. But I agree on the fact that the userland proxy should be disabled. The question is, what's the best workaround then?

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Oct 2, 2015

For reference; this was reverted for now in #16347, more details can be found on that PR as well

@Xyaren
Copy link

@Xyaren Xyaren commented Feb 9, 2019

If I understand correctly, disabling the userland-proxy will stop ipv6 reachability out of the box, as of now, there are no ip6tables rules set.

@agners
Copy link

@agners agners commented Nov 22, 2019

It seems that #5618 is indeed resolved in newer kernels, so can user-landproxy=false considered safe?

@Leo-Icario
Copy link

@Leo-Icario Leo-Icario commented Apr 14, 2020

Can anyone shed some light on the status of this issue?

@thaJeztah thaJeztah added this to To do in 20.10 planning via automation Apr 14, 2020
@ghost
Copy link

@ghost ghost commented Apr 14, 2020

Isn't IPv6-NAT still missing? There were occasional declarations that somebody wants to look at it, but so far it doesn't seem like that was actually done 🤷‍♀️

If I understand the comments above correctly, this would degrade out-of-the-box IPv6 from kinda-works-but-with-wrong-proxy-IPv4-address to IPv6 won't work. There is the alternative to explicitly enable direct IPv6 usage for containers but it has completely different semantics and no working publish mechanism, so it's kind of useless to many outside of datacenter setups that use custom iptables anyway (who won't be using the userland proxy even by today)

Relevant ticket is here: #25407

@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Nov 30, 2020

@thaJeztah Is there any blocker to move this forward?

@ghost
Copy link

@ghost ghost commented Nov 30, 2020

I think #25407 / #41622 might kind of qualify as blockers

@AkihiroSuda
Copy link
Member

@AkihiroSuda AkihiroSuda commented Nov 30, 2020

Thanks for the pointer. Can we disable userland-proxy by default when ipv6 is disabled?

@ghost
Copy link

@ghost ghost commented Nov 30, 2020

I am not aware of any reason why it would be needed for ipv4. Although it sounds like given ipv6 NAT seems to be only weeks away from a merge maybe it would be more consistent to wait until then and just disable it entirely once that is done. Not that it really matters either way though

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Nov 30, 2020

I added this one to the v21.x planning board (https://github.com/moby/moby/projects/11#card-46682701), to be considered for the next release after v20.10

@ppmathis
Copy link

@ppmathis ppmathis commented Jan 2, 2021

As I've just dealt with an issue of Docker not preserving the source IP address for incoming UDP packets over a bridge network, I tried disabling the userland-proxy myself as often suggested in various issues. I've noticed during testing that while TCP continues to work smoothly, a container running nfacctd no longer received any UDP packets on port 5678 despite there being a valid port-mapping with matching iptables rules.

I then stumbled across moby/libnetwork#2423 and I think this is also an issue that would need fixing before userland-proxy can be safely disabled by default? Basically when any incoming UDP packets hit the machine running Docker before its iptables rules are ready, conntrack stores an incorrect state and until that state expires, none of the packets will ever hit the actual container.

As in my case I was dealing with IPFIX which permanently sends UDP packets, this conntrack state never expires by itself. The only workaround I've found was to run conntrack -D --proto udp after starting Docker by modifying the systemd service unit (with a short delay), which just kills the state of existing UDP connections. This could obviously be fine-tuned as described in the issue I linked to only terminate states relevant to a port-mapping, but IMHO the Docker daemon should do this on its own.

Just wanted to mention this in case it is not already tracked.

@thaJeztah thaJeztah added this to To do in 21.x Planning Jan 11, 2021
@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jan 21, 2021

@ppmathis thanks! I edited the top description, and linked some tickets related to disabling the user land proxy, so that those can be looked into before switching the default.

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Jan 21, 2021

I also chatted with the Docker Desktop team, and Docker Desktop now sets up networking without relying on the userland-proxy being enabled, so from their side, switching the default won't be an issue; I think they still set up a proxy though, but through other means.

@ballerburg9005

This comment was marked as spam.

@kkppll-ss
Copy link

@kkppll-ss kkppll-ss commented Sep 18, 2021

Hi, Can I ask the current status of this issue? Is this something you are actively working on?

@thaJeztah
Copy link
Member Author

@thaJeztah thaJeztah commented Sep 20, 2021

For Docker Desktop for Mac and Windows, we're looking at disabling the proxy in the default configuration; on Desktop we have more "control" / "insight" in the exact configuration / setup in which the daemon runs. For Linux, it's a bit more complicated, because there are many setups/scenarios where the daemon runs, which makes it more risky to change the default (as mentioned in this ticket, we tried changing the default, and got reports from users where changing the default caused issues).

Note that if you want it disabled, you can already do this by setting the "userland-proxy": false option in daemon.json (https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)

A good starting point to get the default changed would be if people could help testing the scenarios linked in the top description (and other issues linked to this ticket), to verify if they're still relevant and/or if changing the default would cause a regression for those.
If we're able to verify that those scenarios are no longer relevant, I think we can consider giving it another go (and change the default to have the user land proxy "disabled" by default).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
21.x Planning
  
To do
Linked pull requests

Successfully merging a pull request may close this issue.

None yet