Disable Userland proxy by default

[thaJeztah]

The userland proxy was made optional in #12165 (ported to libnetwork in moby/libnetwork#171),
but still enabled by default because (if I remember correctly) disabling it caused issues on RHEL6 (cf #10676).

Now that we will stop supporting RHEL6 with the coming 1.8 release, I think we can change the default to "disabled". We can still keep the --userland-proxy option around so that users are able to enable it by setting --userland-proxy=true. We can remove the proxy altogether in a future release.

Changing the default might help resolving #11185 (possibly others)

Some issues related to disabling the userland-proxy that we should look into;

• userland-proxy doesn't allow host ip communications #21860
• overlay networking with userland-proxy disabled prevents port exposure #22741
• With --userland-proxy=false, dockerd listens on exposed ports (instead of docker-proxy) #28589
• When using userland-proxy=false many iptables entries instead of multiport #36214
• Cannot disable userland-proxy and go with hairpin #37163
• Cannot communicate with host through gateway IP when userland-proxy: false #40518
• container not able receive UDP traffic when disabled userland-proxy libnetwork#2423
• Broken network connectivity between two containers of different networks over published port on external ip #38784
• userland-proxy: false does not clean-up NAT rule when switching to userland-proxy: true #44721

[thaJeztah]

ping @docker/maintainers @mrjana @mavenugo

[mrjana]

@thaJeztah SGTM. I will push a PR once we get a few more thumbs up from @docker/maintainers.

ping @icecrime as well

[tiborvass]

I personally think @thaJeztah's proposal makes sense. +1 on changing default to false but keeping the flag just in case we have reports of other weird issues.

[LK4D4]

I think we should check if netlink for Hairpin isn't supported with --false and die then.

[tianon]

👍

[icecrime]

I'm all for it, I'm just a bit worried as I don't think so many people are giving --userland-proxy=false a try today (I have it on my host since 1.7.0 without issues), but that's what the freeze period is for.

[thaJeztah]

@icecrime true, but holding off longer won't help there. We've got the flag to give users a way to restore the old behaviour, so even if it results in issues, we can offer users a solution without having to release an update.

[icecrime]

@thaJeztah I agree. Do you want to try this for 1.8.0? I'm assuming we'd need a libnetwork PR to give us HairpinModeSupported() bool.

[thaJeztah]

I think it would be good to have this in 1.8.

@mrjana is already in this discussion and offered to create a PR to accommodate the changes (#14856 (comment))

I'll add it to the milestone (unless @calavera has objections)

[calavera]

sounds great!

[chenchun]

I have a question, does libnetwork or docker checks if a random allocated port is currently used by other processes with --userland-proxy=false? I known previously if failing to start userland-proxy process, such as the port is being allocated by other processes on the host, docker will retry to allocate a new port.

[thaJeztah]

@mrjana looks like there are some issues reported with --userland-proxy=false; #15840 (think there was another one, will add if I remember which one)

[winggundamth]

After I used --userland-proxy=false then I have this problem
#5618

[cpuguy83]

Thanks @winggundamth, we've actually reverted the change that sets this to default because we saw the same thing on our CI.

I'll re-open this now.

[winggundamth]

Thanks @cpuguy83. Hope this can fix soon. In our high load production environment with --userland-proxy=true docker-proxy eat 35% of host memory and continue going up. this really critical for us.

Since I'm not a good coder so I'll be a good tester for this case.

[Soulou]

I want to highlight that #5618 is pretty critical and seems to affect event the most recent kernels. But I agree on the fact that the userland proxy should be disabled. The question is, what's the best workaround then?

[thaJeztah]

For reference; this was reverted for now in #16347, more details can be found on that PR as well

[AkihiroSuda]

@thaJeztah Is there any blocker to move this forward?

[ghost]

I think #25407 / #41622 might kind of qualify as blockers

[AkihiroSuda]

Thanks for the pointer. Can we disable userland-proxy by default when ipv6 is disabled?

[ghost]

I am not aware of any reason why it would be needed for ipv4. Although it sounds like given ipv6 NAT seems to be only weeks away from a merge maybe it would be more consistent to wait until then and just disable it entirely once that is done. Not that it really matters either way though

[thaJeztah]

I added this one to the v21.x planning board (https://github.com/moby/moby/projects/11#card-46682701), to be considered for the next release after v20.10

[ppmathis]

As I've just dealt with an issue of Docker not preserving the source IP address for incoming UDP packets over a bridge network, I tried disabling the userland-proxy myself as often suggested in various issues. I've noticed during testing that while TCP continues to work smoothly, a container running nfacctd no longer received any UDP packets on port 5678 despite there being a valid port-mapping with matching iptables rules.

I then stumbled across moby/libnetwork#2423 and I think this is also an issue that would need fixing before userland-proxy can be safely disabled by default? Basically when any incoming UDP packets hit the machine running Docker before its iptables rules are ready, conntrack stores an incorrect state and until that state expires, none of the packets will ever hit the actual container.

As in my case I was dealing with IPFIX which permanently sends UDP packets, this conntrack state never expires by itself. The only workaround I've found was to run conntrack -D --proto udp after starting Docker by modifying the systemd service unit (with a short delay), which just kills the state of existing UDP connections. This could obviously be fine-tuned as described in the issue I linked to only terminate states relevant to a port-mapping, but IMHO the Docker daemon should do this on its own.

Just wanted to mention this in case it is not already tracked.

[thaJeztah]

@ppmathis thanks! I edited the top description, and linked some tickets related to disabling the user land proxy, so that those can be looked into before switching the default.

[thaJeztah]

I also chatted with the Docker Desktop team, and Docker Desktop now sets up networking without relying on the userland-proxy being enabled, so from their side, switching the default won't be an issue; I think they still set up a proxy though, but through other means.

[kkppll-ss]

Hi, Can I ask the current status of this issue? Is this something you are actively working on?

[thaJeztah]

For Docker Desktop for Mac and Windows, we're looking at disabling the proxy in the default configuration; on Desktop we have more "control" / "insight" in the exact configuration / setup in which the daemon runs. For Linux, it's a bit more complicated, because there are many setups/scenarios where the daemon runs, which makes it more risky to change the default (as mentioned in this ticket, we tried changing the default, and got reports from users where changing the default caused issues).

Note that if you want it disabled, you can already do this by setting the "userland-proxy": false option in daemon.json (https://docs.docker.com/engine/reference/commandline/dockerd/#daemon-configuration-file)

A good starting point to get the default changed would be if people could help testing the scenarios linked in the top description (and other issues linked to this ticket), to verify if they're still relevant and/or if changing the default would cause a regression for those.
If we're able to verify that those scenarios are no longer relevant, I think we can consider giving it another go (and change the default to have the user land proxy "disabled" by default).

[4ndreasH]

I tried to switch off the userland-proxy to have access to the external client IP addresses but my docker network is only reachable from internet with userland-proxy: true. The comments here sound like the reason for my problem is probably IPv6.
My local local network is connected to the internet via DS lite, meaning I only have IPv6 for external access. Does "there is no ipv6 NAT" mean: the userland-proxy has to be switched on if IPv6 has to be used?
To use tools like fail2ban inside docker I need to know the clients IP address. How can this be done with IPv6 in this case?
(Sorry if I misunderstood the problem. I'm not a network guy and appreciate any hints where to read more about the userland proxy).
(openSUSE Tumbleweed Linux with docker 20.10.12-ce)

[unilynx]

@4ndreasH you need the userland proxy for ipv6 or set up the necessary firewall rules yourself.

an alternative may be to use host networking (ie no network isolation at all), but that comes with its own tradeoffs.

[polarathene]

Potentially relevant issue: #44721 (comment)

If disabling becomes the default, I imagine working around the issue when changing to userland-proxy: true will be more of a hassle?

---

UPDATE: Instead of needing to reboot with config updated to enable the proxy, you can fix it by deleting the iptables + ip6tables rules the as linked comment demonstrates for IPv4 with iptables.

There is a lingering postrouting masquerade rule that userland-proxy: false adds, and persists when switching to userland-proxy: true, but this rule is not there on a fresh boot of the system when starting the docker daemon with userland-proxy: true.

[polarathene]

A good starting point to get the default changed would be if people could help testing the scenarios linked in the top description (and other issues linked to this ticket), to verify if they're still relevant and/or if changing the default would cause a regression for those. - (Ref: #14856 (comment))

I went over most of the remaining checklist items and commented on each (except for multi-port and swarm overlay network). All but the last one below seem resolved now?

---

Checklist items

Summary

UPDATE: As it's been three weeks without the actual checklist being updated since I commented, here's an updated reference:

• userland-proxy doesn't allow host ip communications #21860
• overlay networking with userland-proxy disabled prevents port exposure #22741
• With --userland-proxy=false, dockerd listens on exposed ports (instead of docker-proxy) #28589
• When using userland-proxy=false many iptables entries instead of multiport #36214
• Cannot disable userland-proxy and go with hairpin #37163
• Cannot communicate with host through gateway IP when userland-proxy: false #40518
• container not able receive UDP traffic when disabled userland-proxy libnetwork#2423
• Broken network connectivity between two containers of different networks over published port on external ip #38784 (RESOLVABLE)
• userland-proxy: false does not clean-up NAT rule when switching to userland-proxy: true #44721

Additional checklist (presently not tracked):

• Issues referencing this thread:
  • Document how to get real remote client ip for service running in container #15086 (Mostly outdated + resolvable via better docs)
• Individual comments in this thread:
  • Kernel crash (unregister_netdevice) (No longer valid)
  • Kernel crash (unregister_netdevice, duplicate report?) (No longer valid)
  • Excessive memory allocation when mapping large port ranges (Only relevant to userland-proxy: true)
  • UDP race condition (Resolved?)
  • Separate setting to apply only hairpin NAT iptables rules (Not a blocker for default change? May warrant separate feature request?)
  • Cannot reach container via IPv6 when userland-proxy: false (Lacks context, needed ip6tables: true)

---

Click to view reasoning for checklist status change

Can close

• userland-proxy doesn't allow host ip communications #21860

#21860 (comment)

Unable to reproduce. Two examples below show connectivity is working fine either way.

Issue has likely been resolved. Should be fine to close since no one else has chimed in here or reacted to the reported.

---

• With --userland-proxy=false, dockerd listens on exposed ports (instead of docker-proxy) #28589

#28589 (comment)

I think this can be closed?:

• UDP issues have been resolved.

• Port publishing should be done if you're going to bind it anyway.

  • If you don't need it for IPv6, then be specific and bind only to IPv4 on the host.
  • Your host can access a services ports via IP directly if necessary, instead of binding to another IP.

---

• Cannot disable userland-proxy and go with hairpin #37163

#37163 (comment)

Looks like this can be closed?

Not much engagement in the issue over the years and reading through it seems there was a misunderstanding?

---

• Cannot communicate with host through gateway IP when userland-proxy: false #40518

#40518 (comment)

Reproduced, seems to be effectively the same issue as #38784 (which goes more into detail to what's causing it)

Can technically close as duplicate?

---

Still an issue

• Broken network connectivity between two containers of different networks over published port on external ip #38784

Summarized and verified in #38784 (comment)

userland-proxy: false prevents containers connecting across separate docker networks on the same host via a published port on the host IP.

Click to view reasoning for additional checklist status change

Additional references in this discussion

#15086

This was a lengthy and old discussion to wade through, and a common pain point users run into. I summarized the issue: #15086 (comment)

Majority of problems there were related to IPv6 and localhost:

• Since the issue was created, IPv6 support has improved and we now have ip6tables (just need to drop the experimental dependency, and potentially enable by default with default-address-pools + ULA subnet).
• localhost concerns with client IP may be tied to hobby deployment or dev environments. Both which could probably avoid the issue by approaching the connection differently. Rarely did anyone expand on context, so I'm more inclined that it's related to misconfiguration + lack of docs / advice.

I believe improved IPv6 docs will resolve this. While macOS / Windows Docker Desktop users and anyone using localhost that runs into it, could share more context. Chances are docs can resolve that also.

Like other additional checklist items detailed below, I think it should be closed as reasoned by the summary I linked. Let a fresh new issue(s) continue to track any remaining concerns, and gather additional context so those can be addressed (likely via documentation).

Related to the quoted comment, two comments after it discuss userland-proxy: false not being viable for "Docker for Mac", and a user saying that is a problem for their production web deployment (that apparently relied on "Docker for Mac"?). Perhaps a misunderstanding on their part. Later an update about "Docker Desktop" notes that networking no longer relies on userland-proxy being enabled and another comment regarding defaulting to disabled for "Docker Desktop" while Linux is a bit more complicated to have confidence in.

---

I want to highlight that #5618 is pretty critical and seems to affect event the most recent kernels. - (Ref: #14856 (comment))

This was a big issue to go over, but I summarized it and my investigation: #5618 (comment)

That problem was a motivator for reverting the previous attempt to disable userland-proxy by default. I found several reproduction examples that no longer produce the error (at least on the modern Vultr VPS environment I ran them on).

---

--userland-proxy=false breaks kernel in a very bad way #5618 and #17443 - (Ref: #14856 (comment))

No longer valid: #17443 (comment)

---

Changing the default might help resolving #11185 (possibly others) - (Ref: #14856 (comment))

It doesn't look like this issue needs to be open anymore: #11185 (comment)

If it does, it'd be better served with a new issue, just like #5618 would due to the amount of time that has passed and how the history of discussion is filled with noise that doesn't help towards further investigating / resolving.

It's also considered that userland-proxy: false (or the removal of userland-proxy in future) is the desired fix.

---

I then stumbled across moby/libnetwork#2423 and I think this is also an issue that would need fixing before userland-proxy can be safely disabled by default?
Basically when any incoming UDP packets hit the machine running Docker before its iptables rules are ready, conntrack stores an incorrect state and until that state expires, none of the packets will ever hit the actual container. - (Ref: #14856 (comment))

UDP was another common issue I have seen reported. I think that was resolved earlier this year with: #44742

---

I'm using docker with the --iptables=false flag set, because I need to manually set routing with a custom bridge.
I'd like to disable the userland proxy, but since the iptables option is set to false, no hairpin nat rules will be generated.
Maybe a 3rd option should be available for iptables, that only adds these rules. - (Ref: #14856 (comment))

Not sure if this is a concern. I think originally before the optional userland-proxy setting arrived, there was earlier attempts for a --hairpin-nat option that would add these. But if you've disabled iptables, does it still make sense to support adding just the hairpin nat rules?

They could perhaps be documented with the other iptables info, although those docs also advise against disabling iptables stating it can break Docker and adding iptables rules can still happen even if you disable the setting.

---

A comment queried about IPv6 with userland-proxy, on a IPv6 only host, but this is lacking additional context.

It may have been a lack of ip6tables for ULA, or potentially needing NDP setup properly for GUA. They admit to not being too familiar with networking, and when NDP is required, some VPS services implicitly manipulate network config when Docker adds/removes a veth interface, which can reset proxy_ndp.

Actually it was probably an IPv6-only host with IPv4 only docker network which userland-proxy: true supports without ip6tables: true, while userland-proxy: false does not