New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable Userland proxy by default #14856

Open
thaJeztah opened this Issue Jul 22, 2015 · 33 comments

Comments

Projects
None yet
@thaJeztah
Member

thaJeztah commented Jul 22, 2015

The userland proxy was made optional in #12165 (ported to libnetwork in docker/libnetwork#171),
but still enabled by default because (if I remember correctly) disabling it caused issues on RHEL6 (cf #10676).

Now that we will stop supporting RHEL6 with the coming 1.8 release, I think we can change the default to "disabled". We can still keep the --userland-proxy option around so that users are able to enable it by setting --userland-proxy=true. We can remove the proxy altogether in a future release.

Changing the default might help resolving #11185 (possibly others)

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2015

ping @docker/maintainers @mrjana @mavenugo

@mrjana

This comment has been minimized.

Contributor

mrjana commented Jul 22, 2015

@thaJeztah SGTM. I will push a PR once we get a few more thumbs up from @docker/maintainers.

ping @icecrime as well

@tiborvass

This comment has been minimized.

Collaborator

tiborvass commented Jul 22, 2015

I personally think @thaJeztah's proposal makes sense. +1 on changing default to false but keeping the flag just in case we have reports of other weird issues.

@LK4D4

This comment has been minimized.

Contributor

LK4D4 commented Jul 22, 2015

I think we should check if netlink for Hairpin isn't supported with --false and die then.

@tianon

This comment has been minimized.

Member

tianon commented Jul 22, 2015

@icecrime

This comment has been minimized.

Contributor

icecrime commented Jul 22, 2015

I'm all for it, I'm just a bit worried as I don't think so many people are giving --userland-proxy=false a try today (I have it on my host since 1.7.0 without issues), but that's what the freeze period is for.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2015

@icecrime true, but holding off longer won't help there. We've got the flag to give users a way to restore the old behaviour, so even if it results in issues, we can offer users a solution without having to release an update.

@icecrime

This comment has been minimized.

Contributor

icecrime commented Jul 22, 2015

@thaJeztah I agree. Do you want to try this for 1.8.0? I'm assuming we'd need a libnetwork PR to give us HairpinModeSupported() bool.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jul 22, 2015

I think it would be good to have this in 1.8.

@mrjana is already in this discussion and offered to create a PR to accommodate the changes (#14856 (comment))

I'll add it to the milestone (unless @calavera has objections)

@thaJeztah thaJeztah added this to the 1.8.0 milestone Jul 22, 2015

@calavera

This comment has been minimized.

Contributor

calavera commented Jul 22, 2015

sounds great!

@chenchun

This comment has been minimized.

Contributor

chenchun commented Jul 24, 2015

I have a question, does libnetwork or docker checks if a random allocated port is currently used by other processes with --userland-proxy=false? I known previously if failing to start userland-proxy process, such as the port is being allocated by other processes on the host, docker will retry to allocate a new port.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 30, 2015

@mrjana looks like there are some issues reported with --userland-proxy=false; #15840 (think there was another one, will add if I remember which one)

@winggundamth

This comment has been minimized.

winggundamth commented Sep 24, 2015

After I used --userland-proxy=false then I have this problem
#5618

@cpuguy83

This comment has been minimized.

Contributor

cpuguy83 commented Sep 24, 2015

Thanks @winggundamth, we've actually reverted the change that sets this to default because we saw the same thing on our CI.

I'll re-open this now.

@cpuguy83 cpuguy83 reopened this Sep 24, 2015

@winggundamth

This comment has been minimized.

winggundamth commented Sep 24, 2015

Thanks @cpuguy83. Hope this can fix soon. In our high load production environment with --userland-proxy=true docker-proxy eat 35% of host memory and continue going up. this really critical for us.

Since I'm not a good coder so I'll be a good tester for this case.

@Soulou

This comment has been minimized.

Contributor

Soulou commented Sep 24, 2015

I want to highlight that #5618 is pretty critical and seems to affect event the most recent kernels. But I agree on the fact that the userland proxy should be disabled. The question is, what's the best workaround then?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Oct 2, 2015

For reference; this was reverted for now in #16347, more details can be found on that PR as well

@mrschyte

This comment has been minimized.

mrschyte commented Oct 27, 2015

I'm using docker with the --iptables=false flag set, because I need to manually set routing with a custom bridge. I'd like to disable the userland proxy, but since the iptables option is set to false, no hairpin nat rules will be generated. Maybe a 3rd option should be available for iptables, that only adds these rules.

@oleynikd

This comment has been minimized.

oleynikd commented Nov 4, 2015

@mrschyte I'm having exactly the same issue. Have you found some workarounds?

@mrschyte

This comment has been minimized.

mrschyte commented Nov 4, 2015

Denis, I'm currently using docker by setting a fixed ip for the containers.
This can be done using the lxc options before 1.9, and I think 1.9 has a
command line option for it. Another solution would be to listen to docker
events using a script and generate the rules on the fly, but this can be
quite tricky.

@Silex

This comment has been minimized.

Silex commented Jan 28, 2016

Disabling the userland proxy seems to have no effect on my nginx containers... is that normal?

root@box: $ docker version
Client:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.1
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   a34a1d5
 Built:        Fri Nov 20 13:12:04 UTC 2015
 OS/Arch:      linux/amd64
root@box: $ docker daemon --userland-proxy=false
root@box: $ docker run -it --rm -p 80:80 nginx 
172.17.0.1 - - [28/Jan/2016:13:47:14 +0000] "GET / HTTP/1.1" 304 0 "-" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/48.0.2564.82 Chrome/48.0.2564.82 Safari/537.36" "-"

Should I create a new issue about it? As you can see the remote IP is still 172.17.0.1.

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jan 28, 2016

@Silex you need to change the daemon configuration file, just running docker daemon on the command line like that will only start a second daemon

@Silex

This comment has been minimized.

Silex commented Jan 29, 2016

@thaJeztah: I did that first, and as it was not working (still docker ips) I simply stopped the docker service entirely and ran docker daemon manually to be sure it was running with the proper parameters. The paste above are commands run in different terminals... if you want a testcase:

# Terminal 1
root@box: $ service docker stop
root@box: $ docker daemon --userland-proxy=false
# Terminal 2
root@box: $ docker run -it --rm -p 80:80 nginx
# Terminal 3
root@box: $ telnet 127.0.0.1 80
GET / HTTP/1.0

EDIT: could it be that it only works for clients on another computer and not locally? I'll test
EDIT2: indeed, it works if the remote client is another computer. Testing locally does not work.

@Silex

This comment has been minimized.

Silex commented Feb 1, 2016

Should I open an issue for the fact that testing on the same computer doesn't show 127.0.0.1 but the docker0 ip instead?

@sly-x86

This comment has been minimized.

sly-x86 commented Feb 25, 2016

Same problem, change value of --userland-proxy doesn't helped
OS: 15.10 (Wily Werewolf)

root: docker version
Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:40:35 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:40:35 2016
 OS/Arch:      linux/amd64


root: docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 10
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 17
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins:
 Volume: local
 Network: host bridge null
Kernel Version: 4.2.0-19-generic
Operating System: Ubuntu 15.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.71 GiB
Name: xxx
ID: ---
WARNING: No swap limit support
@JonasT

This comment has been minimized.

JonasT commented Apr 11, 2016

Could you try not to break IPv6 port publishing entirely in the process please?

You should probably assign someone to take care of IPv6 and make a more coherent concept... the situation seems to be randomly getting a bit better, then worse, and then nothing happens for a really long time. It would really be good if you made up your mind about proper IPv6 integration quickly (in a way that is coherent with how the IPv4 setup works, most importantly EXPOSE and publish either working for both or none in similar ways), and then stuck to it for once.

@twang2218

This comment has been minimized.

twang2218 commented Aug 10, 2016

Is there any update on this one? I got the same source IP problem as #15086, which can be solved by --userland-proxy=false. I can add the option to some docker hosts, however, I didn't find an easy way to add the option to the Docker for Mac. So, it would be great if --userland-proxy is false by default.

@LK4D4

This comment has been minimized.

Contributor

LK4D4 commented Aug 10, 2016

@twang2218 --userland-proxy=false breaks kernel in a very bad way #5618 and #17443

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Aug 10, 2016

@twang2218 I don't think disabling --userland-proxy in Docker for Mac will be possible; there's a different version of the proxy running in there to integrate with OS X (IIUC)

@twang2218

This comment has been minimized.

twang2218 commented Aug 10, 2016

@LK4D4 It's sad to hear that. That means I cannot use --userland-proxy=false in production.

Then, how to solve the source IP problem without using --userland-proxy=false? It's quite important to have the client IPs for web apps, such as, returning the content based on the clients geolocation, web log analysis, etc.

@justincormack

This comment has been minimized.

Contributor

justincormack commented Aug 10, 2016

That is correct. At some point we would like to look at alternative ways of
doing this in Docker for Mac, but it is not simple.

@cirocosta

This comment has been minimized.

Contributor

cirocosta commented Jun 16, 2017

Hey, sorry for resurrecting this issue but what is the overall status on this? Is using userland-proxy in a recent kernel still a problem or from #5618 we can't know yet?

Thx!

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Jun 16, 2017

@cirocosta yes #5618 is still an issue, so we don't disable it by default. You may not hit the issue in your particular setup / use, so if you're not hit by that issue you can disable it (YMMV)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment