Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

user land proxy uses all RAM memory when exposing a big range of ports #11185

Open
Jaykah opened this issue Mar 5, 2015 · 78 comments

Comments

@Jaykah
Copy link

commented Mar 5, 2015

I have a dockerized SIP application that requires a large number of ports for RTP. Since it recently became possible to expose a port range, I decided to move away from --net=host scenario to using those ranges (-p 30000-40000:30000-40000/udp)

However, when a range is large enough, docker eats up all RAM and fails:

ERRO[0230] Handler for POST /containers/{name:.*}/start returned error: Cannot start container 80df70ab22d94408e9a5a2c60590b1b1281e5a59b5531590738739c9f7c7c485: iptables failed: iptables --wait -t nat -A DOCKER -p udp -d 0/0 --dport 38207 ! -i docker0 -j DNAT --to-destination 172.17.0.3:38207:  (fork/exec /sbin/iptables: cannot allocate memory)
ERRO[0230] HTTP Error: statusCode=500 Cannot start container 80df70ab22d94408e9a5a2c60590b1b1281e5a59b5531590738739c9f7c7c485: iptables failed: iptables --wait -t nat -A DOCKER -p udp -d 0/0 --dport 38207 ! -i docker0 -j DNAT --to-destination 172.17.0.3:38207:  (fork/exec /sbin/iptables: cannot allocate memory)

It would seem logical to combine those ranges when applied to iptables into something like:
--dports 30000:40000, and afaik we do not need to explicitly specify the new destination ports if they're going to match the original ones. Or am I missing something?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Mar 5, 2015

@brahmaroutu perhaps you have thoughts on this issue, since you implemented this in #9097

Also, (possibly) similar, but for EXPOSE; #9021

@brahmaroutu

This comment has been minimized.

Copy link
Contributor

commented Mar 5, 2015

@thaJeztah I tried this test and I get the failure
Error response from daemon: Cannot start container 83bc8a32f7a588903a80ca5dfa6c0d699501ff542c7ad8cf365dd95543ffe4a6: Error starting userland proxy:

But all the ports are mapped when I do an inspect. Looks like it is a timing or memory related issue in the Userland proxy.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Mar 6, 2015

Looks like it is a timing or memory related issue in the Userland proxy

Hmf, really hope we can get rid of that once (the proxy).

@brahmaroutu do you think additional info is required on this? Also, perhaps you are able to answer this;

It would seem logical to combine those ranges when applied to iptables into something like:
--dports 30000:40000, and afaik we do not need to explicitly specify the new destination ports if they're going to match the original ones. Or am I missing something?

@brahmaroutu

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2015

@thaJeztah I agree with @Jaykah that only way to avoid running many parallel jobs to allocate all these ports is to block allocate. Not sure refactoring that code, when Docker network drivers are changing, would be a good idea.
The documentation intended to say that if you specify -p 1-10:100-110 then both the ranges should have 10 ports to match. I did not default the case where -p 1-10 would imply -p 1-10:1-10

@unclejack

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2015

@Jaykah Can you check how many docker proxy instances are started? I wouldn't be surprised if the root cause was the execution of too many docker proxy processes to handle all of these port mappings.

@Jaykah

This comment has been minimized.

Copy link
Author

commented Mar 6, 2015

@unclejack

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2015

@Jaykah Thanks!

@crosbymichael @LK4D4 The user land proxy needs to be removed or at least not enabled for port ranges.

@unclejack unclejack changed the title Docker eats up 100% RAM when exposing a big range of ports user land proxy uses all RAM memory when exposing a big range of ports Mar 6, 2015

@Jaykah

This comment has been minimized.

Copy link
Author

commented Mar 11, 2015

@crosbymichael @LK4D4 @unclejack Hi guys, I understand that the solution to this may take a while, so I was wondering if you could suggest a (temporary) workaround to this issue. If there's a way to disable the proxy that seems like the way to go - unfortunately I'm not familiar enough with Go to do it myself.

Thanks!

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Mar 11, 2015

There's actually some preparations in place to facilitate that (remove the proxy), see #11208 (comment), so there's hope :)

@LK4D4

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2015

@Jaykah I'm pretty sure that there will be solution in 1.6. For now if you know how to build docker you can replace body of this https://github.com/docker/docker/blob/master/daemon/networkdriver/portmapper/proxy.go#L115 function with return nil. But this will break inter-container communication :(

@Jaykah

This comment has been minimized.

Copy link
Author

commented Mar 11, 2015

Thanks guys!

@LK4D4 so I assume I have to change that to:

func (p *proxyCommand) Start() error {
 return nil;
 r, w, err := os.Pipe()
 if err != nil {
  return fmt.Errorf("proxy unable to open os.Pipe %s", err)
 }

Is that correct?

Also, how soon do you think 1.6 will be released?

@LK4D4

This comment has been minimized.

Copy link
Contributor

commented Mar 11, 2015

@Jaykah

func (p *proxyCommand) Start() error {
    return nil
}

It will be beginning of April I think

@MiLk

This comment has been minimized.

Copy link

commented Apr 3, 2015

Same issue here when trying to run https://github.com/QubitProducts/bamboo and listen on 10000-20000.
I had to use --net=host.

@LK4D4

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2015

@MiLk It spawns whole docker binary on each port :)

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2015

This might be good to make only 1 docker-proxy per container (if it is publishing ports) instead of per-port.

@unclejack

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2015

@cpuguy83 I disagree, the only acceptable and proper solution is the complete removal (disabled and deleted from the code) of the user land proxy.

edit: Just to be clear, this has already been in the code once, but older distributions like RHEL6/CentOS6 weren't working with that configuration because of the old 2.6.32 kernel.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Apr 3, 2015

@unclejack We keep talking about that...

@Jaykah

This comment has been minimized.

Copy link
Author

commented Apr 16, 2015

Hey guys. I saw the release of 1.6, any changes to user land proxy in that version?

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Apr 16, 2015

@Jaykah unfortunately, not yet

@allencloud

This comment has been minimized.

Copy link
Contributor

commented May 27, 2015

Hey guys.

Then if I kill the hundreds of docker-proxy processes on my docker daemon machine, will it affect my containers' running?

In addition, I will never use 0.0.0.0:port -> container_ip:port to access container's application.

@LK4D4 @unclejack @thaJeztah

@blop

This comment has been minimized.

Copy link

commented May 28, 2015

Any news to when the userland docker-proxy will be removed?
No milestone yet for this?
Thanks

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented May 29, 2015

@oblop It's optionally disabled on master, will be part of 1.7

@ghost

This comment has been minimized.

Copy link

commented Jun 17, 2015

I also face this problem now. I need to publish a large range of port and it will start a lot of docker-proxy instances. Can anyone tell me how to solve it? I can not use the --net=host because the ssh will not work and some command like passwd will failed.

@LK4D4

This comment has been minimized.

Copy link
Contributor

commented Jun 17, 2015

@darknigh In 1.7 you will be available to disable it with --userland-proxy=false.

@maxhawkins

This comment has been minimized.

Copy link

commented Jul 13, 2015

Did this make it into 1.7? I'm getting "flag not found" when I run with --userland-proxy=false on 1.7.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jul 13, 2015

Did this make it into 1.7? I'm getting "flag not found" when I run with --userland-proxy=false on 1.7.

@maxhawkins yes, it's in 1.7. See: https://github.com/docker/docker/blob/v1.7.0/daemon/config_linux.go#L75. It is an daemon option, so should be provided when starting the docker daemon.

@thaJeztah thaJeztah removed this from the 1.13.0 milestone Feb 20, 2017

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Mar 15, 2017

Working on making docker-proxy from a 1-process per port to 1-process per daemon. This should help drastically in exposing large port ranges, though I expect there some other pieces that need tweaking in the engine as well.

@thaJeztah thaJeztah modified the milestones: 17.05.0, 17.04.0 Mar 21, 2017

@danielapsmaior

This comment has been minimized.

Copy link

commented Apr 6, 2017

Hi,
I'm trying to open a 10000 udp port range. Disabling userland proxy and creating iptables rules myself have resolved the issue apparently.
Is there a way to run those iptables rules automatically when the container starts? The container IP is dynamic. Is there a way to make routing mesh work with this setup?
Thank you all and I hope there's a real fix soon.
Regards.

@cpuguy83

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

@danielapsmaior Honestly, opening up this many ports is absolutely insane and you probably want to use macvlan instead.

@danielapsmaior

This comment has been minimized.

Copy link

commented Apr 6, 2017

@cpuguy83 10 thousand ports is "default" in Asterisk... each VOIP call can consume up to 4 RTP ports. I'll have a look in macvlan, thank you for your answer.

@aboch

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Is there a way to make routing mesh work with this setup?

routing mesh does not require userland-proxy

@danielapsmaior

This comment has been minimized.

Copy link

commented Apr 6, 2017

@aboch but i would have to repeat the iptables config on each node, right?

@aboch

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

No, docker programs the iptables automatically. You should not need to do it manually.
Maybe I did not understand your first comment.

If you start docker daemon with or without --userland-proxy=false, docker will program the iptables for the containers you run with -p. If you start with --userland-proxy=false, extra iptables rules are programmed.

If instead you are running your apps as docker service (you mentioned routing-mesh), and you publish the ports in vip mode (default mode), regardless userland-proxy is on or off, it won't be used to make the services ports accessible from outside of the host. Only iptables rules are used in this case.

$ docker service create --name 10_ports_service  -p 100-110 busybox top
sx6tmo9ybxbhhno5wj4guhh7u
$ sudo iptables -nvL DOCKER-INGRESS
Chain DOCKER-INGRESS (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30010
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30010
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30009
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30009
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30008
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30008
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30007
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30007
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30006
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30006
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30005
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30005
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30004
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30004
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30003
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30003
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30002
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30002
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30001
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30001
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:30000
    0     0 ACCEPT     tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            state RELATED,ESTABLISHED tcp spt:30000
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           
$ 

If instead you using docker service and publishing the ports in host mode, then it will use userland-proxy. But if --userland-proxy=false, then again it will add the necessary iptables rules.

@danielapsmaior

This comment has been minimized.

Copy link

commented Apr 6, 2017

@aboch I'm using docker stack deploy with an yml to publish the services in N swarm nodes. One of the services is Asterisk with 10 thousand ports. The load was getting too high with the port mapping, so I did as stated by ghost.

I'm not exposing the port range in yml anymore, I'm doing the iptables rules manually. That's the issue being discussed on this thread: docker gets slow using the normal method of port range exposing. But I would like to run this iptables config in a script on the stack deploy, when the Asterisk service/container is starting, in all swarm nodes (to maintain routing mesh)...

Thank you for your answer.

@aboch

This comment has been minimized.

Copy link
Contributor

commented Apr 6, 2017

Unfortunately the routing-mesh won't be operational for those ports if you do not specify -p in your service creation.
Because the iptables programming is only one part of it, then there is the ipvs programming in the ingress sandbox.

@raarts

This comment has been minimized.

Copy link

commented Apr 27, 2017

Is there any hope for me that the routing mesh will be enhanced so that creating a service which needs a large range of ports, will generate 1 iptables rule, which forwards the entire range?
If there a feature request for that somewhere? I really need to put VoIP in Docker swarm, but that needs like 10000 udp ports forwarded, which can easily be done with 1 iptables rule. 10000 separate rules is crazy.

@kevzettler

This comment has been minimized.

Copy link

commented Nov 15, 2018

What is the recommended workaround when the host is MacOs because --net=host is not viable per

https://docs.docker.com/network/host/

The host networking driver only works on Linux hosts, and is not supported on Docker for Mac, >Docker for Windows, or Docker EE for Windows Server.

I have a container that on MacOs host that I need to expose a large range of ports on and seeing no viable option

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Nov 15, 2018

@djs55 ^^ do you know a workaround for Docker Desktop?

@djs55

This comment has been minimized.

Copy link

commented Nov 15, 2018

For development use-cases where it's only necessary to be able to connect to the ports from the host, then you could try enabling the experimental SOCKS proxy. This will allow you to connect to the container backend IPs directly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.