Skip to content

use iptables for proxying instead of userspace #3760

Closed
Closed
use iptables for proxying instead of userspace#3760
Labels
priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.release-noteDenotes a PR that will be considered when it comes time to generate release notes.sig/networkCategorizes an issue or PR as relevant to SIG Network.sig/scalabilityCategorizes an issue or PR as relevant to SIG Scalability.
Milestone
v1.2

Description

@thockin
thockin
opened on Jan 23, 2015

I was playing with iptables yesterday, and I protoyped (well, copied from Google hits and mutated) a set of iptables rules that essentially do all the proxying for us without help from userspace. It's not urgent, but I want to file my notes before I lose them.

This has the additional nice side-effect (as far as I can tell) of preserving the source IP and being a large net simplification. Now kube-proxy would just need to sync Services -> iptables. This has the downside of not being compatible with older iptables and kernels. We had a problem with this before - at some point we need to decide just how far back in time we care about.

This can probably be optimized further, but in basic testing, I see sticky sessions working and if I comment that part out I see ~equal probability of hitting each backend. I was not able to get deterministic round-robin working properly (with --nth instead of --probability) but we could come back to that if we want.

This sets up a service portal with the backends listed below

iptables -t nat -N TESTSVC
iptables -t nat -F TESTSVC
iptables -t nat -N TESTSVC_A
iptables -t nat -F TESTSVC_A
iptables -t nat -N TESTSVC_B
iptables -t nat -F TESTSVC_B
iptables -t nat -N TESTSVC_C
iptables -t nat -F TESTSVC_C
iptables -t nat -A TESTSVC -m recent --name hostA --rcheck --seconds 1 --reap -j TESTSVC_A
iptables -t nat -A TESTSVC -m recent --name hostB --rcheck --seconds 1 --reap -j TESTSVC_B
iptables -t nat -A TESTSVC -m recent --name hostC --rcheck --seconds 1 --reap -j TESTSVC_C
iptables -t nat -A TESTSVC -m statistic --mode random --probability 0.333 -j TESTSVC_A
iptables -t nat -A TESTSVC -m statistic --mode random --probability 0.500 -j TESTSVC_B
iptables -t nat -A TESTSVC -m statistic --mode random --probability 1.000 -j TESTSVC_C

iptables -t nat -A TESTSVC_A -m recent --name hostA --set -j DNAT -p tcp --to-destination 10.244.4.6:9376
iptables -t nat -A TESTSVC_B -m recent --name hostB --set -j DNAT -p tcp --to-destination 10.244.1.15:9376
iptables -t nat -A TESTSVC_C -m recent --name hostC --set -j DNAT -p tcp --to-destination 10.244.4.7:9376

iptables -t nat -F KUBE-PORTALS-HOST
iptables -t nat -A KUBE-PORTALS-HOST -d 10.0.0.93/32 -m state --state NEW -p tcp -m tcp --dport 80 -j TESTSVC
iptables -t nat -F KUBE-PORTALS-CONTAINER
iptables -t nat -A KUBE-PORTALS-CONTAINER -d 10.0.0.93/32 -m state --state NEW -p tcp -m tcp --dport 80 -j TESTSVC

Activity

  1. brendandburns commented on Jan 26, 2015

    @brendandburns
    brendandburns
    on Jan 26, 2015
    Contributor

    Cool! I think we should definitely get this merged in. On a separate note, I was seeing the proxy eat ~30% of a core under heavy load, I have to believe that iptables will give us better performance than that.

  2. thockin commented on Jan 26, 2015

    @thockin
    thockin
    on Jan 26, 2015
    MemberAuthor

    We have to prioritize this - it's almost a total rewrite of kube-proxy and
    all the tests thereof. It also has back-compat problems (will not work on
    older kernels or older iptables binaries).

    On Mon, Jan 26, 2015 at 11:06 AM, Brendan Burns notifications@github.com
    wrote:

    Cool! I think we should definitely get this merged in. On a separate note,
    I was seeing the proxy eat ~30% of a core under heavy load, I have to
    believe that iptables will give us better performance than that.

    Reply to this email directly or view it on GitHub
    #3760 (comment)
    .

  3. brendandburns commented on Jan 26, 2015

    @brendandburns
    brendandburns
    on Jan 26, 2015
    Contributor

    Maybe implementing it as a parallel option and slowly migrating makes sense?

    On Mon, Jan 26, 2015 at 12:01 PM, Tim Hockin notifications@github.com
    wrote:

    We have to prioritize this - it's almost a total rewrite of kube-proxy and
    all the tests thereof. It also has back-compat problems (will not work on
    older kernels or older iptables binaries).

    On Mon, Jan 26, 2015 at 11:06 AM, Brendan Burns notifications@github.com
    wrote:

    Cool! I think we should definitely get this merged in. On a separate
    note,
    I was seeing the proxy eat ~30% of a core under heavy load, I have to
    believe that iptables will give us better performance than that.

    Reply to this email directly or view it on GitHub
    <
    #3760 (comment)

    .

    —
    Reply to this email directly or view it on GitHub
    #3760 (comment)
    .

  4. thockin commented on Jan 27, 2015

    @thockin
    thockin
    on Jan 27, 2015
    MemberAuthor

    I'm trying to coax someone else who doesn't know this code well to learn it
    and take it on. I really want to tackle it, but it would be better if
    someone else learned this space (not you! :)

    That said, you also sent (good) email about the massive P1 list - and I
    don;t think this is on that list yet.

    On Mon, Jan 26, 2015 at 1:06 PM, Brendan Burns notifications@github.com
    wrote:

    Maybe implementing it as a parallel option and slowly migrating makes
    sense?

    On Mon, Jan 26, 2015 at 12:01 PM, Tim Hockin notifications@github.com
    wrote:

    We have to prioritize this - it's almost a total rewrite of kube-proxy
    and
    all the tests thereof. It also has back-compat problems (will not work
    on
    older kernels or older iptables binaries).

    On Mon, Jan 26, 2015 at 11:06 AM, Brendan Burns <
    notifications@github.com>
    wrote:

    Cool! I think we should definitely get this merged in. On a separate
    note,
    I was seeing the proxy eat ~30% of a core under heavy load, I have to
    believe that iptables will give us better performance than that.

    Reply to this email directly or view it on GitHub
    <

    #3760 (comment)

    .

    Reply to this email directly or view it on GitHub
    <
    https://github.com/GoogleCloudPlatform/kubernetes/issues/3760#issuecomment-71527216>

    .

    Reply to this email directly or view it on GitHub
    #3760 (comment)
    .

  5. bgrant0607
    added
    sig/networkCategorizes an issue or PR as relevant to SIG Network.
    kind/enhancement
    on Jan 27, 2015
  6. goltermann
    added
    priority/backlogHigher priority than priority/awaiting-more-evidence.
    sig/scalabilityCategorizes an issue or PR as relevant to SIG Scalability.
    on Jan 28, 2015
  7. thockin
    added
    team/cluster
    on Feb 6, 2015

  8. satnam6502 commented on Feb 11, 2015

    @satnam6502
    satnam6502
    on Feb 11, 2015
    Contributor

    Is this a P2? Might it be worth making it a P3 for now?

  9. thockin commented on Feb 11, 2015

    @thockin
    thockin
    on Feb 11, 2015
    MemberAuthor

    I'm hoping to make it work, but we may yet demote it

    On Wed, Feb 11, 2015 at 2:49 PM, Satnam Singh notifications@github.com
    wrote:

    Is this a P2? Might it be worth making it a P3 for now?

    Reply to this email directly or view it on GitHub
    #3760 (comment)
    .

  10. roberthbailey commented on Feb 12, 2015

    @roberthbailey
    roberthbailey
    on Feb 12, 2015
    Contributor

    Doesn't "hope" equate to a P3 that we'll get to if we can?

  11. bgrant0607
    added
    kind/gsoc
    on Feb 19, 2015

  12. bgrant0607 commented on Feb 23, 2015

    @bgrant0607
    bgrant0607
    on Feb 23, 2015
    Member

    From discussion with @thockin: This is a requirement in order to support service port ranges, which aren't required for 1.0, but we would like to support eventually.

  13. bgrant0607
    added
    priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.
    on Feb 23, 2015

  14. 179 remaining items

  15. mcluseau commented on Feb 21, 2016

    @mcluseau
    mcluseau
    on Feb 21, 2016
    Contributor

    I'm using kubernetes 1.1, /opt/bin/kube-proxy ... --proxy-mode=iptables --masquerade-all=false and routing cluser IP network through an host having a kube-proxy. In this setup, my services are seeing the external IP. I use a highly available network namespace who has a external IP and a route to the hosts:

    I0221 01:20:32.695440       1 main.go:224] <A6GSXEKN> Connection from 202.22.xxx.yyy:51954 closed.

  16. yoshiwaan commented on Mar 30, 2016

    @yoshiwaan
    yoshiwaan
    on Mar 30, 2016

    I've learned a lot reading this thread!

    As an FYI this doc states that AWS ELB uses round-robin for TCP connections and least connections for http/https: http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/how-elb-works.html#request-routing

    I agree that focussing on getting requests only to nodes that run pods and to try to serve local pods is the best way to go about it. The nice side benefit of this is there'll be less node-to-node traffic within the cluster and I suppose a latency improvement by always service local requests from service to pod (which I guess is of even more benefit if you have nodes in multiple availability zones in the same cluster).

    In terms of working with a load balancer that doesn't support weighting then you could solve this with your replication controller by trying to always keep the same number of pods on a node (if there's more than 1 per node) and then distributing evenly between them, even if this means having to move pods off of a node in certain situations and allowing only certain replica counts. e.g. for a 4 node cluster with a service connected to a load balancer the only number of acceptable pod replicas would be 1,2,3,4,6,8,9,12,16,20, etc

  17. emaildanwilson commented on Mar 31, 2016

    @emaildanwilson
    emaildanwilson
    on Mar 31, 2016
    Contributor

    We're also looking to solve for traffic to route to local pods only. I'd be fine with the nodeport going away on a node at times when no pods are present locally for a service. This way a simple load balancer TCP health check would prevent requests from going to those nodes. I think if we can at least solve for the iptables\kube-proxy portion of this, then we'll find out what the load balancing implications are when the pods are not balanced across the cluster. I think there are ways to solve that on load balancers without having to set a weight for each node w/ an API call.

    Load balancers already deal w/ this using other dynamic methods. Also, depending on what the service you're running is actually doing inside that container for each api call it may not be able to support 2x the traffic when there are 2 pods on a node vs one anyways. If Kubernetes Limits are set and if maximum levels of usage are being approached on a pod\node could play into this as well which adds yet another layer of complexity to trying to find the right weight setting on the external load balancer.

    I'd say, stay away from that level of complexity and not try to set load balancer weight from kubernetes.

  18. justinsb commented on Apr 1, 2016

    @justinsb
    justinsb
    on Apr 1, 2016
    Member

    @yoshiwaan Can I suggest opening a new issue for the inter-node traffic suggestion, as this issue is now closed. Personally I think a good first step would be to ensure that if a pod is running on the local node, that we route to the local pod. I suspect that this will be sufficient, because you can then scale your RC so that there are pods on every node.

  19. paralin commented on Apr 14, 2016

    @paralin
    paralin
    on Apr 14, 2016
    Contributor

    @justinsb +1, also we're running into a problem now where we need to see client IPs and it's basically impossible with the current setup.

  20. dalanlan commented on Apr 19, 2016

    @dalanlan
    dalanlan
    on Apr 19, 2016
    Contributor

    This could be way too naive, yet I was wondering what's the difference between userspace mode and iptables? I cannot really tell from the user doc.

  21. mcluseau commented on Apr 19, 2016

    @mcluseau
    mcluseau
    on Apr 19, 2016
    Contributor

    Userland mode means kube-proxy handles the connections itself by receiving the connection request from the client and opening a socket to the server, which (1) consume much more CPU and memory and (2) is limited to the number of ports a single can open (<65k). The iptables mode works at a lower level, in the kernel, and uses connection tracking instead, so it's much lighter and handles a lot more connections*.

    (edit) (*) As long as you don't SNAT packets going through, which in turn requires a setup where you are sure packets will cross the connection tracking rules associated with them. For instance, using a routed access design allows you to avoid SNAT, which means the endpoint of the service will see the real client's IP.

  22. dalanlan commented on Apr 19, 2016

    @dalanlan
    dalanlan
    on Apr 19, 2016
    Contributor

    @MikaelCluseau
    meaning kube-proxy is only responsible for setting up and maintaining iptables rules and we no longer get a random local port for each service in iptables mode, right?

  23. mcluseau commented on Apr 19, 2016

    @mcluseau
    mcluseau
    on Apr 19, 2016
    Contributor

    On 04/19/2016 10:51 PM, Emma He wrote:

    meaning kube-proxy is only responsible for setting up and maintaining
    iptables and we no longer get a random local port for each service in
    iptables mode, right?

    Yes.

  24. dalanlan commented on Apr 20, 2016

    @dalanlan
    dalanlan
    on Apr 20, 2016
    Contributor

    Sorry but I absolutely missed this earlier.

    (edit) (*) As long as you don't SNAT packets going through, which in turn requires a setup where you are sure packets will cross the connection tracking rules associated with them. For instance, using a routed access design allows you to avoid SNAT, which means the endpoint of the service will see the real client's IP.

    @MikaelCluseau I was thinking iptables adopts SNAT and DNAT, which is not the case according to you. Could you please clarify this for me?

  25. mcluseau commented on Apr 20, 2016

    @mcluseau
    mcluseau
    on Apr 20, 2016
    Contributor

    On 04/20/2016 01:59 PM, Emma He wrote:

    @MikaelCluseau https://github.com/MikaelCluseau I was thinking
    iptables adopts SNAT and DNAT, which is not the case according to you.
    Could you please clarify this for me?

    It's the tricky part.

    (1) Using service/external IPs requires DNAT.
    (2) If you are sure reply packets will go through the same conntrack
    rule (ie, the same network stack or a replicated conntrack table), you
    can skip the SNAT part (ie, MASQUERADE rules).

    The condition of (2) is usually ok in routed access network designs
    (which is the simplest design I can think of).

    For instance, given

    • a client 1.0.1.1,
    • a service 1.0.2.1,
    • a pod implementing the service 1.0.3.1.

    Then,

    1. Your router/firewall/loadbalancer/host/whatever receives a packet
      for the service so it sees a packet "1.0.1.1 -> 1.0.2.1";
    2. It DNATs it to the endpoint (pod) so the packet will be "1.0.1.1 ->
      1.0.3.1" in the cluster network;
    3. The pod replies with a packet "1.0.3.1 -> 1.0.1.1";
    4. The packet goes through a router/firewall/loadbalancer/host/whatever
      having the conntrack rule, the conntrack system rewrite the packet
      to "1.0.2.1 -> 1.0.1.1" before sending it back to the client.

    If the condition of (2) cannot be met, you have to use SNAT/MASQUERADING
    to be sure that the packet will go back through the
    router/firewall/loadbalancer/host/whatever's conntrack.

  26. thockin commented on Apr 20, 2016

    @thockin
    thockin
    on Apr 20, 2016
    MemberAuthor

    @MikaelCluseau - drop me an email at my github name at google.com - I have
    something for you

    On Tue, Apr 19, 2016 at 8:20 PM, Mikaël Cluseau notifications@github.com
    wrote:

    On 04/20/2016 01:59 PM, Emma He wrote:

    @MikaelCluseau https://github.com/MikaelCluseau I was thinking
    iptables adopts SNAT and DNAT, which is not the case according to you.
    Could you please clarify this for me?

    It's the tricky part.

    (1) Using service/external IPs requires DNAT.
    (2) If you are sure reply packets will go through the same conntrack
    rule (ie, the same network stack or a replicated conntrack table), you
    can skip the SNAT part (ie, MASQUERADE rules).

    The condition of (2) is usually ok in routed access network designs
    (which is the simplest design I can think of).

    For instance, given

    • a client 1.0.1.1,
    • a service 1.0.2.1,
    • a pod implementing the service 1.0.3.1.

    Then,

    1. Your router/firewall/loadbalancer/host/whatever receives a packet
      for the service so it sees a packet "1.0.1.1 -> 1.0.2.1";
    2. It DNATs it to the endpoint (pod) so the packet will be "1.0.1.1 ->
      1.0.3.1" in the cluster network;
    3. The pod replies with a packet "1.0.3.1 -> 1.0.1.1";
    4. The packet goes through a router/firewall/loadbalancer/host/whatever
      having the conntrack rule, the conntrack system rewrite the packet
      to "1.0.2.1 -> 1.0.1.1" before sending it back to the client.

    If the condition of (2) cannot be met, you have to use SNAT/MASQUERADING
    to be sure that the packet will go back through the
    router/firewall/loadbalancer/host/whatever's conntrack.

    —
    You are receiving this because you were mentioned.
    Reply to this email directly or view it on GitHub
    #3760 (comment)

  27. mrkurt commented on Jan 5, 2017

    @mrkurt
    mrkurt
    on Jan 5, 2017

    @justinsb @yoshiwaan did anyone ever create an issue for this? My search fu is failing me, and I have a similar need.

    Can I suggest opening a new issue for the inter-node traffic suggestion, as this issue is now closed. Personally I think a good first step would be to ensure that if a pod is running on the local node, that we route to the local pod. I suspect that this will be sufficient, because you can then scale your RC so that there are pods on every node.

  28. yoshiwaan commented on Jan 6, 2017

    @yoshiwaan
    yoshiwaan
    on Jan 6, 2017

    I didn't raise it myself

  29. mrkurt commented on Jan 6, 2017

    @mrkurt
    mrkurt
    on Jan 6, 2017

    Ahhhhh, I think I found it, this appears to be the feature/fix: kubernetes/enhancements#27

    Seems to be beta as of 1.5.x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    priority/awaiting-more-evidenceLowest priority. Possibly useful, but not yet enough support to actually get it done.release-noteDenotes a PR that will be considered when it comes time to generate release notes.sig/networkCategorizes an issue or PR as relevant to SIG Network.sig/scalabilityCategorizes an issue or PR as relevant to SIG Scalability.

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions