Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

UDP packets from same source port do not round-robin in a kubernetes service #76517

Open
MaurGi opened this Issue Apr 12, 2019 · 2 comments

Comments

Projects
None yet
4 participants
@MaurGi
Copy link

MaurGi commented Apr 12, 2019

What happened:
Have a Go client that opens a UDP connection and then loops forever sending packets to a kubernetes service with multiple replicas in the deployment, configured with type:ClusterIP/sessionAffinity:None

Packets are not load balanced across the pods, they all end up in the same pod until I stop the Go client and restart it.

If I run a script from command line using netcat, and loop it, then the udp packets are correctly load balanced.

Root caused this to having the same source port in the Go program, but I don't want to open a udp connection every time I send a packet, seems very expensive and inefficient.

Also invesitgated on the host iptables, found no trace of established connections or indication of affinity.

What you expected to happen:

UDP packets should be load balanced to all the pods, since there is no concept of connection in UDP and there is no way for me to control this behavior other than creating multiple connections on the client.

This could be optionally configurable with sessionAffinity, but given that I explicitly set sessionAffinity to none I expect the packets to be spread around all the pods in the service/deployment at this time.

How to reproduce it (as minimally and precisely as possible):
Using version v1.14.0 (reproed via minikube)

Instructions to repro:
multiport.zip

  • have docker and minikube installed, minikube running on Ubuntu 18.04
  • open attached zip file
  • run make minikube-deploy to make docker container and deploy to minikube
  • run kubectl get services and note IP address of service multiport
  • run minikube tunnel

in another terminal run the go client:

go run client/multiport_client.go -server <multiport service IP_address> -port 3333 -protocol udp

let this go, it will send packets to the server every second

check the logs of the pods named multiport-* via 'kubectl logs multiport-xyz-abc`

verify that udp messages always go to the same server

stop the go client

Now run the bash test:

for i in {1..100}; do echo "sending"; echo "TEST" | nc -w 1 -u <service IP_address> 3333; done

check the logs of the pods named multiport-*
verify that udp messages now are spread across all the pods

Anything else we need to know?:

See following issue in golang for more information - I initially filed the issue there:
golang/go#31427

Environment:

  • Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.4", GitCommit:"c27b913fddd1a6c480c229191a087698aa92f0b1", GitTreeState:"clean", BuildDate:"2019-02-28T13:37:52Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:45:25Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  • Cloud provider or hardware configuration:
    Ubuntu LTS 18.04 VM hosted on Azure
  • OS (e.g: cat /etc/os-release):
NAME="Ubuntu"
VERSION="18.04.2 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.2 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic
  • Kernel (e.g. uname -a):

Linux mg-net-test 4.18.0-1014-azure #14~18.04.1-Ubuntu SMP Mon Mar 18 16:15:53 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

  • Install tools:
  • Others:
@MaurGi

This comment has been minimized.

Copy link
Author

MaurGi commented Apr 12, 2019

/sig network

@athenabot

This comment has been minimized.

Copy link

athenabot commented Apr 12, 2019

/triage unresolved

🤖 I am a bot run by @vllry. 👩‍🔬

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.