High number of active flow counts in load balancer

[prashant-ztech]

I am running statsrelay as repeater daemon in my application boxes(ec2 m4.large) and forwarding the udp packets to group of boxes listening behind aws network load balancer(udp). This is opening around 50K connection flows in the NLB. The count of metrics published by the application boxes is around 10K/second. What is the reason behind opening such a high number of flows?

[jjneely]

From Amazon:

For UDP traffic, the load balancer selects a target using a flow hash algorithm based on the protocol, source IP address, source port, destination IP address, and destination port. A UDP flow has the same source and destination, so it is consistently routed to a single target throughout its lifetime. Different UDP flows have different source IP addresses and ports, so they can be routed to different targets.

So a flow is a unique source IP/port and destination IP/port. Being that UPD is a connectionless protocol, I'm probably not setting the source port. My guess is that's not consistent. Have you used tcpdump before? That would give us the information we need about the UDP traffic.

[prashant-ztech]

Hi Jack,
Thanks for your quick response. By running tcpdump i found the following

Application to statsrelay traffic

06:07:01.810427 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP
06:07:01.813446 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP
06:07:01.813466 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP
06:07:01.813476 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP
06:07:01.821807 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP
06:07:01.823057 IP 127.0.0.1.37912 > 127.0.0.1.8125: UDP

statsrelay to NLB traffic

06:07:35.781796 IP 10.0.X.X.56544 > 10.0.Y.Y.8125: UDP
06:07:35.781877 IP 10.0.X.X.37597 > 10.0.Y.Y.8125: UDP
06:07:35.781935 IP 10.0.X.X.33168 > 10.0.Y.Y.8125: UDP
06:07:35.781988 IP 10.0.X.X.42942 > 10.0.Y.Y.8125: UDP
06:07:35.782038 IP 10.0.X.X.41297 > 10.0.Y.Y.8125: UDP

As you said the source port is not set. How can we solve this?

[jjneely]

Find the ListenUDP() function. Around line 199. Docs:

https://golang.org/pkg/net/#ListenUDP

The last parameter there is nil which builds the source address "automatically." Patch that to specify the port. Docs for that data structure are here: https://golang.org/pkg/net/#UDPAddr

I'd be glad to accept that patch.

[prashant-ztech]

Hello Jack,

I have made some changes in the similar lines and running this awesome tool in production. See if you like the idea.