feat: network shaping #1280
Merged
feat: network shaping #1280
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3Hren
added
S: Worker
3Hren
requested review from
antmat,
zavgorodnii,
nikonov1101 and
a team
as code owners
3Hren
force-pushed
the
feat/network-limits
branch
from
6b11dea
to
c57c818
Compare
nikonov1101
previously approved these changes
Aug 8, 2018
3Hren
force-pushed
the
feat/network-limits
branch
2 times, most recently
from
552b4f9
to
42975a5
Compare
This enables network traffic shaping on worker, making it possible to limit network bandwidth for each deal separately. Internally this is achieved using linux kernel traffic control mechanism and making friendship with Docker. At the first attempt we achieved this by using policing, which drops excess packets, throttling TCP window sizes and reducing the overall output rate of affected traffic streams. Overly aggressive burst sizes (which is tricky to set properly) led to excess packet drops and throttle the overall output rate, particularly with TCP-based flows. All above is relevant to the TBF (token bucket filtering) classless discipline, which is the easiest way to shape network traffic. An alternative approach is using the HTB (hierarchical token bucket) queueing discipline, which is classful and allows to build hierarchical rules for traffic shaping and policing. For egress traffic intermediate functional block (IFB) devices is used which has separate packet queueing. All this allows both to build hierarchical rules for each packet type, network device, etc; and moreover to restrict traffic for each container and/or for the entire worker. It is clear for now that both direction shaping works without packed dropping spikes. Epic win. After an ask plan is created the following actions are performed: - Create bridge device with its own iptables rules, where all containers for a specific deal will live. This bridge allows tasks to communicate with each-other, but not with tasks spawned within another deal. For that purposes you have to use overlay network drivers. - Apply TC rules to that bridge, that will limit ingress traffic. Yep, ingress, because container's egress traffic seems as an ingress traffic for the Docker device. - Create IFB virtual device. - Mirror egress traffic to that IFB by applying TC filters and create another rules for that traffic.
3Hren
force-pushed
the
feat/network-limits
branch
from
42975a5
to
649fd4e
Compare
nikonov1101
approved these changes
Aug 8, 2018
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This enables network traffic shaping on worker, making it possible to limit network bandwidth for each deal separately.
How it works
Internally this is achieved using linux kernel traffic control mechanism and making friendship with Docker.
At the first attempt we achieved this by using policing, which drops excess packets, throttling TCP window sizes and reducing the overall output rate of affected traffic streams. Overly aggressive burst sizes (which is tricky to set properly) led to excess packet drops and throttle the overall output rate, particularly with TCP-based flows.
All above is relevant to the TBF (token bucket filtering) classless discipline, which is the easiest way to shape network traffic.
An example of iperf3 with limiting ingress to 5Mbit/s and egress to 200Kbit/s using TBF qdisc.
As you see, shaping egress traffic is quite weird. Moreover it's impossible (or quite hard) to limit network for the entire worker or for each container separately.
All that forced us to investigate on classful disciplines.
An alternative approach is using the HTB (hierarchical token bucket) queueing discipline, which is classful and allows to build hierarchical rules for traffic shaping and policing. For egress traffic intermediate functional block (IFB) devices is used which has separate packet queueing.
All this allows both to build hierarchical rules for each packet type, network device, etc; and moreover to restrict traffic for each container and/or for the entire worker.
An example of iperf3 with limiting ingress to 10Mbit/s and egress to 5Mbit/s using HTB.
It is clear for now that both direction shaping works without packed dropping spikes. Epic win.
Workflow
After an ask plan is created the following actions are performed:
What is left to do
libnl3installing.NOTE: previous PR was imprudently merged with conflicts, so it was reverted, sorry.