Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf (throughput, latency) results compared other k8s networking solutions #102

Closed
murali-reddy opened this issue Aug 5, 2017 · 8 comments
Milestone

Comments

@murali-reddy
Copy link
Member

murali-reddy commented Aug 5, 2017

Its good to get comparative performance like in this Kubernetest Network Performance Testing document.

Throughput results for cluster IP/Node port service should be lot better with kube-rotuer due to use of IPVS. But instead of hand-waving, we need to get the real reproducable metrics to prove that.

using iperf to get the kube-router results should be stright forward.

@murali-reddy murali-reddy changed the title perf (throughput, latency) results compared other k8s solutions perf (throughput, latency) results compared other k8s networking solutions Aug 5, 2017
@murali-reddy murali-reddy added this to the v0.1.0 milestone Aug 5, 2017
@murali-reddy
Copy link
Member Author

murali-reddy commented Aug 7, 2017

Deployed 4 node cluster with kube-router on AWS with 2 nodes each in us-west-2a,us-west-2b zones with t2.medium instance

kops create cluster \
  --node-count 4 \
  --zones us-west-2a,us-west-2b \
  --master-zones us-west-2a \
  --dns-zone aws.cloudnativelabs.net \
  --node-size t2.medium \
  --master-size t2.medium \
  --networking kube-router \
  --yes \
  --name mycluster.aws.cloudnativelabs.net

For pod-to-pod connectivity of pods on same node below is iperf3 results

# iperf3 -c 100.96.2.6
Connecting to host 100.96.2.6, port 5201
[  4] local 100.96.2.7 port 59008 connected to 100.96.2.6 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.80 GBytes  32.6 Gbits/sec  329    571 KBytes
[  4]   1.00-2.00   sec  3.89 GBytes  33.4 Gbits/sec    0    658 KBytes
[  4]   2.00-3.00   sec  3.89 GBytes  33.4 Gbits/sec    1    669 KBytes
[  4]   3.00-4.00   sec  3.88 GBytes  33.3 Gbits/sec    0    670 KBytes
[  4]   4.00-5.00   sec  3.81 GBytes  32.7 Gbits/sec    0    716 KBytes
[  4]   5.00-6.00   sec  3.86 GBytes  33.1 Gbits/sec    0    782 KBytes
[  4]   6.00-7.00   sec  3.78 GBytes  32.5 Gbits/sec    0    872 KBytes
[  4]   7.00-8.00   sec  3.87 GBytes  33.2 Gbits/sec    0    872 KBytes
[  4]   8.00-9.00   sec  3.86 GBytes  33.1 Gbits/sec    0    887 KBytes
[  4]   9.00-10.00  sec  3.86 GBytes  33.1 Gbits/sec    0    964 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec  330             sender
[  4]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec                  receiver

iperf Done.

For pod-to-pod connectivity of pods across nodes in same zone below is iperf3 results

# iperf3 -c 100.96.2.6
Connecting to host 100.96.2.6, port 5201
[  4] local 100.96.1.4 port 44184 connected to 100.96.2.6 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   121 MBytes  1.02 Gbits/sec  547    141 KBytes
[  4]   1.00-2.00   sec   110 MBytes   923 Mbits/sec  720    120 KBytes
[  4]   2.00-3.00   sec   102 MBytes   860 Mbits/sec  490   89.1 KBytes
[  4]   3.00-4.00   sec   111 MBytes   933 Mbits/sec  633    123 KBytes
[  4]   4.00-5.00   sec   110 MBytes   923 Mbits/sec  597   87.7 KBytes
[  4]   5.00-6.00   sec   111 MBytes   933 Mbits/sec  559    102 KBytes
[  4]   6.00-7.00   sec   108 MBytes   902 Mbits/sec  659    140 KBytes
[  4]   7.00-8.00   sec  98.8 MBytes   828 Mbits/sec  426    117 KBytes
[  4]   8.00-9.00   sec   101 MBytes   849 Mbits/sec  584   93.3 KBytes
[  4]   9.00-10.00  sec   109 MBytes   912 Mbits/sec  658    122 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.06 GBytes   908 Mbits/sec  5873             sender
[  4]   0.00-10.00  sec  1.05 GBytes   900 Mbits/sec                  receiver

iperf Done.

For pod-to-pod connectivity of pods across nodes in different zones below is iperf3 results

# iperf3 -c 100.96.1.4
Connecting to host 100.96.1.4, port 5201
[  4] local 100.96.2.3 port 52850 connected to 100.96.1.4 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  97.5 MBytes   817 Mbits/sec  306    240 KBytes
[  4]   1.00-2.00   sec  87.5 MBytes   734 Mbits/sec    1    386 KBytes
[  4]   2.00-3.00   sec  95.0 MBytes   797 Mbits/sec    5    317 KBytes
[  4]   3.00-4.00   sec  96.2 MBytes   808 Mbits/sec    0    496 KBytes
[  4]   4.00-5.00   sec  96.2 MBytes   807 Mbits/sec    8    510 KBytes
[  4]   5.00-6.00   sec   101 MBytes   849 Mbits/sec    0    648 KBytes
[  4]   6.00-7.00   sec  97.5 MBytes   818 Mbits/sec    1    547 KBytes
[  4]   7.00-8.00   sec   100 MBytes   839 Mbits/sec    0    675 KBytes
[  4]   8.00-9.00   sec  96.2 MBytes   807 Mbits/sec    5    601 KBytes
[  4]   9.00-10.00  sec  97.5 MBytes   818 Mbits/sec    0    716 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   965 MBytes   809 Mbits/sec  326             sender
[  4]   0.00-10.00  sec   957 MBytes   802 Mbits/sec                  receiver

iperf Done.

@murali-reddy
Copy link
Member Author

murali-reddy commented Aug 7, 2017

Results with calico

kops create cluster \
  --node-count 4 \
  --zones us-west-2a,us-west-2b \
  --master-zones us-west-2a \
  --dns-zone aws.cloudnativelabs.net \
  --node-size t2.medium \
  --master-size t2.medium \
  --networking calico \
  --yes \
  --name mycluster.aws.cloudnativelabs.net

Across pods on same node

# iperf3 -c 100.65.175.196
Connecting to host 100.65.175.196, port 5201
[  4] local 100.65.175.197 port 58310 connected to 100.65.175.196 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.70 GBytes  31.8 Gbits/sec  282    660 KBytes
[  4]   1.00-2.00   sec  3.73 GBytes  32.1 Gbits/sec    0    701 KBytes
[  4]   2.00-3.00   sec  3.65 GBytes  31.3 Gbits/sec    0   1.26 MBytes
[  4]   3.00-4.00   sec  3.62 GBytes  31.1 Gbits/sec    0   1.27 MBytes
[  4]   4.00-5.00   sec  3.62 GBytes  31.1 Gbits/sec   31   1.38 MBytes
[  4]   5.00-6.00   sec  3.69 GBytes  31.7 Gbits/sec    0   1.38 MBytes
[  4]   6.00-7.00   sec  3.63 GBytes  31.2 Gbits/sec    0   1.46 MBytes
[  4]   7.00-8.00   sec  3.65 GBytes  31.3 Gbits/sec    0   1.46 MBytes
[  4]   8.00-9.00   sec  3.65 GBytes  31.3 Gbits/sec    0   1.53 MBytes
[  4]   9.00-10.00  sec  3.68 GBytes  31.6 Gbits/sec    0   1.53 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  36.6 GBytes  31.4 Gbits/sec  313             sender
[  4]   0.00-10.00  sec  36.6 GBytes  31.4 Gbits/sec                  receiver

iperf Done.

Across pods on different nodes in same zone

# iperf3 -c 100.65.175.196
Connecting to host 100.65.175.196, port 5201
[  4] local 100.78.251.193 port 40124 connected to 100.65.175.196 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   108 MBytes   901 Mbits/sec  146    248 KBytes
[  4]   1.00-2.00   sec  98.8 MBytes   829 Mbits/sec   42    451 KBytes
[  4]   2.00-3.00   sec  97.5 MBytes   818 Mbits/sec    0    590 KBytes
[  4]   3.00-4.00   sec  95.0 MBytes   797 Mbits/sec    0    699 KBytes
[  4]   4.00-5.00   sec  85.0 MBytes   713 Mbits/sec   14    584 KBytes
[  4]   5.00-6.00   sec  86.2 MBytes   724 Mbits/sec    0    682 KBytes
[  4]   6.00-7.00   sec  82.5 MBytes   692 Mbits/sec    0    767 KBytes
[  4]   7.00-8.00   sec  92.5 MBytes   776 Mbits/sec    0    851 KBytes
[  4]   8.00-9.00   sec  93.8 MBytes   786 Mbits/sec    0    930 KBytes
[  4]   9.00-10.00  sec  78.8 MBytes   661 Mbits/sec    0    991 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   918 MBytes   770 Mbits/sec  202             sender
[  4]   0.00-10.00  sec   909 MBytes   763 Mbits/sec                  receiver

iperf Done.

Across pods on different nodes in different zone

# iperf3 -c 100.65.175.196
Connecting to host 100.65.175.196, port 5201
[  4] local 100.124.122.65 port 59218 connected to 100.65.175.196 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  90.0 MBytes   755 Mbits/sec   84    291 KBytes
[  4]   1.00-2.00   sec  66.2 MBytes   556 Mbits/sec    0    428 KBytes
[  4]   2.00-3.00   sec  77.5 MBytes   650 Mbits/sec    1    541 KBytes
[  4]   3.00-4.00   sec  91.2 MBytes   765 Mbits/sec    1    649 KBytes
[  4]   4.00-5.00   sec  92.5 MBytes   776 Mbits/sec    0    748 KBytes
[  4]   5.00-6.00   sec  95.0 MBytes   797 Mbits/sec    2    592 KBytes
[  4]   6.00-7.00   sec  92.5 MBytes   776 Mbits/sec    0    698 KBytes
[  4]   7.00-8.00   sec  90.0 MBytes   755 Mbits/sec    0    786 KBytes
[  4]   8.00-9.00   sec  93.8 MBytes   787 Mbits/sec   14    622 KBytes
[  4]   9.00-10.00  sec   102 MBytes   860 Mbits/sec    0    733 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   891 MBytes   748 Mbits/sec  102             sender
[  4]   0.00-10.00  sec   882 MBytes   740 Mbits/sec                  receiver

iperf Done.
# exit

@murali-reddy
Copy link
Member Author

murali-reddy commented Aug 7, 2017

Results for Flannel

 kops create cluster \
  --node-count 4 \
  --zones us-west-2a,us-west-2b \
  --master-zones us-west-2a \
  --dns-zone aws.cloudnativelabs.net \
  --node-size t2.medium \
  --master-size t2.medium \
  --networking canal \
  --yes \
  --name mycluster.aws.cloudnativelabs.net

Pods on the same node

iperf3 -c 100.96.1.3
Connecting to host 100.96.1.3, port 5201
[  4] local 100.96.1.2 port 42664 connected to 100.96.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.65 GBytes  31.4 Gbits/sec  120    831 KBytes
[  4]   1.00-2.00   sec  3.64 GBytes  31.3 Gbits/sec    0    831 KBytes
[  4]   2.00-3.00   sec  3.66 GBytes  31.4 Gbits/sec    0    833 KBytes
[  4]   3.00-4.00   sec  3.65 GBytes  31.4 Gbits/sec    0    834 KBytes
[  4]   4.00-5.00   sec  3.64 GBytes  31.2 Gbits/sec   45    839 KBytes
[  4]   5.00-6.00   sec  3.57 GBytes  30.7 Gbits/sec    0    840 KBytes
[  4]   6.00-7.00   sec  3.58 GBytes  30.8 Gbits/sec    1    880 KBytes
[  4]   7.00-8.00   sec  3.64 GBytes  31.2 Gbits/sec    0    882 KBytes
[  4]   8.00-9.00   sec  3.61 GBytes  31.0 Gbits/sec    0    884 KBytes
[  4]   9.00-10.00  sec  3.62 GBytes  31.1 Gbits/sec    0    884 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  36.3 GBytes  31.1 Gbits/sec  166             sender
[  4]   0.00-10.00  sec  36.3 GBytes  31.1 Gbits/sec                  receiver

iperf Done.

Pods acress the nodes in same AZ

 iperf3 -c 100.96.1.3
Connecting to host 100.96.1.3, port 5201
[  4] local 100.96.2.2 port 56270 connected to 100.96.1.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   120 MBytes  1.01 Gbits/sec   51    229 KBytes
[  4]   1.00-2.00   sec   111 MBytes   933 Mbits/sec  102    211 KBytes
[  4]   2.00-3.00   sec   101 MBytes   849 Mbits/sec   62    212 KBytes
[  4]   3.00-4.00   sec   109 MBytes   912 Mbits/sec   24    198 KBytes
[  4]   4.00-5.00   sec   110 MBytes   923 Mbits/sec   33    236 KBytes
[  4]   5.00-6.00   sec   110 MBytes   923 Mbits/sec   57    206 KBytes
[  4]   6.00-7.00   sec   111 MBytes   933 Mbits/sec   25    164 KBytes
[  4]   7.00-8.00   sec   110 MBytes   923 Mbits/sec   25    198 KBytes
[  4]   8.00-9.00   sec  95.0 MBytes   797 Mbits/sec   18    256 KBytes
[  4]   9.00-10.00  sec  96.2 MBytes   807 Mbits/sec   75    250 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.05 GBytes   901 Mbits/sec  472             sender
[  4]   0.00-10.00  sec  1.04 GBytes   893 Mbits/sec                  receiver

iperf Done.

Pods acress the nodes in different AZ

# iperf3 -c 100.96.1.2
Connecting to host 100.96.1.2, port 5201
[  4] local 100.96.2.5 port 50822 connected to 100.96.1.2 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  66.2 MBytes   556 Mbits/sec   33    638 KBytes
[  4]   1.00-2.00   sec  47.5 MBytes   398 Mbits/sec    0    708 KBytes
[  4]   2.00-3.00   sec  57.5 MBytes   482 Mbits/sec   10    549 KBytes
[  4]   3.00-4.00   sec  57.5 MBytes   482 Mbits/sec    0    624 KBytes
[  4]   4.00-5.00   sec  58.8 MBytes   493 Mbits/sec    6    529 KBytes
[  4]   5.00-6.00   sec  58.8 MBytes   493 Mbits/sec    0    608 KBytes
[  4]   6.00-7.00   sec  63.8 MBytes   535 Mbits/sec    3    519 KBytes
[  4]   7.00-8.00   sec  66.2 MBytes   556 Mbits/sec    0    611 KBytes
[  4]   8.00-9.00   sec  65.0 MBytes   545 Mbits/sec    0    689 KBytes
[  4]   9.00-10.00  sec  66.2 MBytes   556 Mbits/sec    0    758 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec   608 MBytes   510 Mbits/sec   52             sender
[  4]   0.00-10.00  sec   598 MBytes   502 Mbits/sec                  receiver

iperf Done.

@murali-reddy
Copy link
Member Author

murali-reddy commented Aug 7, 2017

Results for Weave

 kops create cluster \
  --node-count 4 \
  --zones us-west-2a,us-west-2b \
  --master-zones us-west-2a \
  --dns-zone aws.cloudnativelabs.net \
  --node-size t2.medium \
  --master-size t2.medium \
  --networking weave \
  --yes \
  --name mycluster.aws.cloudnativelabs.net

Pods on the same node

# iperf3 -c 100.120.0.1
Connecting to host 100.120.0.1, port 5201
[  4] local 100.120.0.2 port 51958 connected to 100.120.0.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  3.86 GBytes  33.1 Gbits/sec  142   1.59 MBytes
[  4]   1.00-2.00   sec  3.85 GBytes  33.0 Gbits/sec   10   1.75 MBytes
[  4]   2.00-3.00   sec  3.84 GBytes  33.0 Gbits/sec    0   1.89 MBytes
[  4]   3.00-4.00   sec  3.91 GBytes  33.6 Gbits/sec   28   2.32 MBytes
[  4]   4.00-5.00   sec  3.88 GBytes  33.4 Gbits/sec    0   2.47 MBytes
[  4]   5.00-6.00   sec  3.86 GBytes  33.1 Gbits/sec    0   2.47 MBytes
[  4]   6.00-7.00   sec  3.85 GBytes  33.1 Gbits/sec    0   2.54 MBytes
[  4]   7.00-8.00   sec  3.87 GBytes  33.3 Gbits/sec    0   2.63 MBytes
[  4]   8.00-9.00   sec  3.80 GBytes  32.6 Gbits/sec    2   3.08 MBytes
[  4]   9.00-10.00  sec  3.82 GBytes  32.8 Gbits/sec   55   3.14 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec  237             sender
[  4]   0.00-10.00  sec  38.5 GBytes  33.1 Gbits/sec                  receiver

iperf Done.

Pods across the nodes in same AZ

#  iperf3 -c 100.104.0.3
Connecting to host 100.104.0.3, port 5201
[  4] local 100.120.0.1 port 40844 connected to 100.104.0.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   125 MBytes  1.05 Gbits/sec   17    459 KBytes
[  4]   1.00-2.00   sec   116 MBytes   975 Mbits/sec    9    519 KBytes
[  4]   2.00-3.00   sec   118 MBytes   985 Mbits/sec   14    554 KBytes
[  4]   3.00-4.00   sec   118 MBytes   986 Mbits/sec   14    597 KBytes
[  4]   4.00-5.00   sec   118 MBytes   986 Mbits/sec   15    450 KBytes
[  4]   5.00-6.00   sec   119 MBytes   996 Mbits/sec   11    502 KBytes
[  4]   6.00-7.00   sec   118 MBytes   986 Mbits/sec   10    571 KBytes
[  4]   7.00-8.00   sec   118 MBytes   986 Mbits/sec   15    450 KBytes
[  4]   8.00-9.00   sec   116 MBytes   975 Mbits/sec   12    485 KBytes
[  4]   9.00-10.00  sec   118 MBytes   986 Mbits/sec   12    588 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.15 GBytes   991 Mbits/sec  129             sender
[  4]   0.00-10.00  sec  1.14 GBytes   983 Mbits/sec                  receiver

iperf Done.

Pods across the nodes in different AZ

# iperf3 -c 100.104.0.3
Connecting to host 100.104.0.3, port 5201
[  4] local 100.116.0.1 port 44394 connected to 100.104.0.3 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   129 MBytes  1.08 Gbits/sec    9    485 KBytes
[  4]   1.00-2.00   sec   119 MBytes   996 Mbits/sec    7    597 KBytes
[  4]   2.00-3.00   sec   119 MBytes   996 Mbits/sec    9    502 KBytes
[  4]   3.00-4.00   sec   119 MBytes   996 Mbits/sec   12    588 KBytes
[  4]   4.00-5.00   sec   119 MBytes   996 Mbits/sec    9    485 KBytes
[  4]   5.00-6.00   sec   116 MBytes   975 Mbits/sec   10    554 KBytes
[  4]   6.00-7.00   sec   119 MBytes   996 Mbits/sec    9    597 KBytes
[  4]   7.00-8.00   sec   118 MBytes   986 Mbits/sec   11    485 KBytes
[  4]   8.00-9.00   sec   118 MBytes   986 Mbits/sec    7    675 KBytes
[  4]   9.00-10.00  sec   118 MBytes   986 Mbits/sec    8    493 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.16 GBytes   999 Mbits/sec   91             sender
[  4]   0.00-10.00  sec  1.15 GBytes   992 Mbits/sec                  receiver

iperf Done.

@murali-reddy murali-reddy mentioned this issue Sep 7, 2017
8 tasks
@vicaya
Copy link

vicaya commented Jan 8, 2018

@murali-reddy have you guys looked into why weave, which uses vxlan tunneling (fast datapath is still vxlan in kernel with ovs) is significantly faster and more stable (esp. for cross AZ traffic) than kube-router and calico, which should always use in kernel L3 datapaths. I expected that kube-router and calico should always be faster than vxlan based solutions...

I'm also more interested in perf results with NetworkPolicy turned on with DefaultDeny.

@jpds
Copy link
Contributor

jpds commented Feb 13, 2018

I would be curious to see what the results are with:

a) jumbo frames
b) one of the 25G instance types from https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html

@dimm0
Copy link
Contributor

dimm0 commented Feb 14, 2018

We already have a cluster setup with 10, 40 and 100G nodes and PerfSonar measurements between all hosts, and so far the results are not great because of lack of jumbo frames support. Once it's done, I can provide some info

@aauren
Copy link
Collaborator

aauren commented May 17, 2021

Closing this as old.

@aauren aauren closed this as completed May 17, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

5 participants