Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Newer
Older
100644 166 lines (111 sloc) 11.543 kb
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
1 = Preliminary benchmark for ELB =
2
3 == WHY ==
e808a82 @sangmin elb-benchmark
sangmin authored
4
5 There are two major design decisions for the upcoming ELB implementation. The first is whether to run the load-balancers on the native OS or inside the VMs that Eucalyptus manages. The second is the choice of the underlying SW load-balancer. We’re considering either HAProxy or Nginix, based on the feature set they cover. The goal of this benchmark is to compare the possible options from performance perspectives.
6
7
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
8 == HOW ==
9 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki1-benchmark-overview.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
10 The picture above illustrates the benchmark’s setup. Across the tests, the client and backend web servers remain the same. This benchmark result covers the HTTP, HTTPS, as well as arbitrary TCP listeners.
11
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
12 == RESULT-HTTP ==
e808a82 @sangmin elb-benchmark
sangmin authored
13 Because everyone’s time is precious, we will begin with the conclusion:
14 * HAproxy outperforms Nginx significantly (especially for smaller messages: 67% better throughput for 1KB download)
15 * There is some performance loss due to VMs (23% loss for small objects with HAproxy)
16
17 The following graphs illustrate the result in detail. During the test, the client downloads objects from web servers in varying size (1K, 10K, 100K files). The first set of graphs shows the number of requests a load-balancer process per second. The four bars represent the four cases: Haproxy on VMs, Nginx on VMs, Haproxy on native host, Nginx on native host. The graphs represent the conclusions above. The Haproxy outperforms Nginx on both VMs and native host. The difference becomes less apparent when the object size is bigger, and almost negligible for 100KB objects.
18
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
19 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki2-http-throughput.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
20
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
21 <p align=center> Figure 1: Haproxy and Nginx, on virtual machines and native OS </p>
e808a82 @sangmin elb-benchmark
sangmin authored
22
23 The next set of graphs show the distribution of response times for each request. The x-axis represents the percent of requests completed within the time in y-axis. For instance, when downloading 1KB object, 95 % of requests completed within 8 ms, for both Nginx and Haproxy. The graphs indicate that both the Haproxy and Nginx show stable response time (not too many spikes) across many requests. In general, Haproxy shows better distribution of the response time than Nginx. For example, 99% of the requests finish within 11ms for Haproxy, compared to 35ms of the Nginx during the 10K file download. For Nginx, the distribution becomes worse when object is bigger (i.e., small number of requests downloading large object will experience very large delay).
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
24 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki3-http-distribution.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
25
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
26 <p align=center> Figure 2: distribution of HTTP response time </p>
e808a82 @sangmin elb-benchmark
sangmin authored
27
28 Finally, the next three graphs show the result when keep-alive is turned on for both load-balancers. Note that in this setup, the client requests keep-alive to the load-balancer (keep-alive is optional for http clients), which will keep the socket open during the entire test run (50K-500K download). It is known that AWS has keep-alive turned on their ELB. For brevity, we ran the tests only on VMs, comparing the Haproxy and the Nginx. The result shows that keep-alive on the LB increases the throughput by about 15 %. The Haproxy performs better than Nginx, by roughly 20 %, when the object is small (1KB). Their difference becomes negligible for bigger objects (10K and 100K) and actually Nginx for the first time perform better for 10KB object than Haproxy.
29
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
30 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki4-http-ka-throughput.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
31
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
32 <p align=center> Figure 3: Haproxy and Nginx with Keep-alive </p>
e808a82 @sangmin elb-benchmark
sangmin authored
33
34
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
35 == RESULT - HTTPS ==
36 <b> Conclusion: Haproxy performs slightly better than Nginx. </b>
e808a82 @sangmin elb-benchmark
sangmin authored
37
38 The following benchmark tests SSL termination on load-balancers. In this test, while the client communicates with the load-balancers over SSL, the communication between load-balancers and the back-end web servers are over HTTP (so we terminate SSL at the load-balancer). The parameters remain the same as the previous HTTP benchmark. We used AES256-SHA as the cipher algorithm. Both HAProxy and Nginx are run inside the VMs in this test (we do not test HTTPS on native host).
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
39 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki5-https-overview.png?raw=true">
40 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki6-http-vs-https.png?raw=true" style="display: block; margin: 0 auto;">
e808a82 @sangmin elb-benchmark
sangmin authored
41
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
42 <p align=center> Figure 4:HTTP vs. HTTPS (HAproxy) </p>
e808a82 @sangmin elb-benchmark
sangmin authored
43
44 The figure 4 illustrates the difference in the throughput of HTTP and HTTPS. In the figure we compare the number of requests processed per second for HAProxy (similar results were obtained for Nginx). We can see roughly 90% less throughput with HTTPS for small objects.
45
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
46 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki7-https-throughput.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
47
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
48 <p align=center> Figure 5: HTTPS throughput: HAProxy vs. Nginx </p>
e808a82 @sangmin elb-benchmark
sangmin authored
49
50 The figure 5 compares the HTTPS throughputs between HAProxy and Nginx. In all three object sizes, we can see HAProxy slightly outperforms Nginx. The biggest difference is when the object size is smallest (1KB): HAProxy achieves 8.6% better throughput than Nginx.
51
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
52 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki8-https-distribution.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
53
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
54 <p align=center> Figure 6: HTTPS response time distribution: Haproxy vs. Nginx </p>
e808a82 @sangmin elb-benchmark
sangmin authored
55
56 Figure 6 presents the distribution of the HTTPS response time, comparing Haproxy and Nginx. The x-axis represents the percent of requests completed within the time in y-axis. In all three graphs, we can see that both Nginx and Haproxy exhibit fairly stable distribution. 95% of requests complete within the reasonable time for all object sizes. However, we can see one issue with Nginx. For the request that’s in the highest 5% of response time, the Nginx shows significantly worse response time than Haproxy. The difference is most apparent when the object size is bigger (100KB). This means that with Nginx there will be small number of requests that show much higher spikes than Haproxy.
57
58
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
59 == RESULT – TCP ==
e808a82 @sangmin elb-benchmark
sangmin authored
60 To benchmark the TCP load-balancing, we’ve come up with two test configurations.
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
61 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki9-tcp-overview.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
62
63 In both configurations, the Haproxy and Nginx are setup as TCP load-balancers, communicating to the backend web servers using TCP. The client takes the data as HTTP or HTTPS depending on the protocol on the backend web servers (so the load-balancers are dumb TCP tunnel).
64 Figure 7 represent the results. In the figure, we compare HAproxy and Nginx, both running on VMs, with the throughput from direct connection to one of the web server (no LB involved). Note that there are two web servers behind the LB, so the comparison with the direct connection would give us insight on how well the LB would perform as SSL pass-through balancers to backend HTTPS servers. There are two findings in the graph:
65 * Haproxy and Nginx are virtually tie. This an expected result, as only with two web servers running HTTPS, we can’t saturate the throughput capacity of both LBs.
66 * For smaller objects, we can expect that SSL pass-through balancing would be horizontally scalable. However, for bigger objects (when workload is more data-bound) it appears that simply adding more HTTPS backend would not result in better throughput.
67
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
68 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki10-tcp-https-throughput.png?raw=true">
e808a82 @sangmin elb-benchmark
sangmin authored
69
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
70 <p align=center> Figure 7: HTTPS through TCP load-balancer </p>
e808a82 @sangmin elb-benchmark
sangmin authored
71
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
72 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki11-tcp-http-throughput.png?raw=true">
73 <p align=center> Figure 8: HTTP through TCP load-balancer </p>
e808a82 @sangmin elb-benchmark
sangmin authored
74
75 Finally figure 8 illustrate the results when load-balancers are for pass-through HTTP traffic. In this setup, we can assume that load-balancers are saturated and thus we can compare the maximum attainable TCP throughput for Haproxy and Nginx. In the figure we can see Haproxy and Nginx are roughly tie in the TCP load-balancing. For 1K objects, Haproxy slightly outperforms Nginx, while 10K result is the opposite.
76
381887d @sangmin Proper markup on the elb-benchmark wiki
sangmin authored
77 == SUMMARY ==
e808a82 @sangmin elb-benchmark
sangmin authored
78
79 We summarize the result of the benchmark as follows:
80 * There are some performance losses due to LBs running inside the VMs (as much as 23%). We can anticipate that tuning the VM parameters and the hypervisors could lessen the loss.
81 * For HTTP and HTTPS, HAproxy apparently outperforms Nginx (by as much as 67%).
82 * For TCP, HAproxy and Nginx show about the same performance.
83 * Through the benchmark, we found that both Haproxy and Nginx can cover the feature set mandated by ELB specification (HTTP, HTTP-KA, HTTPS, TCP, Reconfigurability, etc).
51919bb @sangmin Results with latest nginx (1.2.6)
sangmin authored
84
85
86 == UPDATE (Latest Nginx) ==
87 After the benchmark results were published, one of the Nginx developers reached out to us and discussed the possible improvement. Per his suggestion, we've re-run the benchmark with the latest Nginx stable release (1.2.6) (the previous results were based on nginx on Centos/Rhel 6).
88 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki12-http-nginx1_2_6.png?raw=true">
89 <p align=center> Figure 9: HTTP throughput with Nginx 1.2.6 </p>
90
91 <img src="https://github.com/sangmin/architecture/blob/master/features/elb/3.3/elb-benchmark/wiki13-https-nginx1_2_6.png?raw=true">
92 <p align=center> Figure 10: HTTPS throughput with Nginx 1.2.6 </p>
93
94 === NOTES ===
95 * We ran the benchmark with the latest nginx on Eucalyptus VMs
96 * The graphs indicate that there's notable improvements in http throughput with latest nginx. However still it falls short of the haproxy throughput.
97 * We could not find notable difference in https throughput with the latest nginx.
98
99 The benchmarks were run on the following environment:
100 * The VMs were run on Eucalyptus 3.2 (KVM on Centos 6)
101 * Each VM had 2 cores
102 * We did not tweak either nginx or haproxy configurations in a hope to improve their performances. We applied the most simplistic configuration that will achieve functional correctness. Therefore we do not claim that the results represent the *best* possible performances of either nginx or haproxy.
103
104 Followings are the Haproxy and nginx configuration throughout the benchmark.
105
106 ==== HAPROXY ====
107
108 global
109 maxconn 100000
110 ulimit-n 655360
111
112 listen stats :81
113 mode http
114 stats enable
115 stats hide-version
116 stats realm Haproxy\ Statistics
117 stats uri /
118
119 defaults
120 mode http
121 log 127.0.0.1 local0
122 contimeout 1000
123 clitimeout 10000
124 srvtimeout 10000
125 option tcplog
126 option http-server-close # affects KA on/off
127
128 frontend all
129 bind 0.0.0.0:80
130 default_backend webcluster
131
132 backend webcluster
133 server web1 172.20.110.23:80
134 server web2 172.20.110.28:80
135 balance roundrobin
136
137 ==== Nginx ====
138
139 http {
140 # [...]
141
142 upstream webcluster {
143 server 172.20.110.23:80;
144 server 172.20.110.28:80;
145 }
146
147 # [...]
148
149 server {
150 listen 80;
151 server_name frontend;
152
153 keepalive_requests 500000;
154 keepalive_timeout 1000;
155
156 location / {
157 proxy_pass http://webcluster;
158 }
159 }
160
161 access_log off;
162 # [...]
163 }
9f98165 @sangmin added tagging to elb-benchmark wiki.
sangmin authored
164
9cce72c @sangmin wrong tag..fixed
sangmin authored
165 [[tag:rls-3.3]]
Something went wrong with that request. Please try again.