Skip to content
Newer
Older
100644 192 lines (172 sloc) 6.99 KB
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 13, 2012
1 Benchmark Details:
2
3 * [Amazon EC2 m1.large](http://aws.amazon.com/ec2/instance-types/): 7.5GB memory, 2 virtual cores
4 * 64-bit Ubuntu 11.10
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
5 * Darner 0.1.3, compiled Boost 1.46 and leveldb 1.2.0
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 12, 2012
6 * Kestrel 2.2.0 with OpenJDK 1.6
7
8 # Resident Memory
9
450bf65 @erikfrey use same axis format on all charts, ditch log scale
erikfrey authored Aug 13, 2012
10 How much memory does the queue server use? We are testing both steady-state memory resident, and also how aggressively
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 12, 2012
11 the server acquires and releases memory as queues expand and contract. We tuned Kestrel's JVM down to the smallest
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
12 heap that didn't cause OOM's and didn't impact performance: `-Xmx512m`.
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 12, 2012
13
dfec91b @erikfrey fix github url change for raw files
erikfrey authored Feb 19, 2013
14 ![Resident Memory Benchmark](https://raw.github.com/wavii/darner/master/docs/images/bench_memory_resident.png)
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 12, 2012
15
16 ```
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
17 ubuntu@ip-10-6-51-186:~/darner$ bench/mem_rss.sh
18 kestrel 0 requests: 74476 kB
19 kestrel 1024 requests: 84256 kB
20 kestrel 2048 requests: 109508 kB
21 kestrel 4096 requests: 133492 kB
22 kestrel 8192 requests: 160404 kB
23 kestrel 16384 requests: 182460 kB
24 kestrel 32768 requests: 278340 kB
25 kestrel 65536 requests: 330300 kB
26 kestrel 131072 requests: 397852 kB
27 kestrel 262024 requests: 465148 kB
28 kestrel 524048 requests: 520476 kB
29 kestrel 1048576 requests: 611612 kB
30 darner 0 requests: 2220 kB
31 darner 1024 requests: 3492 kB
32 darner 2048 requests: 5872 kB
33 darner 4096 requests: 8136 kB
34 darner 8192 requests: 15520 kB
35 darner 16384 requests: 25656 kB
36 darner 32768 requests: 27412 kB
37 darner 65536 requests: 24324 kB
38 darner 131072 requests: 28440 kB
39 darner 262024 requests: 28524 kB
40 darner 524048 requests: 32104 kB
41 kestrel 1048576 requests: 33848 kB
3f088bb @erikfrey memory resident benchmark and results
erikfrey authored Aug 12, 2012
42 ```
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
43
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
44 # Queue Flooding
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
45
46 How quickly can we flood items through an empty queue? This tests the raw throughput of the server. We also include
a974d6b @erikfrey this is no longer true
erikfrey authored Aug 26, 2012
47 memcache as an upper bound - a throughput at which we are likely saturating on `send/recv` syscalls.
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
48
dfec91b @erikfrey fix github url change for raw files
erikfrey authored Feb 19, 2013
49 ![Queue Flood Benchmark](https://raw.github.com/wavii/darner/master/docs/images/bench_queue_flood.png)
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
50
51 ```
7280984 @erikfrey to be fair, rerun benchmarks to confirm lower chunk_size doesn't impa…
erikfrey authored Aug 12, 2012
52 ubuntu@domU-12-31-39-0E-0C-72:~/darner$ bench/flood.sh
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
53 warming up kestrel...done.
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
54 kestrel 1 conns: 7163.58 #/sec (mean)
55 kestrel 2 conns: 8802.04 #/sec (mean)
56 kestrel 5 conns: 9742.79 #/sec (mean)
57 kestrel 10 conns: 11200.7 #/sec (mean)
58 kestrel 50 conns: 12038.8 #/sec (mean)
59 kestrel 100 conns: 11705.5 #/sec (mean)
60 kestrel 200 conns: 11700 #/sec (mean)
61 kestrel 300 conns: 11562.7 #/sec (mean)
62 kestrel 400 conns: 11596.9 #/sec (mean)
63 kestrel 600 conns: 11357.2 #/sec (mean)
64 kestrel 800 conns: 11147 #/sec (mean)
65 kestrel 1000 conns: 11218.9 #/sec (mean)
66 kestrel 2000 conns: 11101.9 #/sec (mean)
67 kestrel 4000 conns: 10879 #/sec (mean)
68 kestrel 6000 conns: 10639.4 #/sec (mean)
69 kestrel 8000 conns: 10618 #/sec (mean)
70 kestrel 10000 conns: 10486.6 #/sec (mean)
71 darner 1 conns: 13088.1 #/sec (mean)
72 darner 2 conns: 30102.3 #/sec (mean)
73 darner 5 conns: 35279.6 #/sec (mean)
74 darner 10 conns: 36549.7 #/sec (mean)
75 darner 50 conns: 36846 #/sec (mean)
76 darner 100 conns: 36199.1 #/sec (mean)
77 darner 200 conns: 35906.6 #/sec (mean)
78 darner 300 conns: 35893.8 #/sec (mean)
79 darner 400 conns: 36081.5 #/sec (mean)
80 darner 600 conns: 36616.6 #/sec (mean)
81 darner 800 conns: 34910.1 #/sec (mean)
82 darner 1000 conns: 34668.1 #/sec (mean)
83 darner 2000 conns: 28169 #/sec (mean)
84 darner 4000 conns: 32792.3 #/sec (mean)
85 darner 6000 conns: 31680.7 #/sec (mean)
86 darner 8000 conns: 30726.7 #/sec (mean)
87 darner 10000 conns: 30792.9 #/sec (mean)
88 memcache 1 conns: 15227.7 #/sec (mean)
89 memcache 2 conns: 29133.3 #/sec (mean)
90 memcache 5 conns: 35155.6 #/sec (mean)
91 memcache 10 conns: 46414.5 #/sec (mean)
92 memcache 50 conns: 53347.6 #/sec (mean)
93 memcache 100 conns: 55294.4 #/sec (mean)
94 memcache 200 conns: 53447.4 #/sec (mean)
95 memcache 300 conns: 53864.8 #/sec (mean)
96 memcache 400 conns: 52854.1 #/sec (mean)
97 memcache 600 conns: 52700.9 #/sec (mean)
98 memcache 800 conns: 51546.4 #/sec (mean)
99 memcache 1000 conns: 52438.4 #/sec (mean)
100 memcache 2000 conns: 38255.5 #/sec (mean)
101 memcache 4000 conns: 41442.2 #/sec (mean)
102 memcache 6000 conns: 43224.6 #/sec (mean)
103 memcache 8000 conns: 42844.9 #/sec (mean)
104 memcache 10000 conns: 41347.9 #/sec (mean)
a45702c @erikfrey first benchmark results
erikfrey authored Aug 12, 2012
105 ```
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
106
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
107 # Fairness
108
109 How does the queue server deal with messages of varying sizes? This benchmark pushes large messages through the queue
110 in order to see whether they cause latency spikes for smaller requests, which can lead to timeouts. For the graph
111 below, a flatter curve means each request is more fairly served in time.
112
dfec91b @erikfrey fix github url change for raw files
erikfrey authored Feb 19, 2013
113 ![Fairness Benchmark](https://raw.github.com/wavii/darner/master/docs/images/bench_fairness.png)
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
114
115 ```
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
116 ubuntu@ip-10-6-51-186:~/darner$ bench/fairness.sh
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
117 warming up kestrel...done.
118 kestrel stats:
119 Concurrency Level: 10
120 Gets: 0
121 Sets: 100000
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
122 Time taken for tests: 208.706 seconds
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
123 Bytes read: 800000 bytes
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
124 Read rate: 3.7433 Kbytes/sec
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
125 Bytes written: 8700000 bytes
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
126 Write rate: 40.7084 Kbytes/sec
127 Requests per second: 479.143 #/sec (mean)
128 Time per request: 20868.3 us (mean)
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
129
130 Percentage of the requests served within a certain time (us)
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
131 50%: 827
132 66%: 1434
133 75%: 1984
134 80%: 2528
135 90%: 8809
136 95%: 45956
137 98%: 163431
138 99%: 341297
139 100%: 3557554 (longest request)
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
140 darner stats:
141 Concurrency Level: 10
142 Gets: 0
143 Sets: 100000
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
144 Time taken for tests: 26.26 seconds
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
145 Bytes read: 800000 bytes
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
146 Read rate: 29.7506 Kbytes/sec
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
147 Bytes written: 8700000 bytes
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
148 Write rate: 323.537 Kbytes/sec
149 Requests per second: 3808.07 #/sec (mean)
150 Time per request: 2624.35 us (mean)
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
151
152 Percentage of the requests served within a certain time (us)
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
153 50%: 729
154 66%: 767
155 75%: 817
156 80%: 885
157 90%: 1196
158 95%: 3507
159 98%: 43966
160 99%: 44476
161 100%: 94989 (longest request)
162 ```
6309306 @erikfrey fairness benchmark and tests
erikfrey authored Aug 12, 2012
163
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
164 # Queue Packing
165
166 This tests the queue server's behavior with a backlog of items. The challenge for the queue server is to serve items
167 that no longer all fit in memory. Absolute throughput isn't important here - item sizes are large to quickly saturate
168 free memory. Instead it's important for the throughput to flatten out as the backlog grows.
169
dfec91b @erikfrey fix github url change for raw files
erikfrey authored Feb 19, 2013
170 ![Queue Packing Benchmark](https://raw.github.com/wavii/darner/master/docs/images/bench_queue_packing.png)
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
171
172 ```
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
173 ubuntu@ip-10-6-51-186:~/darner$ bench/packing.sh
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
174 warming up kestrel...done.
7e5f5bd @erikfrey benchmark updates
erikfrey authored Aug 26, 2012
175 kestrel 0 sets: 10350.9 #/sec (mean)
176 kestrel 4096 sets: 10137.9 #/sec (mean)
177 kestrel 16384 sets: 10016.5 #/sec (mean)
178 kestrel 65536 sets: 10073 #/sec (mean)
179 kestrel 262144 sets: 9243 #/sec (mean)
180 kestrel 1048576 sets: 9220.41 #/sec (mean)
181 kestrel 4194304 sets: 9000.9 #/sec (mean)
182 kestrel 16777216 sets: 7990.73 #/sec (mean)
183 darner 0 sets: 25723.5 #/sec (mean)
184 darner 4096 sets: 21853.1 #/sec (mean)
185 darner 16384 sets: 17792 #/sec (mean)
186 darner 65536 sets: 13606.4 #/sec (mean)
187 darner 262144 sets: 13798.8 #/sec (mean)
188 darner 1048576 sets: 14479.1 #/sec (mean)
189 darner 4194304 sets: 13535.5 #/sec (mean)
190 darner 16777216 sets: 12786.9 #/sec (mean)
82f11e1 @erikfrey packing benchmark and results
erikfrey authored Aug 12, 2012
191 ```
Something went wrong with that request. Please try again.