/
index.html
707 lines (512 loc) · 30.4 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Scaling Considerations</title>
<meta name="description" content="Scaling Apache Flagon: An Introduction and First Principles">
<link rel="stylesheet" type="text/css" href="/semantic/dist/semantic.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1.1/jquery.min.js"></script>
<script src="/semantic/dist/semantic.min.js"></script>
<script src="https://d3js.org/d3.v4.min.js"></script>
<script src="/js/main.js"></script>
<link rel="stylesheet" href="/assets/main.css">
<link rel="canonical" href="http://flagon.incubator.apache.org/docs/stack/scaling/">
<link rel="alternate" type="application/rss+xml" title="Apache Flagon" href="/feed.xml">
</head>
<body>
<div class="site-header">
<div id="main-menu" class="ui inverted padded fixed top borderless menu">
<a class="site-title" href="/">
<h3 class="ui inverted header item">Apache Flagon</h3>
</a>
<div class="right menu">
<div class="ui dropdown item">
Products
<div class="menu">
<a class="item" href="/stack">Stack</a>
<a class="item" href="/userale">UserALE</a>
<a class="item" href="/distill">Distill</a>
<a class="item" href="/tap">Tap</a>
<a class="item" href="/stout">Stout</a>
</div>
</div>
<div class="ui dropdown item">
<a href="/docs">Docs</a>
<div class="menu">
<a class="item" href="/docs/stack">Stack</a>
<a class="item" href="/docs/userale">UserALE</a>
<a class="item" href="/docs/useralejs">UserALE.js</a>
<a class="item" href="/docs/useralepy">UserALE.py</a>
<a class="item" href="/docs/distill">Distill</a>
<a class="item" href="/docs/tap">Tap</a>
<a class="item" href="/docs/stout">Stout</a>
</div>
</div>
<a class="item" href="/releases">Releases</a>
<a class="item" href="/community">Community</a>
</div>
</div>
</div>
<div class="main-wrapper">
<div class="docs-wrapper">
<div class="docs-sidebar">
<!-- <div class="ui inverted list">
<a class="item" href="/docs">About the Docs</a>
</div> -->
<div id="docs-nav" class="ui inverted accordion">
<div class="title">
<i class="dropdown icon"></i>
Apache Flagon
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/">About the Docs</a>
<a class="item" href="/docs/contributing/">Contributing</a>
</div>
</div>
<div class="active title">
<i class="dropdown icon"></i>
Stack
</div>
<div class="active content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/stack/">Getting Started</a>
<a class="item" href="/docs/stack/scaling/">Scaling Considerations</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/userale/">About UserALE</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE.js
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/useralejs/">Getting Started</a>
<a class="item" href="/docs/useralejs/build/">UserALE.js Build and Test</a>
<a class="item" href="/docs/useralejs/API/">The UserALE.js API</a>
<a class="item" href="/docs/useralejs/modifying/">Modifying UserALE.js Source Code</a>
<a class="item" href="/docs/useralejs/dataschema/">UserALE.js' Data Schema</a>
<a class="item" href="/docs/useralejs/troubleshooting/">Troubleshooting</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
UserALE.py
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/useralepy/">Installation</a>
<a class="item" href="/docs/useralepy/start.html">Getting Started</a>
<a class="item" href="/docs/useralepy/api.html">API Documentation</a>
<a class="item" href="/docs/useralepy/changelog.html">Changelog</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Distill
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/distill/">Installation</a>
<a class="item" href="/docs/distill/start.html">Getting Started</a>
<a class="item" href="/docs/distill/api.html">API Documentation</a>
<a class="item" href="/docs/distill/analytics.html">Apache Distill Analytics</a>
<a class="item" href="/docs/distill/http.html">Apache Distill HTTP Client</a>
<a class="item" href="/docs/distill/models.html">Apache Distill Models</a>
<a class="item" href="/docs/distill/utilities.html">Apache Distill Utilities</a>
<a class="item" href="/docs/distill/changelog.html">Changelog</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Tap
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/tap/">Getting Started</a>
</div>
</div>
<div class="title">
<i class="dropdown icon"></i>
Stout
</div>
<div class="content">
<div class="ui inverted bulleted list docs-list">
<a class="item" href="/docs/stout/">About Stout</a>
<a class="item" href="/docs/stout/start.html">Getting Started</a>
</div>
</div>
</div>
</div>
<div class="docs-content">
<h2 class="ui header">Scaling Considerations</h2>
<h3 id="scaling-apache-flagon-an-introduction-and-first-principles">Scaling Apache Flagon: An Introduction and First Principles</h3>
<p>This guide touches on some basic principles to keep in mind as you’re planning for scale with <a href="https://github.com/apache/incubator-flagon">Apache Flagon</a>.</p>
<p>We provide high-level guidance and considerations for working with an Elastic stack to scale <a href="/docs/useralejs">Apache UserALE.js</a> services.</p>
<p>This guide also provides an overview of benchmarking tools and methodologies for accurately gauging your scaling needs.</p>
<p><strong>“It Depends…“</strong></p>
<p>The best way to scale Apache Flagon depends entirely on your use-case:</p>
<ul>
<li>how you’ll use your Apache UserALE.js data;</li>
<li>which <a href="/docs/useralejs/dataschema">data streams</a> you’ll use;</li>
<li>how long you need to keep your data.</li>
</ul>
<p><strong>The Apache Flagon Single-Node Elastic Container is an Ingredient, Not a Whole Solution</strong></p>
<p>The single-node Elastic (ELK) build distributed by <a href="/docs/stack">Apache Flagon</a> is not alone suitable for most production-level use-cases.</p>
<p>This build may be suitable for limited user-testing; just a few days of data collection from a specific application, from just a few users.</p>
<p>However, a single-node Elastic build will fail quickly for any enterprise-scale use-cases.</p>
<p>Instead, this container is meant to be a building block for larger solutions:</p>
<ol>
<li>Our ELK .yml config files for our Docker container can be used as the building-blocks for your very own load-balanced, <a href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c">multi-node cluster</a>.</li>
<li>You can use our <a href="https://github.com/apache/incubator-flagon/tree/master/kubernetes">Kubernetes build</a>, which relies on our Docker assets, to scale your Apache Flagon stack to meet your needs.</li>
<li>You can use our single-node container to scale out in <a href="https://aws.amazon.com/elasticbeanstalk/">AWS Elastic Beanstalk (EBS)</a>.</li>
</ol>
<p><strong>Apache Flagon Data Also Scales</strong></p>
<p>Flagon’s behavioral logging capability, <a href="/docs/useralejs">Apache UserALE.js</a> also scales. The most efficient way to manage scale and resources, is to <a href="/docs/useralejs/API">configure</a> or <a href="/docs/useralejs/modifying">modify</a> UserALE.js.</p>
<h3 id="sizing-up-an-elastic-stack">Sizing Up an Elastic Stack</h3>
<p>Elasticsearch isn’t a database, its a document store; UserALE.js “logs” aren’t logs once they’re indexed in Elastic, they become searchable documents.</p>
<p>Elastic has many useful <a href="(https://www.elastic.co/blog/found-sizing-elasticsearch)">guides</a> on sizing and scaling. Below, we’re adding some thoughts based on Apache Flagon’s own eccentricities.</p>
<p>####Document generation rate is the most important consideration to scaling</p>
<p>Default <a href="/docs/useralejs">Apache UserALE.js parameters</a> produce loads of <a href="/docs/useralejs/dataschema">data</a>, even from a single users.</p>
<p>We strongly suggest that you consider whether you need data from all our event-handlers.</p>
<p>If you don’t need mouseover events, for example, you can dramatically reduce the rate at which you generate data and the resources you’ll need.</p>
<p>Instead, you can <a href="/docs/useralejs/modifying">modify source</a> or use the <a href="/docs/useralejs/API">UserALE.js API</a>, and/or use <a href="/docs/useralejs">configurable HTLM5 parameters in our script tag</a> to manage data generation rate.</p>
<p>####Resource needs will also grow with document length</p>
<p><a href="https://blog.appdynamics.com/product/estimating-costs-of-storing-documents-in-elasticsearch/">Strings</a> within UserALE.js logs (see also <a href="https://www.elastic.co/guide/en/elasticsearch/reference/5.5/tune-for-disk-usage.html#_use_literal_best_compression_literal">Elastic’s tips on indexing strings</a>) can add to scaling needs.</p>
<p>One of the discriminating features of Apache UserALE.js is its precision:</p>
<ul>
<li>it captures target DOM elements;</li>
<li>it captures the DOM path that elements are nested in;</li>
<li>it captures loads of metadata (URI, page title, page url, referrer, etc.).</li>
</ul>
<p>UserALE.js fields like <code class="highlighter-rouge">path</code> and <code class="highlighter-rouge">pageUrl</code> can be lengthy for certain pages, increasing string length per document.</p>
<p>Instead, you might consider relying on <code class="highlighter-rouge">pageTitle</code> rather than <code class="highlighter-rouge">pageUrl</code>, or just <code class="highlighter-rouge">target</code> instead of <code class="highlighter-rouge">path</code></p>
<p>Below is a sample <code class="highlighter-rouge">path</code> for a Kibana element:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>```shell
...
"pageTitle": "Discover - Kibana",
"toolName": "test_app",
"userId": "nobody",
"type": "click",
"target": "a.kuiButton kuiButton--small kuiButton--secondary",
"path": [
"a.kuiButton kuiButton--small kuiButton--secondary",
"div.kbnDocTableDetails__actions",
"td",
"tr",
"tbody",
"table.kbn-table table",
"div.kbnDocTable__container",
"doc-table",
"section.dscTable",
"div.dscResults",
"div.dscWrapper__content",
"div.dscWrapper col-md-10",
"div.row",
"main.container-fluid",
"discover-app.app-container",
"div.application tab-discover",
"div.app-wrapper-panel",
"div.app-wrapper",
"div.content",
"div#kibana-body",
"body#kibana-app.coreSystemRootDomElement",
"html",
"#document",
"Window"
...
```
</code></pre>
</div>
<p>Through simple <a href="/docs/useralejs/modifying">modifications to UserALE.js source</a> or with the UserALE.js <a href="/docs/useralejs/API">API</a> you can alias verbose fields in your logs to reduce resource consumption.</p>
<p>####Additional services attached to your stack can increase resource consumption</p>
<p>Apache Flagon scales–Elastic products make it easy to attach other services to your Apache Flagon stack.</p>
<p>The <em>number of services</em> connected to your stack will affect your Elastic stacks’ performance.</p>
<p>Any production-level deployment will require, at minimum a simple three-node <a href="https://dzone.com/articles/elasticsearch-tutorial-creating-an-elasticsearch-c">Elastic cluster</a> (with one load-balancing node).</p>
<p>As you configure that cluster, be mindful that Elasticsearch is indexing and servicing queries and aggregations for connected servies.</p>
<p>Analytical services connected to the stack can consume significant resources and increase indexing and search time.</p>
<p>This can be problematic for real-time analytical and monitoring applications (including Kibana).</p>
<p>For hefty analytical services, it may be worth dedicating specific nodes in your cluster to service them.</p>
<h3 id="benchmarking-tools-and-methods-for-sizing-your-apache-flagon-stack">Benchmarking Tools and Methods for Sizing your Apache Flagon Stack</h3>
<p>For the reasons above, its really critical to do some benchmarking for your use-case prior to deciding on a scaling strategy.</p>
<p>This guide outlines a set of tools and steps for running your own benchmarking study using Flagon’s <a href="/docs/stack">single-node container</a>.</p>
<p>To generate log data, use our <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/example">UserALE.js Example</a> test utility or your own website.</p>
<p>Our test utility that makes it easy to <a href="/docs/useralejs/modifying">modify</a> UserALE.js HTML5 and API parameters on the fly.</p>
<p>However, you’ll want to experiment with your own page/application for more accurate benchmarks.</p>
<ol>
<li>
<p><strong>Start up the Apache Flagon Elastic Stack (detailed instructions <a href="https://github.com/apache/incubator-flagon/tree/master/docker">here</a>).</strong></p>
<p>Important: as noted in the instructions, you’ll need to have collected some log data to establish the index.</p>
</li>
<li><strong>Once Elasticsearch, Logstash, Kibana, and metricbeat are up, look at the <code class="highlighter-rouge">userale</code> index stats.</strong>
<div class="language-shell highlighter-rouge"><pre class="highlight"><code> <span class="c">#Index Stats using Elastic's _stats API</span>
<span class="nv">$ </span>curl localhost:9200/index_name/_stats?pretty<span class="o">=</span><span class="nb">true</span>
<span class="c">#Tailored for Apache Flagon default settings</span>
<span class="nv">$ </span>curl localhost:9200/userale/_stats?pretty<span class="o">=</span><span class="nb">true</span>
<span class="c">#Or, view in your browser</span>
http://localhost:9200/userale/_stats?pretty<span class="o">=</span><span class="nb">true</span>
</code></pre>
</div>
<p>Find the <code class="highlighter-rouge">indices</code> portion of the output. It looks like this (#note annotations):</p>
<div class="language-shell highlighter-rouge"><pre class="highlight"><code> ...
<span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span> <span class="c">#this is the index UserALE.js logs write to</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 1284, <span class="c">#this is the total # of documents in the userale index</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 241212 <span class="c">#this is size of the index in bytes (.24 MB).</span>
...
</code></pre>
</div>
<p>Let’s call this value a simple benchmark.</p>
<p>As you continue benchmarking, the <code class="highlighter-rouge">userale</code> index “size_in_bytes” will be one of your key metrics.</p>
</li>
<li>
<p><strong>Next, let’s see how much data UserALE.js produces with default parameters on your page or app.</strong></p>
<p>Drop in a UserALE.js script-tag into your project (see <a href="/docs/useralejs">instructions</a>).</p>
<p>Here is an example of the script tag (with <code class="highlighter-rouge">settings</code>) we’re using in the UserALE.js Example page for this test:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <script
src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
data-url="http://localhost:8100/"
data-user="example-user"
data-version="1.1.1"
data-tool="Apache UserALE.js Example"
></script>
</code></pre>
</div>
<p>To get a conservative upper-bound, generate as many mouseover and scroll behaviors in your page/app as you can.</p>
<p>Doing that for 5 minutes solid, our <code class="highlighter-rouge">userale</code> index looks like this (#note annotations):</p>
<div class="language-shell highlighter-rouge"><pre class="highlight"><code> <span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 3282, <span class="c">#new userale document count</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 820978 <span class="c">#new size of the index (.82 MB)</span>
</code></pre>
</div>
<p><strong>That’s +1,998 documents (2000) and +579,866 bytes (.58 MB) generated with UserALE.js by one user in 5 mins</strong>.</p>
<p>Assuming this rate over an 8 hour period each day for 20 working days: that’s <strong>1.1 GB per month</strong>.</p>
<p><strong>This is an ultra-conservative, worst-case-scenario estimate</strong> because no one uses pages or applications this way.</p>
<p>If you’re a scientist or researcher, these figures might be fine, but it might be overkill for business analytics.</p>
<p>To find the biggest culprit in data generation: take a look at our <code class="highlighter-rouge">Apache Flagon Page Usage Dashboard</code> to see.</p>
<p><img src="/images/mouseOverBench1.png" width="750" height="500" /></p>
<p>Mouseovers accounted for a lot of the data we just produced–its written frequently to index.</p>
</li>
<li>
<p><strong>Next, scale back UserALE.js mouseover handling and see how this changes data generation rate</strong>.</p>
<p>Using the UserALE.js HTML5 <code class="highlighter-rouge">settings</code> in our script tag, you can “downsample” certain event handler that generate a lot of documents.</p>
<p>Here’s what our script tag looks like now:</p>
<div class="highlighter-rouge"><pre class="highlight"><code> <script
src="file:/// ... /UserALEtest/userale-1.1.0.min.js"
data-url="http://localhost:8100/"
data-user="example-user"
data-version="1.1.1"
data-resolution=1000 #increased the delay between collection of frequent events (e.g., mouseovers).
data-tool="Apache UserALE.js Example"
></script>
</code></pre>
</div>
<p>Next, replicate your benchmarking, behaving in a similar way for the same amount of time.</p>
<p>Here’s what our <code class="highlighter-rouge">userale</code> index looks like now after another 5 minutes of vigorous behavior.</p>
<div class="language-shell highlighter-rouge"><pre class="highlight"><code> <span class="s2">"indices"</span> : <span class="o">{</span>
<span class="s2">"userale"</span> : <span class="o">{</span>
<span class="s2">"uuid"</span> : <span class="s2">"0h0Wxe2cSwqMALs4QCJ8Tw"</span>,
<span class="s2">"primaries"</span> : <span class="o">{</span>
<span class="s2">"docs"</span> : <span class="o">{</span>
<span class="s2">"count"</span> : 4800, <span class="c">#new userale document count</span>
<span class="s2">"deleted"</span> : 0
<span class="o">}</span>,
<span class="s2">"store"</span> : <span class="o">{</span>
<span class="s2">"size_in_bytes"</span> : 115978 <span class="c">#new size of the index (1.1 MB) </span>
</code></pre>
</div>
<p><strong>That is +1518 documents and +295K bytes (.30 MB)</strong></p>
<p>But, it’s 500 fewer than our first benchmark and ~40% less growth in the store.</p>
<p>At <strong>~576 MB per working month</strong> we’ve cut data generation considerably by modifying one parameter in the script tag.</p>
<p>The proportion of mouseover events is down by ~50%, and 25% fewer documents overall:</p>
<p><img src="/images/mouseOverBench2.png" width="750" height="500" /></p>
</li>
<li>
<p><strong>Still too much data? Below are some other ways to curb the growth of your <code class="highlighter-rouge">userale</code> index</strong>.</p>
<ul>
<li>Modify <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> to cut down on event handlers (<code class="highlighter-rouge">attachHandlers.js</code>).</li>
<li>Drop interval logging by modifying UserALE.js <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> (<code class="highlighter-rouge">attachHandlers.js</code> or <code class="highlighter-rouge">packageLogs.js</code>).</li>
<li>Reduce the amount of metadata you collect by modifying <a href="https://github.com/apache/incubator-flagon-useralejs/tree/master/src">UserALE.js source</a> (<code class="highlighter-rouge">packageLogs.js</code>).</li>
<li>Use different UserALE.js script builds for different pages within your site/app to serve different logging needs.</li>
<li>You can use our <a href="/docs/useralejs/API">API</a> for surgical precision in how specific elements (targets) on your page generate data.</li>
</ul>
</li>
</ol>
<h3 id="other-tools-to-support-benchmarking-for-scaling">Other Tools to Support Benchmarking for Scaling</h3>
<p>In our benchmarking guide, we primarily used Elastic’s <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-stats.html"><code class="highlighter-rouge">Stats</code> API</a>.</p>
<p>You can <strong>use other <a href="https://www.datadoghq.com/blog/collect-elasticsearch-metrics/#index-stats-api">Elastic APIs</a></strong> for different views of what’s going on inside your Apache Flagon stack.</p>
<p>For more streamlined views into your indices, try the <code class="highlighter-rouge">CAT</code> API. Try this call in your browser:</p>
<div class="highlighter-rouge"><pre class="highlight"><code>http://localhost:9200/_cat/indices?format=json&bytes=b&pretty
Output is very simple and index sizes stack on top of one another
```shell
[
{
"health" : "green",
"status" : "open",
"index" : ".kibana_1",
"uuid" : "FnI_6AQYQEWp2mIxSlM8HQ",
"pri" : "1",
"rep" : "0",
"docs.count" : "184",
"docs.deleted" : "13",
"store.size" : "366457",
"pri.store.size" : "366457"
},
{
"health" : "yellow",
"status" : "open",
"index" : "metricbeat-6.6.2-2019.04.22",
"uuid" : "pDYNmzsxTFu9Z0Tc1_GdLw",
"pri" : "5",
"rep" : "1",
"docs.count" : "4687",
"docs.deleted" : "0",
"store.size" : "6309920",
"pri.store.size" : "6309920"
},
{
"health" : "green",
"status" : "open",
"index" : "userale",
"uuid" : "0h0Wxe2cSwqMALs4QCJ8Tw",
"pri" : "1",
"rep" : "0",
"docs.count" : "14018",
"docs.deleted" : "0",
"store.size" : "2235963",
"pri.store.size" : "2235963"
},
{
"health" : "yellow",
"status" : "open",
"index" : "metricbeat-6.6.2-2019.04.27",
"uuid" : "wTyUBXvNRMOwpR4lDF9BNA",
"pri" : "5",
"rep" : "1",
"docs.count" : "107124",
"docs.deleted" : "0",
"store.size" : "63263644",
"pri.store.size" : "63263644"
}
]
```
</code></pre>
</div>
<p><strong>Use Flagon’s pre-configured metricbeat service</strong> to run with Flagon.</p>
<p>You can use this utility to see how your Apache Flagon stack is utilizing disk and compute resources.</p>
<p>See a sample view of Metricbeat stats below:</p>
<p><img src="/images/metricBeat.png" width="750" height="500" /></p>
<h3 id="wonky-things-that-can-and-will-happen-as-you-benchmark">Wonky Things that Can and Will Happen as You Benchmark</h3>
<p>You just finished a benchmarking session after modifying UserALE.js to produce less data.</p>
<p>What you find is that your new store size is either dramatically bigger than your last benchmark or smaller (which should be impossible).</p>
<p>What happened is a thing called <a href="https://www.elastic.co/guide/en/elasticsearch/guide/current/merge-process.html">merging</a>.</p>
<p>As data is collected it’s gathered into segments within an index. Each segment is an element of your index and takes up storage.</p>
<p>As Elastic (Lucene) indexes, it merges these segments into larger segments to reduce the overall number of segments to minimize storage.</p>
<p>This means that a call to Elastic’s <code class="highlighter-rouge">STATS</code> API can result in a view into the store size at different stages in the merging process.</p>
<p>If your store size looks smaller than your last benchmark. You should re-run it then wait.</p>
<p>If your store size looks way to big, then wait. After a minute, call the <code class="highlighter-rouge">STATS</code> API again, and you’ll likely see a more sensible store size.</p>
<h3 id="summary">Summary</h3>
<p>Benchmarking and adjusting your data-rate so that you can scale how you want to is made very easy in Apache Flagon.</p>
<p>We combine easily deployed and modified capabilities with the power of Elastic’s APIs and visualization capabilities.</p>
<p>Again, Flagon’s single-node container is not a scaling solution.</p>
<p>It’s a building block benchmarking tool to help you build and manage scale and cost.</p>
<p>Subscribe to our <a href="mailto:dev-subscribe@flagon.incubator.apache.org">dev list</a> and join the conversation!</p>
</div>
</div>
</div>
<div class="footer ui inverted very padded vertical segment">
<div class="ui vertical segment">
<div class="ui four column grid">
<div class="column">
<h4 class="ui inverted header">About</h4>
<div class="ui inverted link list">
<a class="item" href="/stack">Stack</a>
<a class="item" href="/userale">UserALE</a>
<a class="item" href="/distill">Distill</a>
<a class="item" href="/tap">Tap</a>
<a class="item" href="/stout">Stout</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Docs</h4>
<div class="ui inverted link list">
<a class="item" href="/docs/stack">Stack</a>
<a class="item" href="/docs/userale">UserALE</a>
<a class="item" href="/docs/useralejs">UserAle.js</a>
<a class="item" href="/docs/useralepy">UserAle.py</a>
<a class="item" href="/docs/distill">Distill</a>
<a class="item" href="/docs/tap">Tap</a>
<a class="item" href="/docs/stout">Stout</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Community</h4>
<div class="ui inverted link list">
<a class="item" href="https://github.com/apache?q=flagon">GitHub</a>
<a class="item" href="http://stackoverflow.com/questions/tagged/apache-flagon">StackOverflow</a>
<a class="item" href="https://issues.apache.org/jira/browse/FLAGON">Jira Issue Tracker</a>
<a class="item" href="https://cwiki.apache.org/confluence/display/FLAGON">Project Wiki</a>
<a class="item" href="https://twitter.com/ApacheFlagon">Twitter</a>
</div>
</div>
<div class="column">
<h4 class="ui inverted header">Apache</h4>
<div class="ui inverted link list">
<a class="item" href="http://www.apache.org/">Apache Software Foundation</a>
<a class="item" href="http://www.apache.org/licenses/">License</a>
<a class="item" href="http://www.apache.org/foundation/sponsorship.html">Sponsorship</a>
<a class="item" href="http://www.apache.org/foundation/thanks.html">Thanks</a>
<a class="item" href="http://www.apache.org/security/">Security</a>
</div>
</div>
</div>
</div>
<div class="ui inverted vertical segment">
<div class="ui two column middle aligned grid">
<div class="column">
Apache Flagon is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision making process have stabilized in a manner consistent with other successful ASF projects. While incubation status is not necessarily a reflection of the completeness or stability of the code, it does indicate that the project has yet to be fully endorsed by the ASF.
</div>
<div class="center aligned column">
Copyright © 2019 The Apache Software Foundation, Licensed under the <a href="http://www.apache.org/licenses/LICENSE-2.0">Apache License, Version 2.0</a>.<br>
Apache Flagon, Flagon, Apache, the Apache feather logo, the Apache Flagon logo and the Apache Incubator logo are trademarks of The Apache Software Foundation.
</div>
</div>
</div>
<div class="ui inverted vertical segment">
<div class="ui one column middle aligned grid">
<div class="center aligned column">
<a href="http://incubator.apache.org/">
<img src="/images/incubator-logo-white.jpg">
</a>
<a href="https://www.apache.org/events/current-event.html">
<img src="https://www.apache.org/events/current-event-234x60.png" alt="ASF Current Event">
</a>
</div>
</div>
</div>
</div>
</body>
</html>