/
setting-up-splunk-enterprise-for-aws.html
286 lines (252 loc) · 14.2 KB
/
setting-up-splunk-enterprise-for-aws.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
<!DOCTYPE html>
<html lang="en" prefix="og: http://ogp.me/ns# fb: https://www.facebook.com/2008/fbml">
<head>
<title>Donne Martin</title>
<!-- Using the latest rendering mode for IE -->
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="description" content="">
<meta name="author" content="">
<meta name="author" content="Donne Martin" />
<!-- Open Graph tags -->
<meta property="og:site_name" content="Donne Martin" />
<meta property="og:type" content="website"/>
<meta property="og:title" content="Donne Martin"/>
<meta property="og:url" content="."/>
<meta property="og:description" content="Donne Martin"/>
<!-- Bootstrap -->
<link rel="stylesheet" href="./theme/css/bootstrap.min.css" type="text/css"/>
<link href="./theme/css/pygments/monokai.css" rel="stylesheet">
<!-- Custom CSS -->
<link href="./theme/css/agency.css" rel="stylesheet">
<link href="./theme/css/custom.css" rel="stylesheet">
<!-- Custom Fonts -->
<link href="./theme/font-awesome/css/font-awesome.min.css" rel="stylesheet" type="text/css">
<link href="https://fonts.googleapis.com/css?family=Montserrat:400,700" rel="stylesheet" type="text/css">
<link href='https://fonts.googleapis.com/css?family=Kaushan+Script' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Droid+Serif:400,700,400italic,700italic' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Roboto+Slab:400,100,300,700' rel='stylesheet' type='text/css'>
<!-- HTML5 Shim and Respond.js IE8 support of HTML5 elements and media queries -->
<!-- WARNING: Respond.js doesn't work if you view the page via file:// -->
<!--[if lt IE 9]>
<script src="https://oss.maxcdn.com/libs/html5shiv/3.7.0/html5shiv.js"></script>
<script src="https://oss.maxcdn.com/libs/respond../theme/js/1.4.2/respond.min.js"></script>
<![endif]-->
</head><body id="page-top" class="index">
<!-- Banner -->
<!-- End Banner -->
<div class="container">
<div class="row">
<div class="col-lg-12">
<nav class="navbar navbar-default navbar-fixed-top" style="background-color: #000">
<div class="container">
<!-- Brand and toggle get grouped for better mobile display -->
<div class="navbar-header page-scroll">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target="#bs-example-navbar-collapse-1">
<span class="sr-only">Toggle navigation</span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand page-scroll" href=".">Donne Martin</a>
</div>
<!-- Collect the nav links, forms, and other content for toggling -->
<div class="collapse navbar-collapse" id="bs-example-navbar-collapse-1">
<ul class="nav navbar-nav navbar-right">
<li class="hidden">
<a href="#page-top"></a>
</li>
<li>
<a class="page-scroll" href="./#likes">Likes</a>
</li>
<li>
<a class="page-scroll" href="./#portfolio">GitHub</a>
</li>
<li>
<a class="page-scroll" href="./#about">About</a>
</li>
<li>
<a class="page-scroll" href="./#contact">Contact</a>
</li>
<li>
<a class="page-scroll" href="./archives">Blog</a>
</li>
<li>
<a class="page-scroll" href="http://donnemartin.com/viz/">Viz</a>
</li>
</ul>
</div>
<!-- /.navbar-collapse -->
</div>
<!-- /.container-fluid -->
</nav> <section id="content" class="section-top-padding">
<article class="article-top-padding">
<h1>
<a href="./setting-up-splunk-enterprise-for-aws.html"
rel="bookmark"
title="Permalink to Setting Up Splunk Enterprise for AWS">
Setting Up Splunk Enterprise for AWS
</a>
</h1>
<i><time datetime="2015-02-01T00:00:00-05:00"> Sun 01 February 2015</time></i>
<div class="entry-content">
<div class="panel">
<br/>
</div>
<div class="container">
<br/>
<img class="img-responsive" src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_cover.png">
</div>
<hr class="featurette-divider">
<p>I recently hooked up Splunk with AWS to search, monitor, and analyze log files. Splunk indexes data on read and allows for super-fast searching and visualization. I like to think of Splunk as Google Search for log files with visualization built-in.</p>
<h2>Benefits</h2>
<ul>
<li>Allow for real time log analysis</li>
<li>Allow for faster historical log analysis</li>
<li>Provide data access to non-technical users who don’t know Hadoop</li>
<li>Provide access to previously untapped data</li>
<li>Promote transparency</li>
</ul>
<h2>Competitors</h2>
<ul>
<li><a href="http://www.elasticsearch.org/overview/elkdownloads/">ELK: ElasticSearch, LogStash, Kibana</a></li>
<li><a href="http://docs.fluentd.org/articles/free-alternative-to-splunk-by-fluentd">Fluentd</a></li>
<li><a href="https://www.sumologic.com/">Sumo Logic</a></li>
</ul>
<p>Here's a good article comparing <a href="http://riskfocus.com/splunk-vs-elk-part-1-cost/">Splunk vs ELK</a> and a <a href="http://www.quora.com/What-are-the-best-free-alternatives-to-Splunk">Quora post</a> detailing free alternatives to Splunk.</p>
<h2>Product Overview: Splunk Free, Enterprise, Storm, Cloud</h2>
<p>Splunk has an interesting <a href="http://www.splunk.com/view/pricing/SP-CAAADFV">licensing model</a> where the cost per GB of daily indexing decreases the more you index. For example, indexing 1 GB per day would cost $1800 (1 GB x $1800), whereas indexing 10 GB per day would cost $10000 (10 GB x $1000).</p>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_license.png" class="img-responsive">
</p>
<p><a href="http://www.splunk.com/view/cloud/SP-CAAAGE8#aws">Splunk Enterprise</a> is a "bring your own license" model and requires you to host the server(s).</p>
<p><a href="http://docs.splunk.com/Documentation/Splunk/latest/Admin/MoreaboutSplunkFree">Splunk Free</a> is similar to Splunk Enterprise and allows you to index 500 MB per day.</p>
<p><a href="http://docs.splunk.com/Documentation/Storm/Storm/User/StormFreeFAQ">Splunk Storm</a> is a free version of Splunk Cloud that allows for 20 GB of storage and 30 days of data retention. You can have five project members and three projects. It was recently <a href="http://www.splunk.com/en_us/products/eol/storm-eol.html">announced</a> that Splunk Storm will no longer be supported.</p>
<p><a href="http://www.splunk.com/view/cloud/SP-CAAAG58">Splunk Cloud</a> is a hosted solution of Splunk Enterprise without the restrictions of Splunk Storm. Splunk Cloud includes 5 GB daily indexing, 90 days of storage, and a 100% uptime SLA.</p>
<p><a href="http://www.splunk.com/view/SP-CAAAE8W">Product Comparison Matrix</a></p>
<h2>Cost Comparison: Splunk Enterprise vs Splunk Cloud</h2>
<p>If hosting Splunk Enterprise on AWS, you must factor in the cost of compute, storage, bandwidth, server administration, etc. For example:</p>
<ul>
<li>c3.xlarge for one year on-demand costs $1840*</li>
<li>1 TB general purpose SSD costs $1164</li>
</ul>
<p>*4 EC2 Compute Units and 7 GB of RAM is recommended for daily indexing of < 20 GB according to <a href="http://blogs.splunk.com/2012/03/07/splunk-and-aws-sizing-revisited/">Splunk Answers</a></p>
<p>Just factoring in the Splunk Enterprise license, compute costs, and EBS costs (no bandwidth, server administration, etc) totals $4804. Splunk Enterprise seems like a fair choice if just starting out with Splunk with low daily index volumes. Splunk Cloud becomes more appealing for larger workloads (up to 5 GB daily included) or for those who wish to have more of a turnkey solution. Splunk Cloud also runs a c3.4xlarge which has 16 EC2 Compute Units and 30 GB of RAM.</p>
<p>Splunk slashed prices for its cloud offering by 33% in August of 2014 in response to price cuts from Amazon. Amazon has dropped prices over 40 times over the last six years.</p>
<h2>Getting Started: Splunk on the AWS Market Place</h2>
<p>The easiest way to get started with Splunk Enterprise on AWS is to spin up a Splunk instance from the <a href="https://aws.amazon.com/marketplace">AWS Market Place</a>. There are no additional charges per hour other than what you would pay Amazon for compute.</p>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_market.png" class="img-responsive">
</p>
<h2>Setup a Splunk IAM Account</h2>
<p>Create an IAM account that Splunk will use to access your AWS. Enter the IAM credentials in the Splunk for AWS plugin. The following sections describe what IAM permissions are required for each AWS feature.</p>
<h2>Setup Splunk for S3</h2>
<p>Create the S3 bucket and related objects.</p>
<p>Add IAM List and Get permissions for buckets and for objects in buckets.</p>
<h2>Setup Splunk for CloudTrail</h2>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_cloudtrail.png" class="img-responsive">
</p>
<p>Setup Simple Queue Service (SQS) to subscribes to the Simple Notification Service (SNS) notification events from CloudTrail:</p>
<ul>
<li>Enable CloudTrail</li>
<li>Create an S3 Bucket for CloudTrail events</li>
<li>Enable SNS Notifications</li>
<li>Create an SQS</li>
<li>Subscribe to the SNS Notifications that you enabled</li>
</ul>
<p>Add IAM permissions for the following:</p>
<table>
<thead>
<tr>
<th>AWS Product</th>
<th>IAM Permission</th>
</tr>
</thead>
<tbody>
<tr>
<td>CloudTrail</td>
<td>CloudTrail Read Only Access</td>
</tr>
<tr>
<td>S3 bucket that collects your CloudTrail logs</td>
<td>Get (and Delete only if you want to delete bucket log files after loading them into Splunk)</td>
</tr>
<tr>
<td>SQS subscribed to the S3 bucket that collects CloudTrail logs</td>
<td>ReceiveMessage, SendMessage, ListQueues, GetQueueUri</td>
</tr>
</tbody>
</table>
<h2>Setup Splunk for Config</h2>
<p>Config setup is similar to CloudTrail. Refer to the instructions for CloudTrail, substituting Config for CloudTrail.</p>
<h2>Setup Splunk for CloudWatch</h2>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_config.png" class="img-responsive">
</p>
<p>CloudWatch requires no additional configuration for Splunk other than Describe, List, and Get IAM permissions.</p>
<h2>Setup Splunk for Billing</h2>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_billing.png" class="img-responsive">
</p>
<p>Review the <a href="http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/detailed-billing-reports.html">AWS documentation</a> to turn on AWS billing reports.</p>
<p>Add IAM billing permissions for ViewBilling and ViewUsage.</p>
<h2>Setup Splunk Indices</h2>
<p>Create one index per data input. For example, the index aws-billing would correspond to to the AWS billing feature. Be careful with naming indices, as Splunk requires the following indices to be named the following:</p>
<ul>
<li>CloudTrail: aws-cloudtrail</li>
<li>Config: aws-config</li>
</ul>
<p>If you decide to change the names of the indices for CloudTrail and Config, you'll have to update the macros.conf config files.</p>
<h2>Setup Splunk Data Inputs</h2>
<p>Create one data input and match it up with an index. Note Splunk requires the following data input sources for Cloudtrail and Config to be named the following:</p>
<ul>
<li>aws:cloudtrail</li>
<li>aws:config</li>
</ul>
<h2>Search and Visualize!</h2>
<p>You should now be able to search and visualize AWS.</p>
<p align="center">
<img src="https://raw.githubusercontent.com/donnemartin/donnemartin.github.io/master/images/posts/splunk_viz.png" class="img-responsive">
</p>
</div>
<hr class="featurette-divider">
<!-- /.entry-content -->
</article>
</section>
</div>
</div>
</div>
<footer>
<div class="container">
<div class="row">
<div class="col-md-12 text-left">
<span class="copyright">Copyright © Donne Martin 2014-Present</span>
</div>
</div>
</div>
</footer>
<script src="./theme/js/jquery.min.js"></script>
<!-- Include all compiled plugins (below), or include individual files as needed -->
<script src="./theme/js/bootstrap.min.js"></script>
<!-- Enable responsive features in IE8 with Respond.js (https://github.com/scottjehl/Respond) -->
<script src="./theme/js/respond.min.js"></script>
<!-- Plugin JavaScript -->
<script src="http://cdnjs.cloudflare.com/ajax/libs/jquery-easing/1.3/jquery.easing.min.js"></script>
<script src="./theme/js/classie.js"></script>
<script src="./theme/js/cbpAnimatedHeader.js"></script>
<!-- Custom Theme JavaScript -->
<script src="./theme/js/agency.js"></script>
<!-- Google Analytics Universal -->
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-54747412-1', 'auto');
ga('send', 'pageview');
</script>
<!-- End Google Analytics Universal Code -->
</body>
</html>