/
nozzle-tutorial.html.md.erb
268 lines (225 loc) · 17.4 KB
/
nozzle-tutorial.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
---
title: Deploying a Nozzle to the Loggregator Firehose
owner: Logging and Metrics
---
<strong><%= modified_date %></strong>
<% $this_topic = 'nozzle-tutorial' %>
This topic describes deploying a [nozzle](./architecture.html#nozzles) application to the Cloud Foundry (CF) [Loggregator Firehose](./architecture.html#firehose).
The Cloud Foundry Loggregator team created an example nozzle application for use with this tutorial.
The procedure described below deploys this example nozzle to the Firehose of a Cloud Foundry installation deployed locally with BOSH Lite. For more information about BOSH Lite, see the [BOSH Lite GitHub repository](https://github.com/cloudfoundry/bosh-lite).
## Prerequisites
- [BOSH CLI](https://bosh.io/docs/bosh-cli.html) installed locally.
- Spiff installed locally and added to the load path of your shell. See [Spiff on GitHub](https://github.com/cloudfoundry-incubator/spiff).
- BOSH Lite deployed locally using VirtualBox. See [BOSH Lite on GitHub](https://github.com/cloudfoundry/bosh-lite).
- A working [Cloud Foundry](https://github.com/cloudfoundry/cf-release) deployment, including Loggregator, deployed with your local BOSH Lite. This serves as our source of data. See [Deploying Cloud Foundry using BOSH Lite](https://docs.cloudfoundry.org/deploying/boshlite/deploy_cf_boshlite.html), or use the `provision_cf` script included in the [BOSH Lite release](https://github.com/cloudfoundry/bosh-lite#deploy-cloud-foundry).
<p class='note'><strong>Note</strong>: Deploying Cloud Foundry can take up to several hours, depending on your internet bandwith, even when using the automated <code>provision_cf</code> script.</p>
## <a id='dl-bosh-manifest'></a>Step 1: Download Cloud Foundry BOSH Manifest ##
<%= partial 'download_bosh_manifest' %>
## <a id='add-client'></a>Step 2: Add UAA Client ##
You must authorize the example nozzle as a UAA client for your CF deployment. To do this, add an entry for the example nozzle as `client` for `uaa` under the `properties` key in your CF deployment manifest. You must enter the example nozzle object in the correct location in the manifest, and with the correct indentation, as described below.
<p class='note'>Deployment manifests are YAML files. Visit <a href='http://www.yaml.org/start.html'>YAML</a> to learn about YAML syntax.
</p>
1. Open the deployment manifest in a text editor.
1. Locate the left-aligned `properties` key.
1. Under the `properties` key, locate `uaa` at the next level of indentation.
1. Under the `uaa` key, locate the `clients` key at the next level of indentation.
1. Enter properties for the `example-nozzle` at the next level of indentation, exactly as shown below. The `...` in the text below indicate other properties that may populate the manifest at each level in the hierarchy.
```
properties:
...
uaa:
...
clients:
...
example-nozzle:
access-token-validity: 1209600
authorized-grant-types: client_credentials
override: true
secret: example-nozzle
authorities: oauth.login,doppler.firehose
```
1. Save the deployment manifest file.
## <a id='redeploy'></a>Step 3: Redeploy Cloud Foundry ##
Run `bosh -e MY-ENV deploy` to deploy Cloud Foundry with BOSH. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'cf-warden' on 'Bosh Lite Director'
Getting deployment properties from director...<br>
Detecting deployment changes
----------------------------
Releases
No changes<br>
Compilation
No changes<br>
Update
No changes<br>
Resource pools
No changes<br>
Disk pools
No changes<br>
Networks
No changes<br>
Jobs
No changes<br>
Properties
uaa
clients
example-nozzle
+ access-token-validity: 1209600
+ authorized-grant-types: authorization_code,client_credentials,refresh_token
+ override: true
+ secret: example-nozzle
+ scope: openid,oauth.approvals,doppler.firehose
+ authorities: oauth.login,doppler.firehose<br>
Meta
No changes<br>
Please review all changes carefully<br>
Deploying
---------
Are you sure you want to deploy? (type 'yes' to continue):yes
</pre>
## <a id='dl-release'></a>Step 4: Clone Example Release ##
The Cloud Foundry Loggregator team created an example nozzle application for
use with this tutorial.
1. Run `git clone` to clone the main release repository from [GitHub](https://github.com/cloudfoundry-incubator/example-nozzle-release).
<pre class='terminal'>
$ git clone http<span>s</span>://github.com/cloudfoundry-incubator/example-nozzle-release.git
Cloning into 'example-nozzle-release'...
</pre>
1. Navigate to the `example-nozzle-release` directory.
<pre class='terminal'>
$ cd example-nozzle-release
</pre>
1. Run `git submodule update --init --recursive` to update all of the included submodules.
<pre class='terminal'>
$ git submodule update --init --recursive
Submodule 'src/github.com/cloudfoundry-incubator/example-nozzle' (<span>git@</span>github.com:cloudfoundry-incubator/example-nozzle.git) registered for path 'src/github.com/cloudfoundry-incubator/example-nozzle'
Submodule 'src/github.com/cloudfoundry-incubator/uaago' (<span>git@</span>github.com:cloudfoundry-incubator/uaago.git) registered for path 'src/github.com/cloudfoundry-incubator/uaago'
...
Cloning into 'src/github.com/cloudfoundry-incubator/example-nozzle'...
...
</pre>
## <a id='step-id'></a>Step 5: Prepare Nozzle Manifest ##
Complete the following steps to prepare the nozzle deployment manifest:
1. In the `example-nozzle-release` directory, navigate to the `templates` directory.
<pre class='terminal'>
$ cd templates
</pre>
Within this directory, examine the two YAML files.
`bosh-lite-stub.yml` contains the values used to populate the missing
information in `template.yml`.
By combining these two files we create a deployment manifest for our nozzle.
1. Create a `tmp` directory for the compiled manifest.
1. Use [Spiff](https://github.com/cloudfoundry-incubator/spiff) to compile a deployment manifest from the template and stub, and save this manifest.
<pre class='terminal'>
$ spiff merge templates/template.yml templates/bosh-lite-stub.yml > tmp/manifest_bosh_lite.yml
</pre>
1. Run `bosh -e MY-ENV deployments` to identify the names of all deployments in the environment you specify. Replace `MY-ENV` with the alias you set for your BOSH Director.
1. Run `bosh -e MY-ENV env --column=uuid` to obtain your BOSH Director UUID. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev env --column=uuid
</pre>
1. In the compiled nozzle deployment manifest, locate the `director_uuid` property. Replace `PLACEHOLDER-DIRECTOR-UUID` with your BOSH Director UUID.
```
compilation:
cloud_properties:
name: default
network: example-nozzle-net
reuse_compilation_vms: true
workers: 1
director_uuid: PLACEHOLDER-DIRECTOR-UUID # replace this
```
<p class='note'><strong>Note</strong>: If you do not want to see the complete deployment procedure, run the following command to automatically prepare the manifest: <code>scripts/make_manifest_spiff_bosh_lite</code>
</p>
## <a id='create'></a>Step 6: Create Nozzle BOSH Release ##
Use the `bosh -e MY-ENV create-release --name RELEASE-NAME` command to create a BOSH release. Replace `MY-ENV` with the alias you set for your BOSH Director. Replace RELEASE-NAME with `example-nozzle` to match the [UAA client](#add-client) that you created in the CF deployment manifest. For example:
<pre class='terminal'>
$ bosh -e dev create-release --name example-nozzle
Syncing blobs...
...
</pre>
## <a id='upload'></a>Step 7: Upload Nozzle BOSH Release ##
Run `bosh -e MY-ENV upload-release` to upload the release that you created in [Step 6: Create Nozzle BOSH Release](#create). Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev upload-release
Acting as user 'admin' on 'Bosh Lite Director'<br>
Copying packages
\----------------
example-nozzle
golang1.7<br>
Copying jobs
\------------
example-nozzle<br>
Generated /var/folders/4n/qs1rjbmd1c5gfb78m3_06j6r0000gn/T/d20151009-71219-17a5m49/d20151009-71219-rts928/release.tgz
Release size: 59.2M<br>
Verifying release...
...
Release info
\------------
Name: nozzle-test
Version: 0+dev.2<br>
Packages
- example-nozzle (b0944f95eb5a332e9be2adfb4db1bc88f9755894)
- golang1.7 (b68dc9557ef296cb21e577c31ba97e2584a5154b)<br>
Jobs
- example-nozzle (112e01c6ee91e8b268a42239e58e8e18e0360f58)<br>
License
- none<br>
Uploading release
</pre>
## <a id='deploy'></a>Step 8: Deploy Nozzle ##
Run `bosh -e MY-ENV deploy` to deploy the nozzle. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'example-nozzle-lite' on 'Bosh Lite Director'
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully<br>
Deploying
\---------
Are you sure you want to deploy? (type 'yes' to continue):yes
</pre>
## <a id='view'></a>Step 9: View Nozzle Output ##
The example nozzle outputs all of the data originating coming from the Firehose to its log files. To view this data, SSH into the example-nozzle VM and examine the logs.
1. Run `bosh -e MY-ENV ssh` to access the nozzle VM at the IP configured in the nozzle's manifest template stub `./templates/bosh-lite-stub.yml`. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev ssh example-nozzle
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.19.0-25-generic x86_64)
Documentation: <span>https<span>://help.ubuntu.com/
Last login: Wed Sep 23 21:29:50 2015 from 192.0.2.1
</pre>
1. Use the `cat` command to output the `stdout` log file.
<pre class='terminal'>
$ cat /var/vcap/sys/log/example-nozzle/example-nozzle.stdout.log
===== Streaming Firehose (will only succeed if you have admin credentials)
origin:"DopplerServer" eventType:ValueMetric timestamp:1443046217700750747 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"messageRouter.numberOfFirehoseSinks" value:1 unit:"sinks" >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910193187 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeAgentListener.receivedByteCount" delta:771 total:386368 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910360012 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeMarshaller.counterEventMarshalled" delta:8 total:2619 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910252169 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeUnmarshaller.valueMetricReceived" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910294255 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"MessageAggregator.uncategorizedEvents" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910318582 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeMarshaller.valueMetricMarshalled" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910339088 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"MessageAggregator.counterEventReceived" delta:8 total:2619 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910379199 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"DopplerForwarder.sentMessages" delta:17 total:7082 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910394886 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeAgentListener.receivedMessageCount" delta:9 total:4525 >
origin:"router__0" eventType:HttpStartStop timestamp:1443046219105062148 deployment:"cf-warden" job:"router_z1" index:"0" ip:"203.0.113.22" httpStartStop:<startTimestamp:1443046219096144435 stopTimestamp:1443046219105062148 requestId:<low:7224371029360401779 high:6477133057804225906 > peerType:Client method:POST uri:"uaa.service.cf.internal:8080/oauth/token" remoteAddress:"" userAgent:"" statusCode:200 contentLength:-1 >
origin:"api_z1_0" eventType:HttpStartStop timestamp:1443046219109842455 deployment:"cf-warden" job:"api_z1" index:"0" ip:"203.0.113.134" httpStartStop:<startTimestamp:1443046219106687524 stopTimestamp:1443046219109842455 requestId:<low:7948515536237261025 high:3934284340053050957 > peerType:Server method:GET uri:"routing-api.service.cf.internal:3000/v1/routes" remoteAddress:"203.0.113.15:49577" userAgent:"Go 1.1 package http" statusCode:200 contentLength:108 >
origin:"router__0" eventType:HttpStartStop timestamp:1443046219110064368 deployment:"cf-warden" job:"router_z1" index:"0" ip:"203.0.113.22" httpStartStop:<startTimestamp:1443046219106025264 stopTimestamp:1443046219110064368 requestId:<low:7948515536237261025 high:3934284340053050957 > peerType:Client method:GET uri:"routing-api.service.cf.internal:3000/v1/routes" remoteAddress:"" userAgent:"" statusCode:200 contentLength:108 >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177165446 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"logSenderTotalMessagesRead" value:0 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177288325 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"numGoRoutines" value:22 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177346726 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.numBytesAllocated" value:347992 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177274975 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"numCPUS" value:4 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177310389 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.numBytesAllocatedHeap" value:347992 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177330214 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.numBytesAllocatedStack" value:393216 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177353454 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.numMallocs" value:312614 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177360052 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.numFrees" value:309258 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177481456 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" valueMetric:<name:"memoryStats.lastGCPauseTimeNS" value:318837 unit:"count" >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880585603 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"dropsondeListener.receivedMessageCount" delta:59 total:61634 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880895040 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.146" counterEvent:<name:"dropsondeListener.receivedByteCount" delta:9462 total:9111082 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881017888 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"dropsondeUnmarshaller.httpStartStopReceived" delta:7 total:2191 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881011670 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"dropsondeUnmarshaller.valueMetricReceived" delta:28 total:35309 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929574 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"dropsondeUnmarshaller.counterEventReceived" delta:24 total:24134 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881004417 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"httpServer.receivedMessages" delta:59 total:61634 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929568 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"203.0.113.142" counterEvent:<name:"signatureVerifier.validSignatures" delta:59 total:61634 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046220058280679 deployment:"cf-warden" job:"api_z1" index:"0" ip:"203.0.113.134" counterEvent:<name:"MessageAggregator.uncategorizedEvents" delta:44 total:12891 >
</pre>