Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
271 lines (226 sloc) 17.8 KB
title: Deploying a Nozzle to the Loggregator Firehose
owner: Logging and Metrics
<strong><%= modified_date %></strong>
<% $this_topic = 'nozzle-tutorial' %>
This topic describes deploying a [nozzle](./architecture.html#nozzles) application to the Cloud Foundry (CF) [Loggregator Firehose](./architecture.html#firehose).
The Cloud Foundry Loggregator team created an example nozzle application for use with this tutorial.
The procedure described below deploys this example nozzle to the Firehose of a Cloud Foundry installation deployed locally with BOSH Lite. For more information about BOSH Lite, see the [BOSH Lite GitHub repository](
To lighten the load on custom nozzles you develop, you can request Firehose subscriptions that emit only whitelisted metrics. For examples, see `rlpreader` and `rlptypereader` in the [loggregator-tools]( repository, and see the _V2 Subscriptions_ page of the [loggregator-release]( repository for more information.
## Prerequisites
- [BOSH CLI]( installed locally.
- Spiff installed locally and added to the load path of your shell. See [Spiff on GitHub](
- BOSH Lite deployed locally using VirtualBox. See [BOSH Lite on GitHub](
- A working [Cloud Foundry]( deployment, including Loggregator, deployed with your local BOSH Lite. This serves as our source of data. See [Deploying Cloud Foundry using BOSH Lite](, or use the `provision_cf` script included in the [BOSH Lite release](
<p class='note'><strong>Note</strong>: Deploying Cloud Foundry can take up to several hours, depending on your internet bandwith, even when using the automated <code>provision_cf</code> script.</p>
## <a id='dl-bosh-manifest'></a>Step 1: Download Cloud Foundry BOSH Manifest ##
<%= partial 'download_bosh_manifest' %>
## <a id='add-client'></a>Step 2: Add UAA Client ##
You must authorize the example nozzle as a UAA client for your CF deployment. To do this, add an entry for the example nozzle as `client` for `uaa` under the `properties` key in your CF deployment manifest. You must enter the example nozzle object in the correct location in the manifest, and with the correct indentation, as described below.
<p class='note'>Deployment manifests are YAML files. Visit <a href=''>YAML</a> to learn about YAML syntax.
1. Open the deployment manifest in a text editor.
1. Locate the left-aligned `properties` key.
1. Under the `properties` key, locate `uaa` at the next level of indentation.
1. Under the `uaa` key, locate the `clients` key at the next level of indentation.
1. Enter properties for the `example-nozzle` at the next level of indentation, exactly as shown below. The `...` in the text below indicate other properties that may populate the manifest at each level in the hierarchy.
access-token-validity: 1209600
authorized-grant-types: client_credentials
override: true
secret: example-nozzle
authorities: oauth.login,doppler.firehose
1. Save the deployment manifest file.
## <a id='redeploy'></a>Step 3: Redeploy Cloud Foundry ##
Run `bosh -e MY-ENV deploy` to deploy Cloud Foundry with BOSH. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'cf-warden' on 'Bosh Lite Director'
Getting deployment properties from director...<br>
Detecting deployment changes
No changes<br>
No changes<br>
No changes<br>
Resource pools
No changes<br>
Disk pools
No changes<br>
No changes<br>
No changes<br>
+ access-token-validity: 1209600
+ authorized-grant-types: authorization_code,client_credentials,refresh_token
+ override: true
+ secret: example-nozzle
+ scope: openid,oauth.approvals,doppler.firehose
+ authorities: oauth.login,doppler.firehose<br>
No changes<br>
Please review all changes carefully<br>
Are you sure you want to deploy? (type 'yes' to continue):yes
## <a id='dl-release'></a>Step 4: Clone Example Release ##
The Cloud Foundry Loggregator team created an example nozzle application for
use with this tutorial.
1. Run `git clone` to clone the main release repository from [GitHub](
<pre class='terminal'>
$ git clone <span>git@</span>
Cloning into 'example-nozzle-release'...
1. Navigate to the `example-nozzle-release` directory.
<pre class='terminal'>
$ cd example-nozzle-release
1. Run `git submodule update --init --recursive` to update all of the included submodules.
<pre class='terminal'>
$ git submodule update --init --recursive
Submodule 'src/' (<span>git@</span> registered for path 'src/'
Submodule 'src/' (<span>git@</span> registered for path 'src/'
Cloning into 'src/'...
## <a id='step-id'></a>Step 5: Prepare Nozzle Manifest ##
Complete the following steps to prepare the nozzle deployment manifest:
1. In the `example-nozzle-release` directory, navigate to the `templates` directory.
<pre class='terminal'>
$ cd templates
Within this directory, examine the two YAML files.
`bosh-lite-stub.yml` contains the values used to populate the missing
information in `template.yml`.
By combining these two files we create a deployment manifest for our nozzle.
1. Create a `tmp` directory for the compiled manifest.
1. Use [Spiff]( to compile a deployment manifest from the template and stub, and save this manifest.
<pre class='terminal'>
$ spiff merge templates/template.yml templates/bosh-lite-stub.yml > tmp/manifest_bosh_lite.yml
1. Run `bosh -e MY-ENV deployments` to identify the names of all deployments in the environment you specify. Replace `MY-ENV` with the alias you set for your BOSH Director.
1. Run `bosh -e MY-ENV env --column=uuid` to obtain your BOSH Director UUID. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev env --column=uuid
1. In the compiled nozzle deployment manifest, locate the `director_uuid` property. Replace `PLACEHOLDER-DIRECTOR-UUID` with your BOSH Director UUID.
name: default
network: example-nozzle-net
reuse_compilation_vms: true
workers: 1
director_uuid: PLACEHOLDER-DIRECTOR-UUID # replace this
<p class='note'><strong>Note</strong>: If you do not want to see the complete deployment procedure, run the following command to automatically prepare the manifest: <code>scripts/make_manifest_spiff_bosh_lite</code>
## <a id='create'></a>Step 6: Create Nozzle BOSH Release ##
Use the `bosh -e MY-ENV create-release --name RELEASE-NAME` command to create a BOSH release. Replace `MY-ENV` with the alias you set for your BOSH Director. Replace RELEASE-NAME with `example-nozzle` to match the [UAA client](#add-client) that you created in the CF deployment manifest. For example:
<pre class='terminal'>
$ bosh -e dev create-release --name example-nozzle
Syncing blobs...
## <a id='upload'></a>Step 7: Upload Nozzle BOSH Release ##
Run `bosh -e MY-ENV upload-release` to upload the release that you created in [Step 6: Create Nozzle BOSH Release](#create). Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev upload-release
Acting as user 'admin' on 'Bosh Lite Director'<br>
Copying packages
Copying jobs
Generated /var/folders/4n/qs1rjbmd1c5gfb78m3_06j6r0000gn/T/d20151009-71219-17a5m49/d20151009-71219-rts928/release.tgz
Release size: 59.2M<br>
Verifying release...
Release info
Name: nozzle-test
Version: 0+dev.2<br>
- example-nozzle (b0944f95eb5a332e9be2adfb4db1bc88f9755894)
- golang1.7 (b68dc9557ef296cb21e577c31ba97e2584a5154b)<br>
- example-nozzle (112e01c6ee91e8b268a42239e58e8e18e0360f58)<br>
- none<br>
Uploading release
## <a id='deploy'></a>Step 8: Deploy Nozzle ##
Run `bosh -e MY-ENV deploy` to deploy the nozzle. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev deploy
Acting as user 'admin' on deployment 'example-nozzle-lite' on 'Bosh Lite Director'
Getting deployment properties from director...
Unable to get properties list from director, trying without it...
Cannot get current deployment information from director, possibly a new deployment
Please review all changes carefully<br>
Are you sure you want to deploy? (type 'yes' to continue):yes
## <a id='view'></a>Step 9: View Nozzle Output ##
The example nozzle outputs all of the data originating coming from the Firehose to its log files. To view this data, SSH into the example-nozzle VM and examine the logs.
1. Run `bosh -e MY-ENV ssh` to access the nozzle VM at the IP configured in the nozzle's manifest template stub `./templates/bosh-lite-stub.yml`. Replace `MY-ENV` with the alias you set for your BOSH Director. For example:
<pre class='terminal'>
$ bosh -e dev ssh example-nozzle
Welcome to Ubuntu 14.04.1 LTS (GNU/Linux 3.19.0-25-generic x86_64)
Documentation: <span>https<span>://
Last login: Wed Sep 23 21:29:50 2015 from
1. Use the `cat` command to output the `stdout` log file.
<pre class='terminal'>
$ cat /var/vcap/sys/log/example-nozzle/example-nozzle.stdout.log
===== Streaming Firehose (will only succeed if you have admin credentials)
origin:"DopplerServer" eventType:ValueMetric timestamp:1443046217700750747 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"messageRouter.numberOfFirehoseSinks" value:1 unit:"sinks" >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910193187 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"dropsondeAgentListener.receivedByteCount" delta:771 total:386368 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910360012 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"dropsondeMarshaller.counterEventMarshalled" delta:8 total:2619 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910252169 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"dropsondeUnmarshaller.valueMetricReceived" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910294255 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"MessageAggregator.uncategorizedEvents" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910318582 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"dropsondeMarshaller.valueMetricMarshalled" delta:9 total:4525 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910339088 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"MessageAggregator.counterEventReceived" delta:8 total:2619 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910379199 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"DopplerForwarder.sentMessages" delta:17 total:7082 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046218910394886 deployment:"cf-warden" job:"loggregator_trafficcontroller_z1" index:"0" ip:"" counterEvent:<name:"dropsondeAgentListener.receivedMessageCount" delta:9 total:4525 >
origin:"router__0" eventType:HttpStartStop timestamp:1443046219105062148 deployment:"cf-warden" job:"router_z1" index:"0" ip:"" httpStartStop:<startTimestamp:1443046219096144435 stopTimestamp:1443046219105062148 requestId:<low:7224371029360401779 high:6477133057804225906 > peerType:Client method:POST uri:"" remoteAddress:"" userAgent:"" statusCode:200 contentLength:-1 >
origin:"api_z1_0" eventType:HttpStartStop timestamp:1443046219109842455 deployment:"cf-warden" job:"api_z1" index:"0" ip:"" httpStartStop:<startTimestamp:1443046219106687524 stopTimestamp:1443046219109842455 requestId:<low:7948515536237261025 high:3934284340053050957 > peerType:Server method:GET uri:"" remoteAddress:"" userAgent:"Go 1.1 package http" statusCode:200 contentLength:108 >
origin:"router__0" eventType:HttpStartStop timestamp:1443046219110064368 deployment:"cf-warden" job:"router_z1" index:"0" ip:"" httpStartStop:<startTimestamp:1443046219106025264 stopTimestamp:1443046219110064368 requestId:<low:7948515536237261025 high:3934284340053050957 > peerType:Client method:GET uri:"" remoteAddress:"" userAgent:"" statusCode:200 contentLength:108 >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177165446 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"logSenderTotalMessagesRead" value:0 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177288325 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"numGoRoutines" value:22 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177346726 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.numBytesAllocated" value:347992 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177274975 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"numCPUS" value:4 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177310389 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.numBytesAllocatedHeap" value:347992 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177330214 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.numBytesAllocatedStack" value:393216 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177353454 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.numMallocs" value:312614 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177360052 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.numFrees" value:309258 unit:"count" >
origin:"syslog_drain_binder" eventType:ValueMetric timestamp:1443046219177481456 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" valueMetric:<name:"memoryStats.lastGCPauseTimeNS" value:318837 unit:"count" >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880585603 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"dropsondeListener.receivedMessageCount" delta:59 total:61634 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880895040 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"dropsondeListener.receivedByteCount" delta:9462 total:9111082 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881017888 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"dropsondeUnmarshaller.httpStartStopReceived" delta:7 total:2191 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881011670 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"dropsondeUnmarshaller.valueMetricReceived" delta:28 total:35309 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929574 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"dropsondeUnmarshaller.counterEventReceived" delta:24 total:24134 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219881004417 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"httpServer.receivedMessages" delta:59 total:61634 >
origin:"DopplerServer" eventType:CounterEvent timestamp:1443046219880929568 deployment:"cf-warden" job:"doppler_z1" index:"0" ip:"" counterEvent:<name:"signatureVerifier.validSignatures" delta:59 total:61634 >
origin:"MetronAgent" eventType:CounterEvent timestamp:1443046220058280679 deployment:"cf-warden" job:"api_z1" index:"0" ip:"" counterEvent:<name:"MessageAggregator.uncategorizedEvents" delta:44 total:12891 >