Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
172 lines (107 sloc) 11 KB
title: Overview of the Loggregator System
owner: Logging and Metrics
<strong><%= modified_date %></strong>
Loggregator gathers and streams logs and metrics from user apps in a <%= vars.first_product_name %> deployment as well as metrics from <%= vars.product_name %> components. For more information, see the [Loggregator repository on GitHub](
## <a id='use-cases'></a> Using Loggregator
The primary use cases for Loggregator include the following:
- App developers can tail their application logs or dump the recent logs from the Cloud Foundry Command Line Interface (cf CLI), or stream these to a third-party log archive and analysis service.
- Operators and administrators can access the **Loggregator [Firehose](#firehose)**, the combined stream of logs from all apps, and the metrics data from Cloud Foundry components.
- Operators can deploy **nozzles** to the Firehose. A nozzle is a component that monitors the Firehose for specified events and metrics, and streams this data to external services.
## <a id='components'></a> Loggregator Architecture
The diagram below shows the architecture of Loggregator, including the Cloud Foundry components that it interacts with.
<% if vars.product_name == 'CF' %>
<%= image_tag 'architecture/loggregator.png' %>
<% else %>
<%= image_tag 'architecture/loggregatornew.png' %>
<% end %>
<% if vars.product_name == 'CF' %>
<a href="./images/architecture/loggregator.png">View a larger version of this image.</a>
<% else %>
<a href="./images/architecture/loggregatornew.png">View a larger version of this image.</a>
<% end %>
<p class="note"><strong>Note</strong>: The Loggregator system uses <a href="">gRPC</a> for communication between the Metron Agent and the Doppler, and between the Doppler and the Traffic Controller. This improves the stability and the performance of the Loggregator system, but it may require operators to scale their Dopplers.</p>
### <a name='sources'></a>Source
Sources are logging agents that run on the Cloud Foundry components.
### <a name='metron'></a>Metron
Metron Agents are colocated with sources. They collect logs and forward them to the Doppler servers.
### <a name='doppler'></a>Doppler
Dopplers gather logs from the Metron Agents, store them in temporary buffers, and forward them to the Traffic Controller or to third-party syslog drains.
### <a name='traffic-controller'></a>Traffic Controller
The Traffic Controller handles client requests for logs. It gathers and collates messages from all Doppler servers, and provides external API and message translation as needed for legacy APIs. The Traffic Controller
also exposes the Firehose.
### <a name='firehose'></a>Firehose
The Firehose is a WebSocket endpoint that streams all the event data coming from a Cloud Foundry deployment. The data stream includes logs, HTTP events, and container metrics from all applications, and metrics from all Cloud Foundry system components. Logs from system components such as the Cloud Controller are not included in the Firehose and are typically accessed through rsyslog configuration.
Because the data coming from the Firehose may contain sensitive information, such as customer information in the application logs, only users with the correct permissions can access the Firehose.
The Traffic Controller serves the Firehose over WebSocket at the `/firehose` endpoint. The events coming out of the Firehose are formatted as protobuf messages conforming to the [dropsonde protocol](
You can discover the address of the Traffic Controller by hitting the `info` endpoint on the API and retrieving the value of the `doppler_logging_endpoint`.
Example for a BOSH Lite CF environment:
<pre class='terminal'>
$ cf curl /v2/info | jq .doppler\_logging\_endpoint
#### <a name='logs-vs-metrics'></a>Difference Between Logs and Metrics
The Firehose carries both logs and metrics, which differ as follows:
* **Metrics**
- Report a measurement from a VM, or the status of a VM, at its timestamp time
- Follow [dropsonde]( or [statsd]( protocol
- Can route to a monitoring platform to trigger alerts
* **Logs**
- Report events detected, actions taken, errors, or any other messages the operator or developer wanted to generate
- Follow the syslog standard
- Are not used to trigger alerts
<% if vars.product_name == 'CF' %>
<% else %>
### <a name='syslog'></a> CF Syslog Drain
This section describes the CF Syslog Drain components of Loggregator.
* **What CF Syslog Drain does**:
* Loggregator uses the [CF Syslog Drain Release]( to support developers who want to stream app logs to a syslog-compatible aggregation or analytics service. See [Streaming Application Logs to Log Management Services](../devguide/services/log-management.html#user-provided).
* **What CF Syslog Drain consists of**:
* CF Syslog Drain includes the Reverse Log Proxy (RLP) and Syslog Adapter components described below. These components run on VMs deployed with Pivotal Application Service (PAS) that you can scale independently to support large numbers of user-provided syslog drains.
* **How you can scale**:
* See [Configuring System Logging in PAS](../opsguide/logging-config-opsman.html#scaling).
* **When to scale**:
* For guidance on scaling, see the [CF Syslog Drain Performance Scaling Indicators](../monitoring/key-cap-scaling.html#scalable-syslog) section of the _Monitoring Pivotal Cloud Foundry_ guide.
#### <a name='rlp'></a> Reverse Log Proxy (RLP)
RLPs are a BOSH jobs colocated with the Traffic Controller that collect logs from Dopplers and forward them to Syslog Adapters. You can scale this component based on your overall log volume.
#### <a name='adapter'></a> Syslog Adapter
Syslog Adapters are BOSH VMs that manage connections with and write to syslog services, or drains. You can scale this component based on the number of drains. For more information about Syslog Adapter capacity planning, see [Scaling Loggregator](log-ops-guide.html#scaling).
<% end %>
## <a id='other-components'></a> Related Components
This section provides information about the components that are related to the Loggregator system.
<% if vars.product_name == 'CF' %>
<% else %>
### <a name='bosh-metrics'></a> BOSH System Metrics Forwarder
PCF uses the following components to send BOSH-reported component metrics to Loggregator:
* The **BOSH System Metrics Plugin** is deployed on the BOSH Director. This plugin reads health events such as VM heartbeats and alerts from the BOSH Health Monitor JSON plugin and streams them to the BOSH System Metrics Server.
* The **BOSH System Metrics Server** is deployed on the BOSH Director. The server accepts connections from the BOSH System Metrics Forwarder and streams the health events to it over gRPC as follows:
* If two clients connect to the BOSH System Metrics Server using the same subscription ID, the server evenly distributes the event stream between them.
* If two clients connect to the BOSH System Metrics Server using different subscription IDs, each client receives a copy of the event stream.
* The **BOSH System Metrics Forwarder** is colocated on the Traffic Controller. It initiates connections to the BOSH System Metrics Server and receives alerts and heartbeats over secure gRPC. The BOSH System Metrics Forwarder sends heartbeat events as envelopes to Loggregator through a colocated Metron Agent. It does not forward alerts.
<% end %>
### <a name='nozzles'></a> Nozzles
Nozzles are programs which consume data from the Loggregator Firehose. Nozzles can be configured to select, buffer, and transform data, and forward it to other applications and services. Example nozzles include the following:
- The JMX Bridge OpenTSDB Firehose Nozzle, which installs with [JMX Bridge](
- The [Datadog nozzle](, which publishes metrics coming from the Firehose to Datadog
- The [Syslog nozzle](, which filters out log messages coming from the Firehose and sends it to a syslog server
For more information about nozzles, see the [Nozzle Tutorial](./nozzle-tutorial.html).
### <a name="autoscaler"></a> App Autoscaler
App Autoscaler allows you to configure rules that balance the performance and cost of apps by scaling them.
App Autoscaler relies on API endpoints from Loggregator's Log Cache. If you disable Log Cache, App Autoscaler will fail.
For more information, see [App Autoscaler Fails When Loggregator’s Log Cache Is Disabled](
### <a id="log-cache"></a> Log Cache
Log Cache is a Loggregator feature that allows you to view data from the firehose over a specified period of time. It provides an in-memory caching layer and a RESTful interface for retrieving logs.
To use Log Cache, you must enable it in the Pivotal Application Service (PAS) UI. Enable it by following the procedure in [Enable Log Cache](../../opsguide/logging-config-opsman.html#log-cache).
After you enable Log Cache, you can use the API endpoints or CLI plugin to query and filter app logs. Log Cache's API endpoints are available by default. Visit the Cloud Foundry CF CLI Plugins page to [download the Log Cache CLI plugin](
#### <a id="scaling-log-cache"></a> Scaling Log Cache
Log Cache duplicates the data stream available from the firehose, but allows you to specify a smaller period of time. Cached app logs are available on demand; you do not need to stream them continuously. Log Cache returns logs in increments of up to 1000 envelopes per request.
Log Cache is colocated on the Doppler VM. Add more VMs to scale Log Cache. You can scale Log Cache horizontally based on the following formula:
`Log Cache nodes = Envelopes per second / 10,000`
When scaled according to this formula, Log Cache provides logs for the last 15 minutes of event data, with the exception of redeployment. On redeploy, all logs are refreshed. Log Cache cannot display logs from previous deployments.
#### Component Dependencies
Log Cache is closely related to these components in the logging and metrics ecosystem:
* Log Cache depends on **Loggregator** to view and filter logs, so its reliability is dependent on Loggregator's performance. Log Cache uses the available memory on a device to store logs, and so may impact device performance during periods of high memory contention.
* Log Cache is colocated on the **Doppler VMs**. It speeds up the retrieval of data from the Loggregator system, especially for deployments with a large number of Dopplers.
* **App Autoscaler** relies on API endpoints from Loggregator’s Log Cache. If you disable Log Cache, App Autoscaler will fail.
For more information, see [Loggregator Adds Log Cache in PAS Advanced Features](../../pcf-release-notes/runtime-rn.html#log-cache).