Cloud Native Logging
Go Shell HTML Other
Latest commit a525258 Aug 17, 2017 @enocom enocom Allow TC to discover dopplers via a bosh property
[#150115473]

Signed-off-by: Jason Keene <jkeene@pivotal.io>

README.md

Cloud Native Logging slack.cloudfoundry.org CI Badge

Loggregator is a BOSH release deployed as a part of cf-release. Loggregator provides a highly-available (HA) and secure stream of logs and metrics for all applications and components on Cloud Foundry. It does so while not disrupting the behavior of the the applications and components on the platform (i.e. "backpressure").

The Loggregator Design Notes presents an overview of Loggregator components and architecture.

Generating TLS Certificates

In order to secure transport of logs throughout the system, Loggregator needs several certificates to be generated for each of the components. You start this process by running the scripts/generate-certs, and then configure the corresponding certificates for each of the components you are deploying. For more details see our Certificate Configuration README

Streaming Application Logs

Any user of Cloud Foundry can experience Loggregator by using two simple interfaces for streaming application specific logs. These do not require any special User Account and Authentication(UAA) Scope.

Using the CF CLI

The fastest way to see your logs is by running the cf logs command using the CF CLI. Check the Cloud Foundry CLI docs for more details.

Forwarding to a Log Drain

If you’d like to save all logs for an application in a third party or custom tool that expects the syslog format, a log drain allows you to do so. Check the Cloud Foundry docs for more details.

Consuming the Firehose

The firehose is an aggregated stream of all application logs and component metrics on the platform. This allows operators to ensure they capture all logs within a microservice architecture as well as monitor the health of their platform. See the Firehose README.

User Account and Authentication Scope

In order to consume the firehose you’ll need the doppler.firehose scope from UAA. For more details see the Firehose README.

Nozzle Development

Once you have configured appropriate authentication scope you are ready to start developing a nozzle for the firehose. See our Nozzle community page for more details about existing nozzles and how to get started.

Metrics

Loggregator and other Cloud Foundry components emit regular messages through the Firehose that monitor the health, throughput, and details of a component's operations. For more detials about Loggregator’s metrics see our Loggregator Metrics README.

Emitting Logs and Metrics into Loggregator

For components of Cloud Foundry or standalone BOSH deployments, Loggregator provides a set of tools for emitting Logs and Metrics.

Loggregator API

The Loggregator API is a replacement of the Dropsonde Protocol. Loggregator API defines an envelope structure which packages logs and metrics in a common format for distribution throughout Loggregator. See the Loggregator API README for more details.

Including Metron

The Metron Agent listens on both UDP and gRPC endpoints for multiple versions of Loggregator API which it forwards onto the Firehose. To include Metron in your component or deployment see the Setting up up Metron README.

Statsd-injector

The statsd-injector is a companion component to Metron and allows use of the statsd metric aggregator format. For more see the [statsd-injector README][statsd-injector-readme].

Syslog Release

For some components (such as UAA) it makes sense to route logs separate from the Firehose. The syslog release using rsyslog to accomplish this. For more information see the syslog-release README (note this release is maintianed by the bosh team).

Bosh HM Forwarder

The Bosh Health Monitor Forwarder allows operators to capture health metrics from the Bosh Director through the Firehose. For more information see the bosh-hm-forwarder README.

Tools for Testing and Monitoring Loggregator

Tools

Loggregator provides a set of tools written in golang for testing the performance and efficacy of your loggregator installation. These tools emit logs and metrics in a controlled fashion. For more information see the Loggregator tools README.

Health Nozzle

The Loggregator Team is currently proposing supporting a nozzle which aggregates health metrics across various Loggregator releases to monitor ingress, egress, and loss within the Loggregator system. See the Health Nozzle Feature Proposal for more detials.

More Resources and Documentation

Roadmap

We communicate our long term planning using a Product Road Map, and are always looking to gather feedback and input from Loggregator operators. Get in touch or file an issue if you have feature suggestions you'd like to see added.

Pivotal Tracker

Items marked as "In Flight" on the Roadmap are tracked as new Features in Pivotal Tracker.