Skip to content

Usage

Jump to bottom
Ryan Smith edited this page Aug 15, 2013 · 26 revisions

This page contains information regarding the usage of l2met. Usage includes how to drain an apps logs into l2met and what sort of data can be extracted from the logs. If you are interested in how l2met works, checkout the architecture page. If you are interested in the system administration of l2met, checkout the administration page.

L2met has two basic inputs: the drain URL and the log line. The log lines are delivered to the drain URL over HTTPs. Thus, there are 3 topics related to l2met usage:

Logging Convention

Logging is defined to be a stream of bytes emitted by a process to STDOUT. Ultimately, l2met expects the log lines to be delivered in RFC5424 frames, but understanding framing is not necessary in understanding the logging convention. More details about framing can be found in the section on Log Delivery. Thus, the following ruby snippet is an example of a log:

$stdout.puts("hello world")

As an application developer, this is your interface into l2met. Simply write a string to STDOUT, and you will be producing metrics in no time. Here is an example of a ruby process emitting a counter metric:

$stdout.puts("count#bottles-of-beer-on-the-wall=1")

In the string printed to STDOUT, you will notice a key=value pair. The key contains a metric type, followed by a # followed by the metric name. The value simply contains a number. Optionally you can attach a unit to the number. E.g.

$stdout.puts("count#beers-on-the-wall=1bottle")

There are 3 metrics types: counters, samples, and measurements.

# A counter.
$stdout.puts("count#user.clicks=4")
# A Sample.
$stdout.puts("sample#database.size=40.9MB")
# A Measurement.
$stdout.puts("measure#database.query=200ms")

You can combine multiple metrics into a single line:

$stdout.puts("count#signups=1 measure#signup-controller=50ms sample#queue.length=99")

The value of metrics will parsed into a 64bit floating point number. The unit must immediately follow the number and must only include a-zA-Z.

Log Convention to Function Map

  • count# => sum()
  • sample# => last()
  • measure# => min(), median(), p95(), p99(), max(), mean(), sum(), count()

Source

Finally, there is one last keyword to address. The source keyword. This is handy for distinguishing machines or environments. You can associate a set of metrics with a source by adding a source key=value pair to the log line. E.g.

$stdout.puts("source=us-east measure#web.latency=4ms")

The source value will also be passed along to the Librato API.

The Heroku Router

Heroku's router does not follow the l2met logging convention. However, l2met has a builtin parser that will compute the following metrics on the router's logs. Basically, the router log lines are transformed into:

"measure#service=4ms measure#connect=2ms measure#bytes=5"

Legacy Logging Convention

  • 0.X.X measure=name val=x No longer supported.
  • 1.X.X, 2.0.beta1 measure.name=val measure.name1=val1 Deprecated. Will be removed in 3.X.X

Drain URLs

Log lines are delivered to l2met via HTTPs. Thus, you will need a URL in order to send your log stream to l2met. The address url of your l2met server is known as the Drain URL. In addition to holding the host of the l2met server, the Drain URL also contains:

  • Your Librato API credentials. (Signed and encrypted)
  • The resolution at which metrics sent to the URL are to be aggregated. (e.g. 1s 5s 10s 60s)
  • A source prefix which will be prepended to all source keys on your logs lines.
  • A name prefix which will be prepended to all metric names on your log lines.

Encrypted Librato Credentials

L2met does not permanently store any Librato credentials. Therefore, when setting up your Drain URL, you will need to include your encrypted credentials in the user portion of the URL. You can use l2met's sign api to encrypt the Librato credentials. To do so, you will need access to the $SECRETS variable which is set on the l2met server. Here is an example curl(1) command to sign a set of Librato credentials:

$ export libratouser=r@32k.io
$ export libratopass=abc123
$ export secret=def456 # must be set as the $SECRETS env var on your l2met server
$ curl "https://my-l2met.net/sign" --data "$libratouser:$libratopass" -u "$secret:"
gAAAAABR-UAfqpquaZLK77ROMwL3dHXtuHC8jzTVRc6LLsPIPn9Vbe5yVqZ2aRjfyt82KH7mn9iEkah8tO4udeyDH8bguepTebR_tOqnO0bwQuCyIjG-hABav8NmxbrKPGsq5FvdL3ZjdyHr2ewjEWQRJIEKs8vqNlUrBRQKSLqxbtBHiVmWwmM=

The value that is returned from the /sign api is your signed & encrypted Librato credentials. You place them in the user portion of the Drain URL. E.g.

"https://gAAAAABR-UAfqpquaZLK77ROMwL3dHXtuHC8jzTVRc6LLsPIPn9Vbe5yVqZ2aRjfyt82KH7mn9iEkah8tO4udeyDH8bguepTebR_tOqnO0bwQuCyIjG-hABav8NmxbrKPGsq5FvdL3ZjdyHr2ewjEWQRJIEKs8vqNlUrBRQKSLqxbtBHiVmWwmM=@my-l2met.net/logs"

Drain Resolution

You can configure the resolution of your metrics on a per drain basis. For instance, on one app you may wish to have 1s resolution. To do so, add the following query parameter to your Drain URL. ?resolution=1 E.g.

"https://ENCRYPTED-CREDS@my-l2met.net/logs?resolution=1"

A single l2met server can handle different drains with different resolutions.

Prefix

It is handy to apply a prefix to all metrics on both the source and the metric-name level. You can do this by using the following query parameters:

"https://ENCRYPTED-CREDS@my-l2met.net/logs?source-prefix=aws&prefix=v0.1"

Notice that the parameter prefix=v0.1 prefixes the metric name and the parameter source-prefix=aws prefixes the source.

Log Delivery

As previously mentioned, logs are delivered to l2met via HTTPs. This means that logs are easily authenticated and encrypted. Thus suitable for cloud installations. If you are draining your Heroku apps into l2met, the delivery is handled on your behalf. All you must do is add a drain to your heroku app. E.g.

$ heroku drains:add https://ENCRYPTED-CREDS@my-l2met.net/logs -a my-app

L2met accepts POST requests to /logs and will read the body of the request looking for length prefixed, RFC5425 compatible logs lines. If you are running your own infrastructure, you can use log-shuttle to connect your app's STDOUT to l2met. Log-shuttle will read an app's stdout, batch log lines, format them using the RFC5424 protocol, and deliver them via HTTPs. Heroku uses log-shuttle under the hood.

The following curl(1) command illustrates how logs are expected to be delivered to l2met:

$ curl "https://long-token@my-l2met.net/logs" --data "94 <190>1 2013-03-27T20:02:24+00:00 hostname token shuttle - - measure#hello=99 measure#world=100"
Clone this wiki locally