Automatic submission of system metrics to graphite, for golang applications
Switch branches/tags
Nothing to show
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
collector
grafana
LICENSE
README.md
runstats.go

README.md

golang-metrics

This is a simple library to automatically submit metrics to a remote graphite/carbon server.

Using this library is as simple as adding the following import to your application:

_ "github.com/skx/golang-metrics"

Importing the library is all that you need to do, because the init() function will launch a goroutine to collect and submit metrics.

Sample Application

The following example shows how little effort is required. As documented above the import carries out the setup and launches the metric-collection automatically.

The following program just prints a message, once a second, but due to the magical import it can also collect and submit metrics:

  package main

  import (
    "fmt"
    "time"

    _ "github.com/skx/golang-metrics"
  )

  func main() {
    for {
	   fmt.Printf("This process is alive, and submitting metrics!\n")
	   time.Sleep(1 * time.Second)
    }
  }

Configuration

There is only one thing you must configure, the address of the host to submit your metrics to. The environmental variable METRICS will be used for that purpose.

  • If there is no METRICS variable defined then no metrics will be collected.
    • Because collecting things is pointless without somewhere to send them.

Assuming you've saved the example included earlier in ~/main.go you can start it like so:

 $ METRICS=metrics.example.com go run ./main.go

The metrics will be submitted to the host metrics.example.com, and each metric will prefixed by the basename of your binary, and the prefix go. i.e:

  • go.$(basename argv[0]).*

As a concrete example the binary located at /opt/overseer/bin/overseer would submit metrics with names like these:

  • go.overseer.cpu.cgo_calls
  • go.overseer.cpu.goroutines
  • go.overseer.mem.alloc
  • ..
  • go.overseer.uptime

If your host runs the submission agent upon a non-standard port you can specify that as you would expect:

 $ METRICS=metrics.example.com:2233 ./application

Advanced Configuration

The only mandatory environmental-variable is METRICS, however there are a couple more environmental variables you might wish to set to tweak behaviour:

Setting Purpose
METRICS_DELAY If this is set to an integer then it will be used to control how often metrics are collected and sent. The default is 10 meaning metrics will be submitted every ten seconds.
METRICS_PREFIX If this is set then all metrics will have the specified prefix. (Remember to add the trailing period!)
METRICS_PROTOCOL If this is set to tcp then TCP will be used, instead of UDP updates.
METRICS_VERBOSE If this is non-empty metrics will be echoed to STDOUT.

Meaning of Metrics

The meaning of the submitted metrics should be obvious, but if you need a reference you can consult the documentation for the MemStats structure:

Grafana Integration

You can find a sample grafana dashboard in this repository, which can be imported to your installation. That will use the prefix go.* to automatically create a drop-down of all your available applications.

The obligatory screenshot can be viewed here:

Systemd

If you're launching your application under the control of systemd you can configure the metrics target in your .service file by adding an Environment setting. For example:

 [Unit]
 Description=My service..

 [Service]
 WorkingDirectory=/srv/blah
 User=blah
 Environment="METRICS=metrics.example.com:2003"
 Environment="METRICS_DELAY=20"
 ExecStart=/srv/blah/bin/application arugments
 Restart=always
 StartLimitInterval=2
 StartLimitBurst=20
 PrivateTmp=yes
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

Adding multiple Environment=".." lines will do the right thing. If you want to set multiple environmental variables you should probably use the Systemd EnvironmentFile setting instead:

 [Unit]
 Description=My service..

 [Service]
 WorkingDirectory=/srv/blah
 User=blah
 EnvironmentFile=-/etc/default/my-service
 ExecStart=/srv/blah/bin/application arugments
 Restart=always
 StartLimitInterval=2
 StartLimitBurst=20
 PrivateTmp=yes
 RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

(Here the "-" prefix to the path means that no error messages are generated if the specified environment file does not exist.)

Licensing / Credits

The metric-collection code was copied from the library written by Brian Hatfield, which is available here:

That code was licensed under the MIT license, which has been included in this repository.