Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Suck up metrics from various sources and spit them into Graphite
branch: master

This branch is 61 commits behind vmetrix:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
bin
conf
lib
LICENSE
README.md

README.md

vacuumetrix

Sucks up metrics from various external sources and puts the data into internal systems. This is a good thing to do because whilst all the external SAAS services we use are fantastic, it's also sometimes very useful to be able to be able to view and manipulate the data in one place. For example, we might want to make a single graph with some internally generated application metrics, with some perfomance metrics from New Relic, with some metrics from an AWS Elastic Load Balancer. Perhaps. Anyway, vacuumetrix is a collection of ruby scripts that talk to various APIs and allows the collection of this data in one place.

Currently supported Inputs

  • New Relic
  • AWS Cloudwatch (some services)
  • Neustar Web Performance Management
  • facebook "likes"
  • twitter "followers"

Currently supported Outputs

  • Graphite
  • Ganglia
  • OpenTSDB

Installation

git clone this repo

git clone https://github.com/99designs/vacuumetrix.git  

Install dependancies

Debian/Ubuntu

apt-get install ruby build-essential libcurl3 libcurl3-gnutls libcurl4-openssl-dev

Ruby Gems

gem install json socket

If you're outputting to ganglia gmond you'll also need to

gem install gmetric

Configuration

The config.rb file contains all the local configuration variables.

cd conf 
cp config.rb-sample config.rb

New Relic

update config.rb with your organization's API key and account number

gem install curb xml-simple

AWS

update config.rb with your organization's AWSAccessKeyId and AWSSecretKey with permission to read the Cloudwatch API

gem install fog

Neustar

update config.rb with your organization's API Key and Secret

gem install curb time

Twitter

gem install curb

Output

your Graphite server your Ganglia-gmond server

(Note: If you haven't already done so you will need to activate the New Relic REST API. See here: http://blog.newrelic.com/2011/06/20/new-data-api/

Running the scripts

The scripts are in the bin directory and are designed to be run from cron every minute.

cron:

*   *   *   *   *   /opt/vacuumetrix/bin/NewrelicEnduser.rb 123 metricyouwant

NewrelicEnduser.rb

Get New Relic End User (RUM) stats. Supply two args, app and metric.

NewrelicThresholds.rb

Get the threshhold values for all your applications. This includes average RAM, CPU, DB etc.

AWScloudwatchEBS.rb

Get EBS metrics. No arguments. Run every 5 minutes. No point running more frequently.

AWScloudwatchELB.rb

Get Elastic Load Balancer metrics. Supply the name of the ELB.

AWScloudwatchRDS.rb

Get RDS metrics. Supply the name of the Relational Database Service instance. (Tested with MySQL). YMMV.

AWScountEC2.rb

Count the number of EC2 instances of each flavor. No arguments.

AWScloudwatchElasticache.rb

Get Elasticache metrics. Interesting ones anyway. No arguments.

Neustar.rb

Get Neustar Web Performance Metrics. For each monitor get duration and status for each monitored location. No arguments. Run every 5 minutes. (Not very efficient at the moment. Needs some tuning).

facebook.rb

Argument is the name of the page you want to check the like count of.

twitter.rb

Argument is the name of the twitter user you want to check the number of followers of.

TODO

Suck in

  • Other AWS metrics
  • Google Analytics

Spit out

  • Statsd

Pull requests are appreciated.

Something went wrong with that request. Please try again.