Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Update to work with Bundler & Rails 3.0 and make easier to apply #1

Open
wants to merge 29 commits into from

5 participants

@matellis

Hello

Thanks for developing resque-statsd. We had some difficulty using this with Rails 3.0 & Bundler. Specifically, the way we had to use environment variables meant we couldn't simply add extend Resque::Plugins::Statsd in the right places like we do with other extensions because the initializer hadn't run yet.

Our patch doesn't work with Rails 2.x, it would need to be adjusted to try both ENV['RAILS_ENV'] as well as Rails.env.

Thanks

Mat & Jon

MrJoy and others added some commits
@MrJoy MrJoy Changing to lazy-instantiation model, so we can switch to a more robu…
…st configuration model.
c7d0793
@MrJoy MrJoy Adding the gemspec... 6c8ef63
@MrJoy MrJoy Hoisting defaults, adding env-search, fixing stupid. e1b9d27
@MrJoy MrJoy Making config actually work properly.
Using a separate helper class to avoid weirdness with module lifecycle (yes yes, I need to learn how that aspect of Ruby works in more depth.).
1156393
@matellis matellis Updated readme af6e780
@jamster
Owner

Thanks for contributing! I've actually lost my statsd/graphite setup due to a cloud server failure and my lack of time to get it back up, so I haven't messed around with it for quite some time. But, you've given me incentive to get it back up and test all this. Super busy this week, hope to get it in this weekend and I'll happily merge your commits.

Cheers,
Jason

@matellis
@jamster
Owner

Yep, if you have that script handy, that would be great!

@matellis
@jamster
Owner

didn't see an attachment. was there one? (never used puppet before, is it like ruby gems where it will pull from a centralized source?)

@matellis
noahhl and others added some commits
@noahhl noahhl Add a before_perform_statsd method 27a8797
@MrJoy MrJoy Namespace things a bit.
(So if you're tracking something BESIDES Resque jobs in your app, you don't get frustrated...)
145f99f
@noahhl noahhl Add a gemspec 7953b8c
@naseem naseem Logging split by job as well as queue in statsd 0885070
@naseem naseem resque is the default namespace, so removing from module 8614020
@naseem naseem Adding some monkey patching to monitor time spent in queue. d095825
@naseem naseem updating instructions to include require in initializer 2f9a1aa
@naseem naseem typo 0c05d0c
@naseem naseem Making this monkey patch work with a similar one adding functionality…
… to the same Resque.push function

Old code was getting over-written by the last-included module monkey-patch
d1c6ff3
@noahhl noahhl Add a before_perform_statsd method
(Updated to match new queue/job-name split, etc.)

Conflicts:

	lib/resque/plugins/statsd.rb
8e1e256
@MrJoy MrJoy Updating dependencies. 872ae18
@MrJoy MrJoy Tidying up whitespace cruft. b4623a0
@MrJoy MrJoy Merge with '--strategy ours' to hide commits on GH fork view. c233240
@MrJoy MrJoy Tabs to spaces. 36641d7
@MrJoy MrJoy Swallowing errors so statsd issues don't take down your app. b48d536
@MrJoy MrJoy Quarantine ourselves. 0cb40c2
@MrJoy MrJoy Letting dependencies float, and removing explicit bundler dep as that…
…'s just silly.
4f300a3
@MrJoy MrJoy Updating dependencies. 0840e19
@MrJoy MrJoy Freshening gemspec via new Jeweler build. cfb25af
@MrJoy MrJoy Bumping version to reflect all the shiny little tidy-ups happening here. 85a4f43
@MrJoy MrJoy Updating authorship so people using our fork know who to kvetch at. 7e72ae9
@MrJoy MrJoy Tidying up, and skipping a few files that don't belong in the gem. cbc7783
@MrJoy MrJoy While we're at it, what is the point of including the test suite in t…
…he gem?

It just bloats deployments and doesn't actually make it easy/feasible to run the
tests out of the gems anyway.

So, if you wanna run the tests (if/when they ever exist) -- clone the repo.
27ab1ae
@MrJoy MrJoy Updating gemspec. a07359e
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Aug 25, 2011
  1. @MrJoy
  2. @MrJoy

    Adding the gemspec...

    MrJoy authored
  3. @MrJoy
  4. @MrJoy

    Making config actually work properly.

    MrJoy authored
    Using a separate helper class to avoid weirdness with module lifecycle (yes yes, I need to learn how that aspect of Ruby works in more depth.).
  5. @matellis

    Updated readme

    matellis authored
Commits on Oct 6, 2011
  1. @noahhl
Commits on Oct 25, 2011
  1. @MrJoy

    Namespace things a bit.

    MrJoy authored
    (So if you're tracking something BESIDES Resque jobs in your app, you don't get frustrated...)
Commits on Nov 9, 2011
  1. @noahhl

    Add a gemspec

    noahhl authored
Commits on Nov 17, 2011
  1. @naseem
  2. @naseem
Commits on Nov 28, 2011
  1. @naseem
  2. @naseem
  3. @naseem

    typo

    naseem authored
Commits on Nov 29, 2011
  1. @naseem

    Making this monkey patch work with a similar one adding functionality…

    naseem authored
    … to the same Resque.push function
    
    Old code was getting over-written by the last-included module monkey-patch
Commits on Dec 28, 2011
  1. @noahhl @MrJoy

    Add a before_perform_statsd method

    noahhl authored MrJoy committed
    (Updated to match new queue/job-name split, etc.)
    
    Conflicts:
    
    	lib/resque/plugins/statsd.rb
  2. @MrJoy

    Updating dependencies.

    MrJoy authored
  3. @MrJoy

    Tidying up whitespace cruft.

    MrJoy authored
  4. @MrJoy
Commits on Mar 23, 2012
  1. @MrJoy

    Tabs to spaces.

    MrJoy authored
  2. @MrJoy
  3. @MrJoy

    Quarantine ourselves.

    MrJoy authored
  4. @MrJoy
  5. @MrJoy

    Updating dependencies.

    MrJoy authored
  6. @MrJoy
  7. @MrJoy
  8. @MrJoy
  9. @MrJoy
  10. @MrJoy

    While we're at it, what is the point of including the test suite in t…

    MrJoy authored
    …he gem?
    
    It just bloats deployments and doesn't actually make it easy/feasible to run the
    tests out of the gems anyway.
    
    So, if you wanna run the tests (if/when they ever exist) -- clone the repo.
  11. @MrJoy

    Updating gemspec.

    MrJoy authored
This page is out of date. Refresh to see the latest.
View
2  .document
@@ -1,5 +1,5 @@
lib/**/*.rb
bin/*
--
+-
features/**/*.feature
LICENSE.txt
View
2  .gitignore
@@ -14,7 +14,7 @@ doc
# jeweler generated
pkg
-# Have editor/IDE/OS specific files you need to ignore? Consider using a global gitignore:
+# Have editor/IDE/OS specific files you need to ignore? Consider using a global gitignore:
#
# * Create a file at ~/.gitignore
# * Include files you want ignored
View
1  .rvmrc
@@ -0,0 +1 @@
+rvm use ruby-1.9.2-p290@resque-statsd
View
9 Gemfile
@@ -6,11 +6,10 @@ source "http://rubygems.org"
# Add dependencies to develop your gem here.
# Include everything needed to run rake, tests, features, etc.
group :development do
- # gem "rspec", "~> 2.3.0"
- gem "yard", "~> 0.6.0"
- gem "bundler", "~> 1.0.0"
- gem "jeweler", "~> 1.5.2"
+ # gem "rspec"
+ gem "yard"
+ gem "jeweler"
# gem "rcov", ">= 0"
end
-gem 'jamster-statsd'
+gem 'jamster-statsd'
View
17 Gemfile.lock
@@ -3,18 +3,21 @@ GEM
specs:
git (1.2.5)
jamster-statsd (0.1.0)
- jeweler (1.5.2)
- bundler (~> 1.0.0)
+ jeweler (1.8.3)
+ bundler (~> 1.0)
git (>= 1.2.5)
rake
- rake (0.8.7)
- yard (0.6.5)
+ rdoc
+ json (1.6.5)
+ rake (0.9.2.2)
+ rdoc (3.12)
+ json (~> 1.4)
+ yard (0.7.5)
PLATFORMS
ruby
DEPENDENCIES
- bundler (~> 1.0.0)
jamster-statsd
- jeweler (~> 1.5.2)
- yard (~> 0.6.0)
+ jeweler
+ yard
View
51 README.mdown
@@ -3,8 +3,8 @@
Resque Statsd is a Resque plugin that will collect and send data samples from your Resque Jobs. It samples 4 statistics for any Job for which the plugin extends, enqueue count, completion count, failure count, and the time to process. It will monitor for each specific job, in addition to the totals for all jobs (not including timing stats)
## CURRENT STATUS
-
-Tests on the way, but being used in production already
+
+This is a fork of jamster's original gem designed to make it easier to run with Bundler and Rails 3.0.
## Final Outcome
@@ -16,21 +16,40 @@ Tests on the way, but being used in production already
### Install
- gem install resque-statsd
- gem install jamster-statsd # really jnunemaker's version of the statsd rubygem
-
+Add this to your Gemfile:
+
+ gem "resque-statsd", :git => "git://github.com/cloudability/resque-statsd.git"
+
### Rails Setup
-in an initializer, set up the Statsd
+in an initializer, set up the Statsd
+
+ touch config/initializers/resque-statsd.rb
- touch config/initializers/resque-statsd.rb
-
in file add code
- ENV['GRAPHITE_HOST'] = 'graphite.YOURHOST.com'
- ENV['APP_NAME'] = 'YOURAPPNAME_resque' # I add the _resque to separate against web stats
- require 'resque-statsd' # Needs to come after the ENV variable setting
-
+ require 'enqueue_time'
+
+ module Resque
+ include Resque::EnqueueTime
+ end
+
+ StatsdHelper.graphite_host = 'graphite.YOUR_HOST.com'
+ StatsdHelper.namespace = "YOUR_APP.#{Rails.env.development? ? ENV['USER'] : Rails.env}.resque"
+
+Change `YOUR_HOST` and `YOUR_APP` accordingly.
+
+### In Your Code
+
+Extend your class like this:
+
+ class MyJob
+ extend Resque::Plugins::Statsd
+ def self.perform(*payload)
+ # ..
+ end
+ end
+
## Background Reading
### Graphite
@@ -39,7 +58,7 @@ http://graphite.wikidot.com/
### StatsD Comes From Etsy
-The folks at Etsy (kastner) have come up with a neat node.js app to listen for Graphite stats.
+The folks at Etsy (kastner) have come up with a neat node.js app to listen for Graphite stats.
* http://codeascraft.etsy.com/2010/12/08/track-every-release/
* http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-everything/
@@ -47,14 +66,14 @@ The folks at Etsy (kastner) have come up with a neat node.js app to listen for G
### Ruby StatsD Client
-I've required a particular version (jnunemaker's) of the ruby StatsD client
+I've required a particular version (jnunemaker's) of the ruby StatsD client
Read the post here
-
+
* http://railstips.org/blog/archives/2011/03/21/hi-my-name-is-john/
* A Ruby statsd client (https://github.com/etsy/statsd)
## Contributing to resque-statsd
-
+
* Check out the latest master to make sure the feature hasn't been implemented or the bug hasn't been fixed yet
* Check out the issue tracker to make sure someone already hasn't requested it and/or contributed it
* Fork the project
View
26 Rakefile
@@ -11,18 +11,20 @@ require 'rake'
require 'jeweler'
Jeweler::Tasks.new do |gem|
- # gem is a Gem::Specification... see http://docs.rubygems.org/read/chapter/20 for more options
gem.name = "resque-statsd"
- gem.homepage = "http://github.com/jamster/resque-statsd"
+ gem.homepage = "http://github.com/cloudability/resque-statsd"
gem.license = "MIT"
gem.summary = %Q{Adds simple counters and timers for statsd into your Resque jobs}
gem.description = %Q{Will add a counter for enqueuing, performing, failing and timing Jobs}
- gem.email = "jayamster@gmail.com"
- gem.authors = ["Jason Amster"]
- # Include your dependencies below. Runtime dependencies are required when using your gem,
- # and development dependencies are only needed for development (ie running rake tasks, tests, etc)
- # gem.add_runtime_dependency 'jabber4r', '> 0.1'
- # gem.add_development_dependency 'rspec', '> 1.2.3'
+ gem.email = "engineering@cloudability.com"
+ gem.authors = ["Jon Frisby", "Jason Amster"]
+ gem.require_path = 'lib'
+
+ gem.files.reject! do |fn|
+ fn =~ /^\.(rvmrc|rspec|document|gitignore)$/ ||
+ fn =~ /^VERSION$/ ||
+ fn =~ /^(spec|features)\//
+ end
end
Jeweler::RubygemsDotOrgTasks.new
@@ -31,13 +33,13 @@ Jeweler::RubygemsDotOrgTasks.new
# RSpec::Core::RakeTask.new(:spec) do |spec|
# spec.pattern = FileList['spec/**/*_spec.rb']
# end
-#
+#
# RSpec::Core::RakeTask.new(:rcov) do |spec|
# spec.pattern = 'spec/**/*_spec.rb'
# spec.rcov = true
# end
-#
+#
# task :default => :spec
-require 'yard'
-YARD::Rake::YardocTask.new
+# require 'yard'
+# YARD::Rake::YardocTask.new
View
2  VERSION
@@ -1 +1 @@
-0.0.1
+0.0.4
View
49 lib/enqueue_time.rb
@@ -0,0 +1,49 @@
+# Monkey-patch to record queue length since resque doesn't have hooks that give access to the
+# redis stored object, so we can add metadata
+# Ideal solution: contribute to the resque project to add hooks for the push and perform action:
+# https://github.com/defunkt/resque/issues/464
+
+module Resque
+ module EnqueueTime
+
+ def self.included(base)
+ base.class_eval do
+ alias_method :push_without_enqueued_at, :push
+ # Wrapper for the original Resque push method, which adds
+ # enqueued_at time
+ def push(queue, item)
+ begin
+ if item.respond_to?(:[]=)
+ item[:enqueued_at] = Time.now
+ end
+ rescue Exception => e
+ Rails.logger.error "Error in Resque::EnqueueTime: #{e.message}"
+ end
+ push_without_enqueued_at queue, item
+ end
+ end
+ end
+
+ end
+end
+
+
+module Resque
+ class Job
+ alias_method :perform_without_enqueue_time, :perform
+
+ def perform
+ begin
+ if payload['enqueued_at']
+ queue_time = (Time.now - DateTime.parse(payload['enqueued_at'])) * 1000.0
+ StatsdHelper.statsd.timing("queues.#{queue}.queue_time", queue_time)
+ StatsdHelper.statsd.timing("total.queue_time", queue_time)
+ end
+ rescue Exception => e
+ Rails.logger.error "Error in Resque::EnqueueTime in recording queue_time in statsd: #{e.message}"
+ end
+ perform_without_enqueue_time
+ end
+
+ end
+end
View
39 lib/resque-statsd.rb
@@ -1,18 +1,43 @@
require 'rubygems'
-require 'statsd' # Really Jnunemakers's mine branch of statsd
+require 'statsd' # Really Jnunemakers's mine branch of statsd
require 'resque/plugins/statsd'
-# Set up the client
-$resque_statsd = Statsd.new(ENV['GRAPHITE_HOST'], 8125)
-$resque_statsd.namespace="#{ENV['APP_NAME']}_#{ENV['RAILS_ENV']}.resque"
+class StatsdHelper
+ def self.graphite_host
+ return @graphite_host || ENV['GRAPHITE_HOST'] || 'localhost'
+ end
+ def self.graphite_host=(val)
+ @graphite_host = val
+ end
+
+ def self.graphite_port
+ return @graphite_port || ENV['GRAPHITE_PORT'] || 8125
+ end
+ def self.graphite_port=(val)
+ @graphite_port = val
+ end
+
+ def self.namespace
+ return @namespace || ENV['GRAPHITE_NAMESPACE'] || 'resque'
+ end
+ def self.namespace=(val)
+ @namespace = val
+ end
+ # Set up the client lazily, to minimize order-of-operations headaches.
+ def self.statsd
+ if(@stats.nil?)
+ @statsd = Statsd.new(graphite_host, graphite_port)
+ @statsd.namespace = namespace
+ end
+ return @statsd
+ end
+end
module Resque
module Plugins
module Statsd
-
- VERSION = "0.0.1"
-
+ VERSION = "0.0.4"
end
end
end
View
77 lib/resque/plugins/statsd.rb
@@ -1,29 +1,78 @@
module Resque
module Plugins
module Statsd
-
def after_enqueue_statsd(*args)
- $resque_statsd.increment("#{@queue}.enqueued")
- $resque_statsd.increment("total.enqueued")
-
+ StatsdHelper.statsd.increment("queues.#{@queue}.enqueued")
+ StatsdHelper.statsd.increment("jobs.#{self.name}.enqueued")
+ StatsdHelper.statsd.increment("total.enqueued")
+ rescue SocketError => se
+ # Common cause of this is failure of getaddrinfo (I.E. can't route to
+ # statsd server or some such). This may happen in development, when
+ # your net connection isn't in a good way, for example.
+ #
+ # Ignoring this because we don't want unreachability of statsd to
+ # impair the ability of an app to actually operate.
+ end
+
+ def before_perform_statsd(*args)
+ StatsdHelper.statsd.increment("queues.#{@queue}.started")
+ StatsdHelper.statsd.increment("jobs.#{self.name}.started")
+ StatsdHelper.statsd.increment("total.started")
+ rescue SocketError => se
+ # Common cause of this is failure of getaddrinfo (I.E. can't route to
+ # statsd server or some such). This may happen in development, when
+ # your net connection isn't in a good way, for example.
+ #
+ # Ignoring this because we don't want unreachability of statsd to
+ # impair the ability of an app to actually operate.
end
-
+
def after_perform_statsd(*args)
- $resque_statsd.increment("#{@queue}.finished")
- $resque_statsd.increment("total.finished")
+ StatsdHelper.statsd.increment("queues.#{@queue}.finished")
+ StatsdHelper.statsd.increment("jobs.#{self.name}.finished")
+ StatsdHelper.statsd.increment("total.finished")
+ rescue SocketError => se
+ # Common cause of this is failure of getaddrinfo (I.E. can't route to
+ # statsd server or some such). This may happen in development, when
+ # your net connection isn't in a good way, for example.
+ #
+ # Ignoring this because we don't want unreachability of statsd to
+ # impair the ability of an app to actually operate.
end
-
+
def on_failure_statsd(*args)
- $resque_statsd.increment("#{@queue}.failed")
- $resque_statsd.increment("total.failed")
+ StatsdHelper.statsd.increment("queues.#{@queue}.failed")
+ StatsdHelper.statsd.increment("jobs.#{self.name}.failed")
+ StatsdHelper.statsd.increment("total.failed")
+ rescue SocketError => se
+ # Common cause of this is failure of getaddrinfo (I.E. can't route to
+ # statsd server or some such). This may happen in development, when
+ # your net connection isn't in a good way, for example.
+ #
+ # Ignoring this because we don't want unreachability of statsd to
+ # impair the ability of an app to actually operate.
end
-
+
def around_perform_statsd(*args)
- $resque_statsd.time("#{@queue}.processed") do
- yield
+ # We don't want to swallow a SocketError if it didn't actually come
+ # from our attempts to talk to statsd.
+ failure_is_in_app = true
+ StatsdHelper.statsd.time("queues.#{@queue}.processed") do
+ StatsdHelper.statsd.time("jobs.#{self.name}.processed") do
+ yield
+ failure_is_in_app = false
+ end
end
+ rescue SocketError => se
+ # Common cause of this is failure of getaddrinfo (I.E. can't route to
+ # statsd server or some such). This may happen in development, when
+ # your net connection isn't in a good way, for example.
+ #
+ # Ignoring this iff it's statsd-related because we don't want
+ # unreachability of statsd to impair the ability of an app to actually
+ # operate.
+ raise se if(failure_is_in_app)
end
-
end
end
end
View
54 resque-statsd.gemspec
@@ -0,0 +1,54 @@
+# Generated by jeweler
+# DO NOT EDIT THIS FILE DIRECTLY
+# Instead, edit Jeweler::Tasks in Rakefile, and run 'rake gemspec'
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+ s.name = "resque-statsd"
+ s.version = "0.0.4"
+
+ s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
+ s.authors = ["Jon Frisby", "Jason Amster"]
+ s.date = "2012-03-23"
+ s.description = "Will add a counter for enqueuing, performing, failing and timing Jobs"
+ s.email = "engineering@cloudability.com"
+ s.extra_rdoc_files = [
+ "LICENSE.txt",
+ "README.mdown"
+ ]
+ s.files = [
+ "Gemfile",
+ "Gemfile.lock",
+ "LICENSE.txt",
+ "README.mdown",
+ "Rakefile",
+ "lib/enqueue_time.rb",
+ "lib/resque-statsd.rb",
+ "lib/resque/plugins/statsd.rb",
+ "resque-statsd.gemspec"
+ ]
+ s.homepage = "http://github.com/cloudability/resque-statsd"
+ s.licenses = ["MIT"]
+ s.require_paths = ["lib"]
+ s.rubygems_version = "1.8.13"
+ s.summary = "Adds simple counters and timers for statsd into your Resque jobs"
+
+ if s.respond_to? :specification_version then
+ s.specification_version = 3
+
+ if Gem::Version.new(Gem::VERSION) >= Gem::Version.new('1.2.0') then
+ s.add_runtime_dependency(%q<jamster-statsd>, [">= 0"])
+ s.add_development_dependency(%q<yard>, [">= 0"])
+ s.add_development_dependency(%q<jeweler>, [">= 0"])
+ else
+ s.add_dependency(%q<jamster-statsd>, [">= 0"])
+ s.add_dependency(%q<yard>, [">= 0"])
+ s.add_dependency(%q<jeweler>, [">= 0"])
+ end
+ else
+ s.add_dependency(%q<jamster-statsd>, [">= 0"])
+ s.add_dependency(%q<yard>, [">= 0"])
+ s.add_dependency(%q<jeweler>, [">= 0"])
+ end
+end
+
View
2  spec/spec_helper.rb
@@ -8,5 +8,5 @@
Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each {|f| require f}
RSpec.configure do |config|
-
+
end
Something went wrong with that request. Please try again.