Permalink
Browse files

Adding connection_pool for redis

  • Loading branch information...
1 parent 5d02ab2 commit 299b2b2c27b14122121bfe96600f7ab5a13e1e97 @jc00ke jc00ke committed Sep 18, 2011
Showing with 44 additions and 3 deletions.
  1. +1 −0 Gemfile
  2. +2 −1 History.md
  3. +9 −2 README.md
  4. +32 −0 test/test_girl_friday.rb
View
@@ -6,4 +6,5 @@ gemspec
# Needed for testing only!
gem 'minitest'
gem 'redis'
+gem 'connection_pool'
gem 'flexmock-minitest'
View
@@ -4,7 +4,8 @@ Changes
HEAD
---------
-* Switch Redis.new to Redis.connect so an :url option can be passed in.
+* Can now pass a [connection\_pool](https://github.com/mperham/connection_pool) in as a Redis instance.
+* Switch Redis.new to Redis.connect so a :url option can be passed in.
Nice for using on Heroku with Redis To Go.
* Allow stacking of error handlers, fixes GH-11
View
@@ -32,18 +32,25 @@ In your Rails app, create a `config/initializers/girl_friday.rb` which defines y
EMAIL_QUEUE = GirlFriday::WorkQueue.new(:user_email, :size => 3) do |msg|
UserMailer.registration_email(msg).deliver
end
+
IMAGE_QUEUE = GirlFriday::WorkQueue.new(:image_crawler, :size => 7) do |msg|
ImageCrawler.process(msg)
end
- SCRAPE_QUEUE = GirlFriday::WorkQueue.new(:scrape_sites, :size => 4, :store => GirlFriday::Store::Redis, :store_config => [{ :host => 'host' }] do |msg|
+ SCRAPE_QUEUE = GirlFriday::WorkQueue.new(:scrape_sites, :size => 4, :store => GirlFriday::Store::Redis, :store_config => [{ :host => 'host' }]) do |msg|
Page.scrape(msg)
end
- TRANSCODE_QUEUE = GirlFriday::WorkQueue.new(:scrape_sites, :size => 4, :store => GirlFriday::Store::Redis, :store_config => [{ :redis => $redis }] do |msg|
+ TRANSCODE_QUEUE = GirlFriday::WorkQueue.new(:scrape_sites, :size => 4, :store => GirlFriday::Store::Redis, :store_config => [{ :redis => $redis }]) do |msg|
VideoProcessor.transcode(msg)
end
+ require 'connection_pool'
+ redis_pool = ConnectionPool.new(:size => 5, :timeout => 5){ Redis.new }
+ CLEAN_FILTER_QUEUE =GirlFriday::WorkQueue.new(:clean_filter, :store => GirlFriday::Store::Redis, :store_config => [{ :redis => redis_pool}]) do |msg|
+ Filter.clean(msg)
+ end
+
:size is the number of workers to spin up and defaults to 5. Keep in mind, ActiveRecord defaults to a connection pool size of 5 so if your workers are accessing the database you'll want to ensure that the connection pool is large enough by modifying `config/database.yml`.
In your controller action or model, you can call `#push(msg)`
@@ -140,6 +140,38 @@ def test_should_persist_with_redis_instance
end
end
+ def test_should_persist_with_redis_connection_pool
+ begin
+ require 'redis'
+ require 'connection_pool'
+ redis = ConnectionPool.new(:size => 5, :timeout => 5){ Redis.new }
+ redis.flushdb
+ rescue LoadError
+ return puts "Skipping redis test, 'redis' gem not found: #{$!.message}"
+ rescue Errno::ECONNREFUSED
+ return puts 'Skipping redis test, not running locally'
+ end
+
+ mutex = Mutex.new
+ total = 100
+ count = 0
+ incr = Proc.new do
+ mutex.synchronize do
+ count += 1
+ end
+ end
+
+ async_test do |cb|
+ queue = GirlFriday::WorkQueue.new('test', :size => 2, :store => GirlFriday::Store::Redis, :store_config => [{ :redis => redis }]) do |msg|
+ incr.call
+ cb.call if count == total
+ end
+ total.times do
+ queue.push(:text => 'foo')
+ end
+ end
+ end
+
def test_should_allow_graceful_shutdown
mutex = Mutex.new
total = 100

0 comments on commit 299b2b2

Please sign in to comment.