Use adapters for the link and page queues #37

Closed
wants to merge 12 commits into
from

Conversation

Projects
None yet
4 participants

For now just the default (Ruby Queue class) and Redis. All tests pass.

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
+
+module Anemone
+ module Queue
+ class Redis
+
+ def initialize(queue_type, opts = {})
+ if [:link, :page].include? !queue_type
+ raise 'You must specify a queue type (:link or :page)'
+ end
+ @redis = ::Redis.new(opts)
+ @prefix = "#{opts[:key_prefix] || 'anemone'}:#{queue_type}"
+ clear
+ end
+
+ def <<(job)
+ id = @redis.incr("#{@prefix}:counter")
@EvilScott

EvilScott Oct 24, 2011

Use Redis List instead of multiple keys http://redis.io/commands#list

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
@@ -0,0 +1,69 @@
+begin
+ require 'redis'
+rescue LoadError
+ puts "You need the redis-client gem to use Anemone::Queue::Redis"
+ exit
+end
+
+module Anemone
+ module Queue
+ class Redis
+
+ def initialize(queue_type, opts = {})
@EvilScott

EvilScott Oct 24, 2011

Use self.hash (get rid of negatives?) instead of passing in queue_type for uniq key

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
+
+ def initialize(queue_type, opts = {})
+ if [:link, :page].include? !queue_type
+ raise 'You must specify a queue type (:link or :page)'
+ end
+ @redis = ::Redis.new(opts)
+ @prefix = "#{opts[:key_prefix] || 'anemone'}:#{queue_type}"
+ clear
+ end
+
+ def <<(job)
+ id = @redis.incr("#{@prefix}:counter")
+ job.each { |k,v| @redis.hset("#{@prefix}:#{id}", k, v) }
+ end
+
+ def deq
@EvilScott

EvilScott Oct 24, 2011

Keep a key of number of waiting threads. incr at the beginning of this function, decr at the end.

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
+ job.each { |k,v| @redis.hset("#{@prefix}:#{id}", k, v) }
+ end
+
+ def deq
+ key = keys.last
+ val = rget(key)
+ @redis.del(key)
+ val
+ end
+
+ def empty?
+ keys.count == 0
+ end
+
+ def size
+ keys.count

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
+ end
+
+ def <<(job)
+ id = @redis.incr("#{@prefix}:counter")
+ job.each { |k,v| @redis.hset("#{@prefix}:#{id}", k, v) }
+ end
+
+ def deq
+ key = keys.last
+ val = rget(key)
+ @redis.del(key)
+ val
+ end
+
+ def empty?
+ keys.count == 0

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue/redis.rb
+ key = keys.last
+ val = rget(key)
+ @redis.del(key)
+ val
+ end
+
+ def empty?
+ keys.count == 0
+ end
+
+ def size
+ keys.count
+ end
+
+ def num_waiting
+ keys.count
@EvilScott

EvilScott Oct 24, 2011

Return key of num waiting threads to_i

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/queue.rb
@@ -0,0 +1,15 @@
+module Anemone
+ module Queue
+
+ def self.Default(*args)
@EvilScott

EvilScott Oct 24, 2011

Rename to Basic

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/core.rb
@@ -151,8 +161,17 @@ module Anemone
@urls.delete_if { |url| !visit_link?(url) }
return if @urls.empty?
- link_queue = Queue.new
- page_queue = Queue.new
+ # create link queue
+ if Queue.methods.include? !@opts[:link_queue_adapter]
+ raise "Unknown link queue adapter #{@opts[:link_queue_adapter]}"
+ end
+ link_queue = Queue.send(@opts[:link_queue_adapter], :link, @opts[:link_queue_options])
@EvilScott

EvilScott Oct 24, 2011

link_queue = @opts[:link_queue] || Anemone::Queue.Default

@EvilScott EvilScott commented on an outdated diff Oct 24, 2011

lib/anemone/core.rb
@@ -55,7 +57,15 @@ module Anemone
# proxy server port number
:proxy_port => false,
# HTTP read timeout in seconds
- :read_timeout => nil
+ :read_timeout => nil,
+ # use default queue adapter for link queue
+ :link_queue_adapter => :Default,
+ # link queue options sub-hash
+ :link_queue_options => {},
@EvilScott

EvilScott Oct 24, 2011

No need for options here, as they are passed into Queue.Redis

Scott Reis added some commits Oct 25, 2011

Scott Reis use redis list instead of multiple keys for queue, rename Default to …
…Basic, update tests, and remove default adapter options (since they will be passed into the adapters explicitly
8fd91ab
Scott Reis refactor num_waiting to not use brpop since it blocks the socket 6d97652

@EvilScott EvilScott commented on an outdated diff Oct 28, 2011

lib/anemone/queue/redis.rb
+end
+
+module Anemone
+ module Queue
+ class Redis
+
+ def initialize(opts = {})
+ @redis = ::Redis.new(opts)
+ @list = "#{opts[:key_prefix] || 'anemone'}:#{self.hash.abs}"
+ @waiting = "#{@list}:waiting"
+ @timeout = opts[:timeout] || 10
+ clear
+ end
+
+ def <<(job)
+ @redis.lpush(@list,job.to_json)
@EvilScott

EvilScott Oct 28, 2011

Lazy load Thread local redis connection

@EvilScott EvilScott commented on an outdated diff Oct 28, 2011

lib/anemone/queue/redis.rb
+ class Redis
+
+ def initialize(opts = {})
+ @redis = ::Redis.new(opts)
+ @list = "#{opts[:key_prefix] || 'anemone'}:#{self.hash.abs}"
+ @waiting = "#{@list}:waiting"
+ @timeout = opts[:timeout] || 10
+ clear
+ end
+
+ def <<(job)
+ @redis.lpush(@list,job.to_json)
+ end
+
+ def deq
+ json = @redis.rpop(@list)
@EvilScott

EvilScott Oct 28, 2011

Lazy load redis here too

@EvilScott EvilScott commented on an outdated diff Oct 28, 2011

lib/anemone/queue/redis.rb
+ def empty?
+ size == 0
+ end
+
+ def size
+ @redis.llen(@list)
+ end
+
+ def num_waiting
+ @redis.get(@waiting).to_i
+ end
+
+ def clear
+ @redis.del(@list, @waiting)
+ end
+
@EvilScott

EvilScott Oct 28, 2011

def redis
Thread[:redis] ||= ::Redis.new(@opts)
end

@EvilScott EvilScott commented on an outdated diff Oct 28, 2011

lib/anemone/queue/redis.rb
@@ -0,0 +1,54 @@
+begin
+ require 'redis'
+rescue LoadError
+ puts "You need the redis-client gem to use Anemone::Queue::Redis"
+ exit
+end
+
+module Anemone
+ module Queue
+ class Redis
+
+ def initialize(opts = {})
+ @redis = ::Redis.new(opts)
@EvilScott

EvilScott Oct 28, 2011

Store opts in instance var

Contributor

bernd commented Jan 20, 2012

I've been using this since November. Works fine for me.

When I understand this correctly, this is to switch the in-memory queue to Redis and other memory control patches?

After 4 years, it may be time to let this die 💀

EvilScott closed this Aug 28, 2015

Contributor

brutuscat commented Aug 28, 2015

@EvilScott maybe you could take a look at my fork and submit there instead? https://github.com/brutuscat/medusa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment