Skip to content
Browse files

fixed race case, bumped version.

  • Loading branch information...
1 parent 0041fde commit e20a5663f8bb86e6502626c3cdad84bc8b8f5e39 @arya committed Jul 6, 2009
Showing with 16 additions and 16 deletions.
  1. +2 −0 README.markdown
  2. +1 −1 Rakefile
  3. +11 −13 lib/pandemic/server_side/request.rb
  4. +2 −2 pandemic.gemspec
View
2 README.markdown
@@ -1,6 +1,8 @@
# Pandemic
Pandemic is a map-reduce-ish framework. It allows you to partition requests and distribute them as you please, then process the request (or parts of it) on any number of nodes, and then reduce the response however you please. It's designed to serve requests in real-time, but can also be used for offline tasks.
+One example of where it's currently being used is to do online analysis of user behavior to detect automation. It averages 200 requests per second (300 peak) across 6 nodes, each node dealing with millions of requests each day.
+
It's different from the typical map-reduce framework in that it doesn't have a master-worker structure. Every node does can do everything. It's actually not strictly a map-reduce framework, it's a bit more lenient on what you can do to your data/request other than map and reduce.
The framework is designed to be as flexible as possible, there is no rigid request format, or API, you can specify it however you want. You can send it http-style headers and a body, you can send it JSON, or you can even just send it a single line and have it do whatever you want. The only requirement is that you write your handler to appropriately act on the request and return the response.
View
2 Rakefile
@@ -2,7 +2,7 @@ require 'rubygems'
require 'rake'
require 'echoe'
-Echoe.new('pandemic', '0.3.9') do |p|
+Echoe.new('pandemic', '0.4.0') do |p|
p.description = "Distribute MapReduce to any of the workers and it will spread, like a pandemic."
p.url = "https://github.com/arya/pandemic/"
p.author = "Arya Asemanfar"
View
24 lib/pandemic/server_side/request.rb
@@ -21,7 +21,8 @@ def initialize(body)
@request_number = @@request_count.inc
@body = body
@responses = []
- @responses_mutex = Mutex.new
+ @responses_mutex = Monitor.new
+ @waiter = @responses_mutex.new_cond
@complete = false
end
@@ -42,7 +43,7 @@ def add_response(response)
end
def wakeup_waiting_thread
- @waiting_thread.wakeup if @waiting_thread && @waiting_thread.status == "sleep"
+ @waiter.signal if @waiter
end
def responses
@@ -57,18 +58,15 @@ def cancel!
end
def wait_for_responses
- return if @complete
- @waiting_thread = Thread.current
- if Config.response_timeout <= 0
- Thread.stop
- else
- sleep Config.response_timeout
+ @responses_mutex.synchronize do
+ return if @complete
+ if Config.response_timeout <= 0
+ @waiter.wait
+ else
+ @waiter.wait(Config.response_timeout)
+ end
+ @responses.freeze
end
- # there is a race case where if the sleep finishes,
- # and response comes in and has the mutex, and then array is frozen
- # it would be ideal to use monitor wait/signal here but the monitor implementation is currently flawed
- @responses_mutex.synchronize { @responses.freeze }
- @waiting_thread = nil
end
def hash
View
4 pandemic.gemspec
@@ -2,11 +2,11 @@
Gem::Specification.new do |s|
s.name = %q{pandemic}
- s.version = "0.3.9"
+ s.version = "0.4.0"
s.required_rubygems_version = Gem::Requirement.new(">= 1.2") if s.respond_to? :required_rubygems_version=
s.authors = ["Arya Asemanfar"]
- s.date = %q{2009-06-25}
+ s.date = %q{2009-07-06}
s.description = %q{Distribute MapReduce to any of the workers and it will spread, like a pandemic.}
s.email = %q{aryaasemanfar@gmail.com}
s.extra_rdoc_files = ["lib/pandemic/client_side/cluster_connection.rb", "lib/pandemic/client_side/config.rb", "lib/pandemic/client_side/connection.rb", "lib/pandemic/client_side/connection_proxy.rb", "lib/pandemic/client_side/pandemize.rb", "lib/pandemic/connection_pool.rb", "lib/pandemic/mutex_counter.rb", "lib/pandemic/server_side/client.rb", "lib/pandemic/server_side/config.rb", "lib/pandemic/server_side/handler.rb", "lib/pandemic/server_side/peer.rb", "lib/pandemic/server_side/processor.rb", "lib/pandemic/server_side/request.rb", "lib/pandemic/server_side/server.rb", "lib/pandemic/util.rb", "lib/pandemic.rb", "README.markdown"]

0 comments on commit e20a566

Please sign in to comment.
Something went wrong with that request. Please try again.