Permalink
Browse files

simple wordcount map-reduce job within Tokyo Cabinet

  • Loading branch information...
1 parent 12ebbb2 commit 64b35478785f2769bac22e4c1ecb4c31a3d9a4a0 @igrigorik committed Jul 9, 2009
Showing with 65 additions and 0 deletions.
  1. +30 −0 map-reduce/README.rdoc
  2. +23 −0 map-reduce/wordcount.lua
  3. +12 −0 map-reduce/wordcount.rb
View
@@ -0,0 +1,30 @@
+= Map-Reduce with Tokyo Cabinet
+
+Executing map-reduce jobs within Tokyo Cabinet
+
+== Starting server with incr extension
+ > ttserver -ext wordcount.lua test.tch
+
+== Executing from command line
+ > tcrmgr put localhost 1 "This is a pen."
+ > tcrmgr put localhost 2 "Hello world"
+ > tcrmgr put localhost 2 "Life is good"
+ > tcrmgr ext localhost wordcount
+ a 1
+ good 1
+ is 2
+ life 1
+ pen 1
+ this 1
+
+== Executing via Ruby
+
+ > ruby wordcount.rb
+ a 1
+ beautiful 1
+ bob 1
+ hello 2
+ what 1
+ world 2
+
+Source: http://alpha.mixi.co.jp/blog/?p=566
View
@@ -0,0 +1,23 @@
+--
+-- wordcount map-reduce
+--
+
+function wordcount()
+ function mapper(key, value, mapemit)
+ for word in string.gmatch(string.lower(value), "%w+") do
+ mapemit(word, 1)
+ end
+ return true
+ end
+
+ local res = ""
+ function reducer(key, values)
+ res = res .. key .. "\t" .. #values .. "\n"
+ return true
+ end
+
+ if not _mapreduce(mapper, reducer) then
+ res = nil
+ end
+ return res
+end
View
@@ -0,0 +1,12 @@
+require 'rubygems'
+require 'rufus/tokyo/tyrant' # sudo gem install rufus-tokyo
+
+t = Rufus::Tokyo::Tyrant.new('127.0.0.1', 1978)
+
+t['1'] = "hello world"
+t['2'] = "what a beautiful world"
+t['3'] = "hello bob"
+
+puts t.ext(:wordcount, '', '')
+
+t.close

0 comments on commit 64b3547

Please sign in to comment.