Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Added MongoDB integtration. Working on publishing data in Sinatra wit…

…h Bootstrap.
  • Loading branch information...
commit 5c16640230a58aa16534226a6331afc343785488 1 parent ff69807
@rjurney authored
View
4 README.md
@@ -65,4 +65,8 @@ Generating to/from pair sent counts from all emails:
sent_counts = FOREACH froms GENERATE FLATTEN(group) AS (from, to), SIZE(pairs) AS total;
STORE sent_counts INTO '/tmp/sent_counts.avro' USING AvroStorage();
+### Publishing with MongoDB
+ MongoDB (from "humongous") is an open source, high-performance, schema-free, document-oriented database written in the C++ programming language
+
+MongoDB is available [here](http://www.mongodb.org/downloads). Once you install it and get it running,
View
6 src/pig/pairs.pig
@@ -3,6 +3,9 @@ REGISTER /me/pig/build/ivy/lib/Pig/json-simple-1.1.jar
REGISTER /me/pig/contrib/piggybank/java/piggybank.jar
REGISTER /me/pig/build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar
REGISTER /me/pig/build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar
+REGISTER /me/mongo-hadoop/mongo-2.3.jar
+REGISTER /me/mongo-hadoop/core/target/mongo-hadoop-core-1.0-SNAPSHOT.jar
+REGISTER /me/mongo-hadoop/pig/target/mongo-pig-1.0-SNAPSHOT.jar
DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage();
sh rm -rf '/tmp/sent_counts.avro' /* Workaround for PIG-2441 */
@@ -15,4 +18,5 @@ pairs = FOREACH pairs GENERATE LOWER(from) AS from, LOWER(to) AS to;
froms = GROUP pairs BY (from, to);
sent_counts = FOREACH froms GENERATE FLATTEN(group) AS (from, to), SIZE(pairs) AS total;
-STORE sent_counts INTO '/tmp/sent_counts.avro' USING AvroStorage();
+-- STORE sent_counts INTO '/tmp/sent_counts.avro' USING AvroStorage();
+STORE sent_counts INTO 'mongodb://localhost/test.pig' USING com.mongodb.hadoop.pig.MongoStorage;
View
4 src/ruby/Gemfile
@@ -4,5 +4,7 @@ gem "json"
gem "avro"
gem "mail"
gem "ruby-gmail"
-gem "aws-s3", :require => "aws/s3"
+# gem "aws-s3", :require => "aws/s3"
gem "yajl-ruby", :require => "yajl"
+gem "mongo"
+gem "sinatra"
View
20 src/ruby/Gemfile.lock
@@ -3,11 +3,7 @@ GEM
specs:
avro (1.6.1)
yajl-ruby
- aws-s3 (0.6.2)
- builder
- mime-types
- xml-simple
- builder (3.0.0)
+ bson (1.5.2)
i18n (0.6.0)
json (1.6.3)
mail (2.3.0)
@@ -15,15 +11,24 @@ GEM
mime-types (~> 1.16)
treetop (~> 1.4.8)
mime-types (1.17.2)
+ mongo (1.5.2)
+ bson (= 1.5.2)
polyglot (0.3.3)
+ rack (1.3.5)
+ rack-protection (1.1.4)
+ rack
ruby-gmail (0.2.1)
mail (>= 2.2.1)
shared-mime-info
shared-mime-info (0.1)
+ sinatra (1.3.1)
+ rack (~> 1.3, >= 1.3.4)
+ rack-protection (~> 1.1, >= 1.1.2)
+ tilt (~> 1.3, >= 1.3.3)
+ tilt (1.3.3)
treetop (1.4.10)
polyglot
polyglot (>= 0.3.1)
- xml-simple (1.1.1)
yajl-ruby (1.1.0)
PLATFORMS
@@ -31,8 +36,9 @@ PLATFORMS
DEPENDENCIES
avro
- aws-s3
json
mail
+ mongo
ruby-gmail
+ sinatra
yajl-ruby
View
10 src/ruby/web/index.rb
@@ -0,0 +1,10 @@
+require 'rubygems'
+require 'sinatra'
+require 'mongo'
+
+@conn = Mongo::Connection.new
+@db = @conn['test']
+@coll = @db['pig']
+
+puts "There are #{@coll.count} records. Here they are:"
+@coll.find.each { |doc| puts doc.inspect }
Please sign in to comment.
Something went wrong with that request. Please try again.