Permalink
Browse files

add strat

  • Loading branch information...
technoweenie committed Mar 17, 2011
1 parent da59653 commit c6d6711cc25662fe65d710fb4401a182dfca1d8e
View
@@ -15,8 +15,10 @@ group :server do
gem "redis"
gem "riak-client"
gem "sinatra"
+ gem "stratocaster", :path => "vendor/stratocaster"
end
group :importers do
gem "twitter"
+ gem "instagram"
end
View
@@ -1,3 +1,8 @@
+PATH
+ remote: ./vendor/stratocaster
+ specs:
+ stratocaster (0.0.1)
+
GEM
remote: http://rubygems.org/
specs:
@@ -25,9 +30,14 @@ GEM
faraday (~> 0.5.3)
hashie (0.4.0)
i18n (0.5.0)
+ instagram (0.3.2)
+ addressable
+ nibbler (>= 1.2.0)
+ yajl-ruby
multi_json (0.0.5)
multi_xml (0.2.0)
multipart-post (1.1.0)
+ nibbler (1.2.1)
rack (1.2.1)
redis (2.1.1)
riak-client (0.8.3)
@@ -62,9 +72,11 @@ DEPENDENCIES
adapter-riak
excon
faraday
+ instagram
redis
riak-client
sinatra
+ stratocaster!
toystore
twitter
yajl-ruby
View
@@ -24,6 +24,10 @@ namespace :import do
import_tweet ENV['TWEET'].to_i
end
+ task :instagram => :init do
+ #import_instagram
+ end
+
# STARS_URL - String URL to AllOfTheStars instance. Defaults to
# http://allofthestars.com
# CLUSTER_ID - String Cluster ID
@@ -1,3 +1,7 @@
+require 'bundler'
+
+Bundler.setup(:default, :server)
+
require 'toystore'
require 'adapter'
require 'adapter/riak'
@@ -0,0 +1,22 @@
+The MIT License
+
+Copyright (c) GitHub, Inc
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+
@@ -0,0 +1,110 @@
+# Stratocaster
+
+## Overview
+
+Stratocaster is a system for storing and retrieving messages on
+timelines. A message can contain any arbitrary payload. A timeline is a
+filtered stream of messages. Complex querying is replaced in favor of
+creating multiple timelines as filters for the messages. Stratocaster
+uses abstract adapters to persist the data, instead of being bound to
+any one type of data store.
+
+Some of these ideas came from the way [FriendFeed uses MySQL to store
+schemaless objects][friendfeed].
+
+[friendfeed]: http://bret.appspot.com/entry/how-friendfeed-uses-mysql
+
+### Message
+
+A Message is a schema-less entity intended to be delivered to one or
+more timelines. Stratocaster will take any ActiveModel compatible model
+(ActiveRecord, [ToyStore][toys], etc) and convert it to a Hash with
+these required keys:
+
+* `id` - the unique identifier.
+* `created_at` - the timestamp the Message was created.
+* `actor` - an object with at least an `id` property to identify the
+ user that created the message.
+* `payload` - an object with custom values for the Message.
+
+A Message would look something like this as a Ruby hash:
+
+ {'id' => 123,
+ 'created_at' => <time>,
+ 'actor' => {'id' => 12, 'name' => 'bob'},
+ 'payload' => {'repository' => {'id' => 1, 'name' => 'user/repo'},
+ 'title' => '...', ...},
+ ...
+ }
+
+[toys]: https://github.com/newtoy/toystore
+
+### Timeline
+
+A Timeline is a pre-computed view of messages that meet a certain
+criteria. A Stratocaster instance knows which possible timelines a
+message can be delivered to. As each message comes in, Stratocaster
+finds the timelines that are applicable to the message.
+
+Each timeline is responsible for persisting the Message by its Id and
+retrieving the Ids in pages. These Ids are then passed to the data
+store that Messages are persisted.
+
+## Ruby API
+
+First, you need to define the Timelines:
+
+ class RepositoryTimeline < Stratocaster::Timeline
+ # Timelines can use multiple adapters
+ adapter :redis, :host => '...', :default => true
+ adapter :mysql, ...
+
+ # This method is used to determine if this Timeline should receive
+ # the incoming Message.
+ def self.accept?(message)
+ !message.repository
+ end
+
+ def initialize(repository_id)
+ @key = "repo:#{repository_id}"
+ end
+
+ def self.deliver(message)
+ adapters.each do |adapter|
+ adapter.deliver(message.id)
+ end
+ end
+ end
+
+Create an instance of Stratocaster to start processing messages.
+
+ strat = Stratocaster.new PublicTimeline, RepositoryTimeline, ...
+
+ # Add or remove them on the instance.
+ strat.timelines << ActorTimeline
+ strat.timelines.unshift RecipientTimeline
+
+Now, you can start processing messages!
+
+ strat.receive(message)
+
+Internally, this calls:
+
+ strat.timelines.each do |timeline|
+ timeline.deliver(message) if timeline.accept?(message)
+ end
+
+To query a timeline, create an instance of the Timeline.
+
+ repo = Repository.find(1)
+ redis_timeline = RepositoryTimeline.new(repo.id, :per_page => 50)
+
+ # Get the first page of the most recent messages.
+ redis_timeline.page(1) # uses the default adapter
+ # => [12, 26, 230]
+
+ # specify the adapter
+ mysql_timeline = RepositoryTimeline.new(repo.id, :adapter => :mysql)
+
+ # or change it on an existing timeline
+ redis_timeline.adapter = :mysql
@@ -0,0 +1,151 @@
+require 'rubygems'
+require 'rake'
+require 'date'
+
+#############################################################################
+#
+# Helper functions
+#
+#############################################################################
+
+def name
+ @name ||= Dir['*.gemspec'].first.split('.').first
+end
+
+def version
+ line = File.read("lib/#{name}.rb")[/^\s*VERSION\s*=\s*.*/]
+ line.match(/.*VERSION\s*=\s*['"](.*)['"]/)[1]
+end
+
+def date
+ Date.today.to_s
+end
+
+def rubyforge_project
+ name
+end
+
+def gemspec_file
+ "#{name}.gemspec"
+end
+
+def gem_file
+ "#{name}-#{version}.gem"
+end
+
+def replace_header(head, header_name)
+ head.sub!(/(\.#{header_name}\s*= ').*'/) { "#{$1}#{send(header_name)}'"}
+end
+
+#############################################################################
+#
+# Standard tasks
+#
+#############################################################################
+
+task :default => :test
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+ test.libs << 'lib' << 'test'
+ test.pattern = 'test/**/*_test.rb'
+ test.verbose = true
+end
+
+desc "Generate RCov test coverage and open in your browser"
+task :coverage do
+ require 'rcov'
+ sh "rm -fr coverage"
+ sh "rcov test/test_*.rb"
+ sh "open coverage/index.html"
+end
+
+require 'rake/rdoctask'
+Rake::RDocTask.new do |rdoc|
+ rdoc.rdoc_dir = 'rdoc'
+ rdoc.title = "#{name} #{version}"
+ rdoc.rdoc_files.include('README*')
+ rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+desc "Open an irb session preloaded with this library"
+task :console do
+ sh "irb -rubygems -I ./lib -r #{name}.rb"
+end
+
+#############################################################################
+#
+# Custom tasks (add your own tasks here)
+#
+#############################################################################
+
+
+
+#############################################################################
+#
+# Packaging tasks
+#
+#############################################################################
+
+desc "Create tag v#{version} and build and push #{gem_file} to Rubygems"
+task :release => :build do
+ unless `git branch` =~ /^\* master$/
+ puts "You must be on the master branch to release!"
+ exit!
+ end
+ sh "git commit --allow-empty -a -m 'Release #{version}'"
+ sh "git tag v#{version}"
+ sh "git push origin master"
+ sh "git push origin v#{version}"
+ sh "gem push pkg/#{name}-#{version}.gem"
+end
+
+desc "Build #{gem_file} into the pkg directory"
+task :build => :gemspec do
+ sh "mkdir -p pkg"
+ sh "gem build #{gemspec_file}"
+ sh "mv #{gem_file} pkg"
+end
+
+desc "Generate #{gemspec_file}"
+task :gemspec => :validate do
+ # read spec file and split out manifest section
+ spec = File.read(gemspec_file)
+ head, manifest, tail = spec.split(" # = MANIFEST =\n")
+
+ # replace name version and date
+ replace_header(head, :name)
+ replace_header(head, :version)
+ replace_header(head, :date)
+ #comment this out if your rubyforge_project has a different name
+ replace_header(head, :rubyforge_project)
+
+ # determine file list from git ls-files
+ files = `git ls-files`.
+ split("\n").
+ sort.
+ reject { |file| file =~ /^\./ }.
+ reject { |file| file =~ /^(rdoc|pkg)/ }.
+ map { |file| " #{file}" }.
+ join("\n")
+
+ # piece file back together and write
+ manifest = " s.files = %w[\n#{files}\n ]\n"
+ spec = [head, manifest, tail].join(" # = MANIFEST =\n")
+ File.open(gemspec_file, 'w') { |io| io.write(spec) }
+ puts "Updated #{gemspec_file}"
+end
+
+desc "Validate #{gemspec_file}"
+task :validate do
+ libfiles = Dir['lib/*'] - ["lib/#{name}.rb", "lib/#{name}"]
+ unless libfiles.empty?
+ puts "Directory `lib` should only contain a `#{name}.rb` file and `#{name}` dir."
+ exit!
+ end
+ unless Dir['VERSION*'].empty?
+ puts "A `VERSION` file at root level violates Gem best practices."
+ exit!
+ end
+end
+
@@ -0,0 +1,42 @@
+# Stratocaster is a system for storing and retrieving Messages in Timelines.
+# A Timeline is a secondary index of Messages. Complex SQL queries are
+# replaced in favor of multiple/overlapping Timelines to filter Messages.
+# Abstract adapters are used to persist the data.
+class Stratocaster
+ VERSION = "0.0.1"
+
+ # Public: Each Stratocaster instance tracks which possible Timeline classes
+ # it can deliver Messages to.
+ attr_reader :timelines
+
+ def initialize(*timelines)
+ timelines.flatten!
+ @timelines = timelines
+ end
+
+ # Public: Processes a received Message. Since Stratocaster only stores the
+ # ID (usually, depends on the adapter), assume that the message is already
+ # stored in some other ActiveModel-compatible store (ActiveRecord, ToyStore,
+ # etc).
+ #
+ # message - A Hash:
+ # id - The String Message ID.
+ # created_at - The Time the Message was created.
+ # actor - A Hash with at least an `id` property to identify the
+ # user that created the message.
+ # payload - A Hash holding custom values for the Message.
+ #
+ # Returns an Array of String keys of Timelines that this message was
+ # delivered to.
+ def receive(message)
+ keys = @timelines.map do |timeline|
+ timeline.deliver(message) if timeline.accept?(message)
+ end
+ keys.compact!
+ keys
+ end
+end
+
+# Load up the rest of Stratocaster.
+require 'stratocaster/adapter'
+require 'stratocaster/timeline'
Oops, something went wrong.

0 comments on commit c6d6711

Please sign in to comment.