Skip to content
This repository

Mongodb EntityStore #49

Open
wants to merge 2 commits into from

3 participants

John Maxwell Ryan Tomayko sigerello
John Maxwell

Hi,

I've updated the EntityStore side of Rack::Cache to support a MongoDB entitystore. This strikes me as ideal for deployment scenarios such as Heroku, when you want the benefit of having a central cache store but don't want the cost of having to pay for large Memcached instances. MongoDB is also persistent, keeping the cache warm through resets/failure etc.

I haven't changed any of the methods/interface/api of the existing features, and using it is as simple as passing a:

:entitystore => ENV['MONGO_URI']

to Rack::Cache, or you can pass a Mongo::DB object directly in. I've updated all the tests, and it passes. I didn't implement the MetaStore in Mongo, as it seemed less important due to it's smaller size.

Hope you like it, I'd be delighted to get any feedback on it as well!

Thanks,

John

Ryan Tomayko
Owner

Uggh, sorry @jgwmaxwell. I still haven't had a chance to look over this diff. It's on my list. Anything to report since submitting the pull request? Are you using this store anywhere today?

John Maxwell

Hi, sorry, been ill for a while and off the computer.

Yeah, it's in use on http://cambelt.co to cache the images the site creates, not been high traffic, but it is working well and hasn't missed a beat there. The other two places I am using it are on Intranets, and thus not postable.

It just gets on with it the way you'd expect - seems to be working fine to me?

sigerello

Hi, @rtomayko. Are you going to merge it? It seems pretty nice!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
22  Gemfile.lock
... ...
@@ -1,22 +0,0 @@
1  
-PATH
2  
-  remote: .
3  
-  specs:
4  
-    rack-cache (1.0.3)
5  
-      rack (>= 0.4)
6  
-
7  
-GEM
8  
-  remote: http://rubygems.org/
9  
-  specs:
10  
-    bacon (1.1.0)
11  
-    dalli (1.0.5)
12  
-    memcached (1.3)
13  
-    rack (1.3.2)
14  
-
15  
-PLATFORMS
16  
-  ruby
17  
-
18  
-DEPENDENCIES
19  
-  bacon
20  
-  dalli
21  
-  memcached
22  
-  rack-cache!
92  lib/rack/cache/entitystore.rb
@@ -168,6 +168,98 @@ def self.resolve(uri)
168 168
     DISK = Disk
169 169
     FILE = Disk
170 170
 
  171
+    # Base class for mongo entity stores.
  172
+    class Mongodb < EntityStore
  173
+
  174
+      attr_reader :cache
  175
+
  176
+      extend Rack::Utils
  177
+
  178
+      def open(key)
  179
+        data = read(key)
  180
+        data && [data]
  181
+      end
  182
+
  183
+      # Resolving the URI is almost the same as Memcached.
  184
+      # other than needing a database to connect to. Mongo
  185
+      # also needs a connection name, this is currently hard
  186
+      # coded as 'entitystore' but should probably be an ENV
  187
+      # variable.
  188
+      def self.resolve(uri)
  189
+        if uri.respond_to?(:scheme)
  190
+          server = uri.to_s
  191
+          options = parse_query(uri.query)
  192
+          options.keys.each do |key|
  193
+            value =
  194
+              case value = options.delete(key)
  195
+              when 'true' ; true
  196
+              when 'false' ; false
  197
+              else value.to_sym
  198
+              end
  199
+            options[key.to_sym] = value
  200
+          end
  201
+          options[:database] = uri.path.sub(/^\//, '')
  202
+          new server, options
  203
+        else
  204
+          # if the object provided is not a URI, pass it straight through
  205
+          # to the underlying implementation.
  206
+          new uri
  207
+        end
  208
+      end
  209
+
  210
+      def initialize(server="localhost:27017", options={})
  211
+        @cache =
  212
+          if server.respond_to?(:collection)
  213
+            server['entitystore']
  214
+          else
  215
+            # use from_uri to make connecting to the DB simpler. Probably should support
  216
+            # connection pooling and stuff in here too.
  217
+            require 'mongo'
  218
+            ::Mongo::Connection.from_uri(server, options)[options[:database]]['entitystore']
  219
+          end
  220
+      end
  221
+
  222
+      def exist?(key)
  223
+        # use find_one to return a single result and close the cursor
  224
+        !cache.find_one({:key => key}).nil?
  225
+      end
  226
+
  227
+      def read(key)
  228
+        # as all data is being written in binary mode, check if nil, and
  229
+        # return nil if so before converting to a useful format if there
  230
+        # is a record returned.
  231
+        data = cache.find_one({:key => key})
  232
+        if data.nil?
  233
+          nil
  234
+        else
  235
+          # need to call `.to_s` on a mongo binary field to make it useable.
  236
+          data = data['string'].to_s if data
  237
+          data.force_encoding('BINARY') if data.respond_to?(:force_encoding)
  238
+        end
  239
+      end
  240
+
  241
+      def write(body, ttl=nil)
  242
+        buf = StringIO.new
  243
+        key, size = slurp(body){|part| buf.write(part) }
  244
+        # essentially the same as memcached so far, but encode all cached resources
  245
+        # as BSON::Binary instances to ensure they don't break the db encoding
  246
+        string = ::BSON::Binary.new(buf.string)
  247
+        # Call the indexing facility to run in the background every 5 minutes - multiple
  248
+        # calls to this do nothing, so it is safe to call on each write.
  249
+        cache.ensure_index([['key', ::Mongo::ASCENDING]], {background:true})
  250
+        # using `:upsert => true` allows us to override any existing record with that key
  251
+        [key, size] if cache.update({:key => key}, {:key => key, :string => string, :expires => ttl}, {:upsert => true})
  252
+      end
  253
+
  254
+      def purge(key)
  255
+        cache.remove({:key => key})
  256
+        nil
  257
+      end
  258
+    end
  259
+
  260
+    MONGO = Mongodb
  261
+    MONGODB = Mongodb
  262
+
171 263
     # Base class for memcached entity stores.
172 264
     class MemCacheBase < EntityStore
173 265
       # The underlying Memcached instance used to communicate with the
2  lib/rack/cache/storage.rb
@@ -44,6 +44,8 @@ def create_store(type, uri)
44 44
         case
45 45
         when defined?(::Dalli) && uri.kind_of?(::Dalli::Client)
46 46
           type.const_get(:Dalli).resolve(uri)
  47
+        when defined?(::Mongo) && uri.kind_of?(::Mongo::DB)
  48
+          type.const_get(:Mongodb).resolve(uri)
47 49
         when defined?(::Memcached) && uri.respond_to?(:stats)
48 50
           type.const_get(:MemCached).resolve(uri)
49 51
         else
1  rack-cache.gemspec
@@ -67,6 +67,7 @@ Gem::Specification.new do |s|
67 67
   s.add_development_dependency 'bacon'
68 68
   s.add_development_dependency 'memcached'
69 69
   s.add_development_dependency 'dalli'
  70
+  s.add_development_dependency 'mongo'
70 71
 
71 72
   s.has_rdoc = true
72 73
   s.homepage = "http://tomayko.com/src/rack-cache/"
14  test/entitystore_test.rb
... ...
@@ -1,6 +1,7 @@
1 1
 # coding: utf-8
2 2
 require "#{File.dirname(__FILE__)}/spec_setup"
3 3
 require 'rack/cache/entitystore'
  4
+require 'rack/cache/metastore'
4 5
 
5 6
 class Object
6 7
   def sha_like?
@@ -211,6 +212,19 @@ def sha_like?
211 212
     end
212 213
   end
213 214
 
  215
+  need_mongodb 'entity store tests' do
  216
+    describe 'Mongodb' do
  217
+      before do
  218
+        @store = Rack::Cache::EntityStore::Mongodb.new($mongodb)
  219
+      end
  220
+      after do
  221
+        @store = nil
  222
+      end
  223
+      behaves_like 'A Rack::Cache::EntityStore Implementation'
  224
+    end
  225
+
  226
+  end
  227
+
214 228
 
215 229
   need_dalli 'entity store tests' do
216 230
     describe 'Dalli' do
25  test/spec_setup.rb
... ...
@@ -1,6 +1,7 @@
1 1
 require 'pp'
2 2
 require 'tmpdir'
3 3
 require 'stringio'
  4
+require 'uri'
4 5
 
5 6
 [STDOUT, STDERR].each { |io| io.sync = true }
6 7
 
@@ -14,8 +15,28 @@
14 15
 # Set the MEMCACHED environment variable as follows to enable testing
15 16
 # of the MemCached meta and entity stores.
16 17
 ENV['MEMCACHED'] ||= 'localhost:11211'
  18
+ENV['MONGODB'] ||= 'localhost:27017'
17 19
 $memcached = nil
18 20
 $dalli = nil
  21
+$mongodb = nil
  22
+
  23
+def have_mongodb?(server=ENV['MONGODB'])
  24
+  return $mongodb unless $mongodb.nil?
  25
+  require 'mongo'
  26
+  $mongodb = Mongo::Connection.from_uri("mongodb://" + server)['rackcache']
  27
+  $mongodb['test'].insert({:ping => ' '})
  28
+  true
  29
+rescue LoadError => boom
  30
+  warn "mongo library not available. related tests will be skipped."
  31
+  $mongodb = false
  32
+  false
  33
+rescue => boom
  34
+  warn "mongo not working. related tests will be skipped."
  35
+  $mongodb = false
  36
+  false
  37
+end
  38
+
  39
+have_mongodb?
19 40
 
20 41
 def have_memcached?(server=ENV['MEMCACHED'])
21 42
   return $memcached unless $memcached.nil?
@@ -61,6 +82,10 @@ def have_dalli?(server=ENV['MEMCACHED'])
61 82
 
62 83
 have_dalli?
63 84
 
  85
+def need_mongodb(forwhat)
  86
+  yield if have_mongodb?
  87
+end
  88
+
64 89
 def need_dalli(forwhat)
65 90
   yield if have_dalli?
66 91
 end
14  test/storage_test.rb
@@ -91,4 +91,18 @@
91 91
 
92 92
   end
93 93
 
  94
+  if have_mongodb?
  95
+
  96
+    describe 'Mongo Store URIs' do
  97
+      %w[mongodb:].each do |scheme|
  98
+        it "resolves #{scheme} entity store URIs" do
  99
+          uri = scheme + '//' + ENV['MONGODB'] + "/rackcache"
  100
+          @storage.resolve_entitystore_uri(uri).
  101
+            should.be.kind_of Rack::Cache::EntityStore
  102
+        end
  103
+      end
  104
+    end
  105
+
  106
+  end
  107
+
94 108
 end
Commit_comment_tip

Tip: You can add notes to lines in a file. Hover to the left of a line to make a note

Something went wrong with that request. Please try again.