Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Newer
Older
100644 303 lines (276 sloc) 9.955 kB
d6dbbb4 @floere + everything is now namespaced
authored
1 module Picky
2116d5f @floere + Internals restructuring
authored
2
536a852 @floere + backend rewrite, towards making it more exchangeable
authored
3 module Backends
9fe8c86 @floere + complete index/backend rewrite, - configuration cludge
authored
4
d6dbbb4 @floere + everything is now namespaced
authored
5 #
6 #
bda840e @floere + Do not use the backend class, but the backend instance (enables pas…
authored
7 class Redis < Backend
d6dbbb4 @floere + everything is now namespaced
authored
8
ff9ac9c @floere + Redis backend uses Memory behaviour when it is not used as a realti…
authored
9 attr_reader :client,
8590da8 @floere + redis/sqlite option is now called realtime
authored
10 :realtime
d6dbbb4 @floere + everything is now namespaced
authored
11
bda840e @floere + Do not use the backend class, but the backend instance (enables pas…
authored
12 def initialize options = {}
1690372 @floere + use hiredis if it's there for a 13% faster search.
authored
13 maybe_load_hiredis
14 check_hiredis_gem
15 check_redis_gem
16
a5015c2 @floere + Rewrote file backend to use the memory backend with the weights index
authored
17 @client = options[:client] || ::Redis.new(:db => (options[:db] || 15))
8590da8 @floere + redis/sqlite option is now called realtime
authored
18 @realtime = options[:realtime]
1690372 @floere + use hiredis if it's there for a 13% faster search.
authored
19 end
20 def maybe_load_hiredis
21 require 'hiredis'
22 rescue LoadError
23 # It's ok.
24 end
25 def check_hiredis_gem
26 require 'redis/connection/hiredis'
27 rescue LoadError
28 # It's ok, the next check will fail if this one does.
29 end
30 def check_redis_gem
31 require 'redis'
a18fcc5 @floere + Redis lib only loaded if the Redis backend is used
authored
32 rescue LoadError => e
33 warn_gem_missing 'redis', 'the Redis client'
d6dbbb4 @floere + everything is now namespaced
authored
34 end
64dab00 @floere + moving to 1.5.3
authored
35
dc976ae @floere + notes, docs, specs
authored
36 # Returns an object that on #initial, #load returns an object that responds to:
222c07a @floere ! doc
authored
37 # [:token] # => [id, id, id, id, id] (an array of ids)
27bf3db @floere + doc
authored
38 #
2a08726 @floere Add hints.
authored
39 def create_inverted bundle, _ = nil
8590da8 @floere + redis/sqlite option is now called realtime
authored
40 List.new client, "#{PICKY_ENVIRONMENT}:#{bundle.identifier}:inverted", realtime: realtime
bcf042a @floere + backend rewrite, + specs
authored
41 end
dc976ae @floere + notes, docs, specs
authored
42 # Returns an object that on #initial, #load returns an object that responds to:
222c07a @floere ! doc
authored
43 # [:token] # => 1.23 (a weight)
27bf3db @floere + doc
authored
44 #
2a08726 @floere Add hints.
authored
45 def create_weights bundle, _ = nil
8590da8 @floere + redis/sqlite option is now called realtime
authored
46 Float.new client, "#{PICKY_ENVIRONMENT}:#{bundle.identifier}:weights", realtime: realtime
bcf042a @floere + backend rewrite, + specs
authored
47 end
dc976ae @floere + notes, docs, specs
authored
48 # Returns an object that on #initial, #load returns an object that responds to:
222c07a @floere ! doc
authored
49 # [:encoded] # => [:original, :original] (an array of original symbols this similarity encoded thing maps to)
27bf3db @floere + doc
authored
50 #
2a08726 @floere Add hints.
authored
51 def create_similarity bundle, _ = nil
8590da8 @floere + redis/sqlite option is now called realtime
authored
52 List.new client, "#{PICKY_ENVIRONMENT}:#{bundle.identifier}:similarity", realtime: realtime
bcf042a @floere + backend rewrite, + specs
authored
53 end
dc976ae @floere + notes, docs, specs
authored
54 # Returns an object that on #initial, #load returns an object that responds to:
222c07a @floere ! doc
authored
55 # [:key] # => value (a value for this config key)
27bf3db @floere + doc
authored
56 #
2a08726 @floere Add hints.
authored
57 def create_configuration bundle, _ = nil
8590da8 @floere + redis/sqlite option is now called realtime
authored
58 String.new client, "#{PICKY_ENVIRONMENT}:#{bundle.identifier}:configuration", realtime: realtime
d6dbbb4 @floere + everything is now namespaced
authored
59 end
2fc80c8 @floere + backends experimental rewrite
authored
60 # Returns an object that on #initial, #load returns an object that responds to:
61 # [id] # => [:sym1, :sym2]
62 #
2a08726 @floere Add hints.
authored
63 def create_realtime bundle, _ = nil
3375659 @floere ! Redis realtime mode, + Lots of specs for Redis realtime
authored
64 List.new client, "#{PICKY_ENVIRONMENT}:#{bundle.identifier}:realtime", realtime: realtime
2fc80c8 @floere + backends experimental rewrite
authored
65 end
9fe8c86 @floere + complete index/backend rewrite, - configuration cludge
authored
66
a90e1be @floere + redis scripting implementation
authored
67 # Does the Redis version already include
68 # scripting support?
69 #
70 def redis_with_scripting?
37cdf1d @floere + Redis scripting works
authored
71 at_least_version redis_version, [2, 6, 0]
a90e1be @floere + redis scripting implementation
authored
72 end
73
74 # Compares two versions each in an array [major, minor, patch]
75 # format and returns true if the first version is higher
76 # or the same as the second one. False if not.
77 #
78 # Note: Destructive.
79 #
80 def at_least_version major_minor_patch, should_be
81 3.times { return false if major_minor_patch.shift < should_be.shift }
82 true
83 end
84
85 # Returns an array describing the
86 # current Redis version.
87 #
88 # Note: This method assumes that clients answer
89 # to #info with a hash (string/symbol keys)
90 # detailing the infos.
91 #
92 # Example:
93 # backend.redis_version # => [2, 4, 1]
94 #
95 def redis_version
96 infos = client.info
97 version_string = infos['redis_version'] || infos[:redis_version]
98 version_string.split('.').map &:to_i
99 end
100
b22adb8 @floere + first test with an optimized redis backend shows that the redis cli…
authored
101 # Returns the total weight for the combinations.
102 #
103 def weight combinations
104 # Note: A nice experiment that generated far too many strings.
105 #
106 # if redis_with_scripting?
107 # @@weight_script = "local sum = 0; for i=1,#(KEYS),2 do local value = redis.call('hget', KEYS[i], KEYS[i+1]); if value then sum = sum + value end end return sum;"
108 #
109 # require 'digest/sha1'
110 # @@weight_sent_once = nil
111 #
112 # # Scripting version of #ids.
113 # #
114 # class << self
115 # def weight combinations
116 # namespaces_keys = combinations.inject([]) do |namespaces_keys, combination|
117 # namespaces_keys << "#{combination.bundle.identifier}:weights"
118 # namespaces_keys << combination.token.text
119 # end
120 #
121 # # Assume it's using EVALSHA.
122 # #
123 # begin
124 # client.evalsha @@weight_sent_once,
125 # namespaces_keys.size,
126 # *namespaces_keys
127 # rescue RuntimeError => e
128 # # Make the server have a SHA-1 for the script.
129 # #
130 # @@weight_sent_once = Digest::SHA1.hexdigest @@weight_script
131 # client.eval @@weight_script,
132 # namespaces_keys.size,
133 # *namespaces_keys
134 # end
135 # end
136 # end
137 # else
138 # class << self
139 # def weight combinations
3375659 @floere ! Redis realtime mode, + Lots of specs for Redis realtime
authored
140 combinations.score
b22adb8 @floere + first test with an optimized redis backend shows that the redis cli…
authored
141 # end
142 # end
143 # end
144 # # Call the newly installed version.
145 # #
146 # weight combinations
147 end
148
c10bdf5 @floere + typed backends now responsible for intersecting ids, + this means t…
authored
149 # Returns the result ids for the allocation.
150 #
151 # Developers wanting to program fast intersection
152 # routines, can do so analogue to this in their own
153 # backend implementations.
154 #
155 # Note: We use the amount and offset hints to speed Redis up.
156 #
4fcd563 @floere + Redis scripting branch, first try
authored
157 def ids combinations, amount, offset
5ff80f9 @floere ! Redis []=
authored
158 # TODO This is actually not correct:
159 # A dumped/loaded Redis backend should use
160 # the Redis backend calculation method.
161 # So loaded? would be more appropriate.
a7b6bc6 @floere + Note on Redis backend
authored
162 #
8590da8 @floere + redis/sqlite option is now called realtime
authored
163 if realtime
ff9ac9c @floere + Redis backend uses Memory behaviour when it is not used as a realti…
authored
164 # Just checked once on the first call.
165 #
166 if redis_with_scripting?
45f0656 @floere ! Redis script reuse in 2.6
authored
167 @ids_script = "local intersected = redis.call('zinterstore', ARGV[1], #(KEYS), unpack(KEYS)); if intersected == 0 then redis.call('del', ARGV[1]); return {}; end local results = redis.call('zrange', ARGV[1], tonumber(ARGV[2]), tonumber(ARGV[3])); redis.call('del', ARGV[1]); return results;"
e1d3808 @floere + Redis scripting. Third try.
authored
168
ff9ac9c @floere + Redis backend uses Memory behaviour when it is not used as a realti…
authored
169 require 'digest/sha1'
45f0656 @floere ! Redis script reuse in 2.6
authored
170 @ids_script_hash = nil
e1d3808 @floere + Redis scripting. Third try.
authored
171
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
172 # Overrides _this_ method.
ff9ac9c @floere + Redis backend uses Memory behaviour when it is not used as a realti…
authored
173 #
af5bc74 @floere + extract different ids method implementations
authored
174 extend Scripting
ff9ac9c @floere + Redis backend uses Memory behaviour when it is not used as a realti…
authored
175 else
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
176 # Overrides _this_ method.
177 #
af5bc74 @floere + extract different ids method implementations
authored
178 extend NonScripting
4fcd563 @floere + Redis scripting branch, first try
authored
179 end
180 else
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
181 # Remove _this_ method and use the super
182 # class method from now on.
1441c53 @floere - duplicate code
authored
183 #
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
184 # Note: This fails if there are multiple
185 # Redis backends with different versions.
af5bc74 @floere + extract different ids method implementations
authored
186 #
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
187 self.class.send :remove_method, __method__
4fcd563 @floere + Redis scripting branch, first try
authored
188 end
5f4f9af @floere + simply remove the method if it's not a realtime (index instantly ch…
authored
189 # Call the newly installed / super class version.
190 #
191 ids combinations, amount, offset
4fcd563 @floere + Redis scripting branch, first try
authored
192 end
c10bdf5 @floere + typed backends now responsible for intersecting ids, + this means t…
authored
193
194 # Generate a multiple host/process safe result id.
195 #
196 # Note: Generated when this class loads.
197 #
198 require 'socket'
199 def self.extract_host
200 @host ||= Socket.gethostname
201 end
202 def host
203 self.class.extract_host
204 end
205 extract_host
206 def pid
207 @pid ||= Process.pid
208 end
209 # Use the host and pid (generated lazily in child processes) for the result.
210 #
211 def generate_intermediate_result_id
2f895f8 @floere + unshift, << to index, category as realtime methods.
authored
212 @intermediate_result_id ||= "#{host}:#{pid}:picky:result"
c10bdf5 @floere + typed backends now responsible for intersecting ids, + this means t…
authored
213 end
af5bc74 @floere + extract different ids method implementations
authored
214
90d202e @floere + Extract common functionality
authored
215 def identifiers_for combinations
216 combinations.inject([]) do |identifiers, combination|
217 identifiers << "#{PICKY_ENVIRONMENT}:#{combination.identifier}"
218 end
219 end
220
af5bc74 @floere + extract different ids method implementations
authored
221 # Uses Lua scripting on Redis 2.6.
222 #
223 module Scripting
224 def ids combinations, amount, offset
90d202e @floere + Extract common functionality
authored
225 identifiers = identifiers_for combinations
af5bc74 @floere + extract different ids method implementations
authored
226
227 # Assume it's using EVALSHA.
228 #
229 begin
230 if identifiers.size > 1
3a256a9 @floere + 4.12.12, handle SCRIPT FLUSH, if called in Redis
authored
231 # Reuse script already installed in Redis.
232 #
087fbba @floere + Make use of Redis' SCRIPT LOAD feature
authored
233 # Note: This raises an error in Redis,
3a256a9 @floere + 4.12.12, handle SCRIPT FLUSH, if called in Redis
authored
234 # when the script is not installed.
235 #
236 client.evalsha @ids_script_hash,
237 identifiers,
238 [
239 generate_intermediate_result_id,
240 offset,
241 (offset + amount)
242 ]
af5bc74 @floere + extract different ids method implementations
authored
243 else
45f0656 @floere ! Redis script reuse in 2.6
authored
244 # No complex calculation necessary.
245 #
af5bc74 @floere + extract different ids method implementations
authored
246 client.zrange identifiers.first,
247 offset,
248 (offset + amount)
249 end
087fbba @floere + Make use of Redis' SCRIPT LOAD feature
authored
250 rescue ::Redis::CommandError => e
3a256a9 @floere + 4.12.12, handle SCRIPT FLUSH, if called in Redis
authored
251 # Install script in Redis.
252 #
087fbba @floere + Make use of Redis' SCRIPT LOAD feature
authored
253 @ids_script_hash = client.script 'load', @ids_script
254 retry
af5bc74 @floere + extract different ids method implementations
authored
255 end
256 end
257 end
258
259 # Does not use Lua scripting, < Redis 2.6.
260 #
261 module NonScripting
262 def ids combinations, amount, offset
90d202e @floere + Extract common functionality
authored
263 identifiers = identifiers_for combinations
af5bc74 @floere + extract different ids method implementations
authored
264
265 result_id = generate_intermediate_result_id
266
267 # Little optimization.
268 #
269 if identifiers.size > 1
270 # Intersect and store.
271 #
272 intersected = client.zinterstore result_id, identifiers
273
274 # Return clean and early if there has been no intersection.
275 #
276 if intersected.zero?
277 client.del result_id
278 return []
279 end
280
281 # Get the stored result.
282 #
283 results = client.zrange result_id, offset, (offset + amount)
284
285 # Delete the stored result as it was only for temporary purposes.
286 #
287 # Note: I could also not delete it, but that
288 # would not be clean at all.
289 #
290 client.del result_id
291 else
292 results = client.zrange identifiers.first, offset, (offset + amount)
293 end
294
295 results
296 end
297 end
c10bdf5 @floere + typed backends now responsible for intersecting ids, + this means t…
authored
298
2116d5f @floere + Internals restructuring
authored
299 end
9fe8c86 @floere + complete index/backend rewrite, - configuration cludge
authored
300
2116d5f @floere + Internals restructuring
authored
301 end
9fe8c86 @floere + complete index/backend rewrite, - configuration cludge
authored
302
2116d5f @floere + Internals restructuring
authored
303 end
Something went wrong with that request. Please try again.