Skip to content
This repository was archived by the owner on Jan 15, 2024. It is now read-only.

Commit dd5a7c1

Browse files
Merge Replica Set Refactor
* Removes Server, and Socket; replaced with Node, and Connection. Replica sets are now much more robustly supported, including failover and discovery. * Refactors specs. Internal APIs are now tested with integration specs through the public APIs. * More documentation.
1 parent 9084773 commit dd5a7c1

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

42 files changed

+2057
-3322
lines changed

Diff for: .gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -2,5 +2,6 @@ Gemfile.lock
22
doc
33
.yardoc
44
.rvmrc
5+
.env
56
perf/results
67
tmp/

Diff for: README.md

+160
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,21 @@ session[:artists].find(name: "Syd Vicious").
1515
)
1616
```
1717

18+
## Features
19+
20+
* Automated replica set node discovery and failover.
21+
* No C or Java extensions
22+
* No external dependencies
23+
* Simple, stable, public API.
24+
25+
### Unsupported Features
26+
27+
* GridFS
28+
* Map/Reduce
29+
30+
These features are possible to implement, but outside the scope of Moped's
31+
goals. Consider them perfect opportunities to write a companion gem!
32+
1833
# Project Breakdown
1934

2035
Moped is composed of three parts: an implementation of the [BSON
@@ -43,6 +58,31 @@ id.generation_time # => 2012-04-11 13:14:29 UTC
4358
id == Moped::BSON::ObjectId.from_string(id.to_s) # => true
4459
```
4560

61+
<table><tbody>
62+
63+
<tr><th>new</th>
64+
<td>Creates a new object id.</td></tr>
65+
66+
<tr><th>from_string</th>
67+
<td>Creates a new object id from an object id string.
68+
<br>
69+
<code>Moped::BSON::ObjectId.from_string("4f8d8c66e5a4e45396000009")</code>
70+
</td></tr>
71+
72+
<tr><th>from_time</th>
73+
<td>Creates a new object id from a time.
74+
<br>
75+
<code>Moped::BSON::ObjectId.from_time(Time.new)</code>
76+
</td></tr>
77+
78+
<tr><th>legal?</th>
79+
<td>Validates an object id string.
80+
<br>
81+
<code>Moped::BSON::ObjectId.legal?("4f8d8c66e5a4e45396000009")</code>
82+
</td></tr>
83+
84+
</tbody></table>
85+
4686
### Moped::BSON::Code
4787

4888
The `Code` class is used for working with javascript on the server.
@@ -299,6 +339,126 @@ scope.one # nil
299339

300340
</tbody></table>
301341

342+
# Exceptions
343+
344+
Here's a list of the exceptions generated by Moped.
345+
346+
<table><tbody>
347+
348+
<tr><th>Moped::Errors::ConnectionFailure</th>
349+
<td>Raised when a node cannot be reached or a connection is lost.
350+
<br>
351+
<strong>Note:</strong> this exception is only raised if Moped could not
352+
reconnect, so you shouldn't attempt to rescue this.</td></tr>
353+
354+
<tr><th>Moped::Errors::OperationFailure</th>
355+
<td>Raised when a command fails or is invalid, such as when an insert fails in
356+
safe mode.</td></tr>
357+
358+
<tr><th>Moped::Errors::QueryFailure</th>
359+
<td>Raised when an invalid query was sent to the database.</td></tr>
360+
361+
<tr><th>Moped::Errors::AuthenticationFailure</th>
362+
<td>Raised when invalid credentials were passed to `session.login`.</td></tr>
363+
364+
<tr><th>Moped::Errors::SocketError</th>
365+
<td>Not a real exception, but a module used to tag unhandled exceptions inside
366+
of a node's networking code. Allows you to `rescue Moped::SocketError` which
367+
preserving the real exception.</td></tr>
368+
369+
</tbody></table>
370+
371+
Other exceptions are possible while running commands, such as IO Errors around
372+
failed connections. Moped tries to be smart about managing its connections,
373+
such as checking if they're dead before executing a command; but those checks
374+
aren't foolproof, and Moped is conservative about handling unexpected errors on
375+
its connections. Namely, Moped will *not* retry a command if an unexpected
376+
exception is raised. Why? Because it's impossible to know whether the command
377+
was actually received by the remote Mongo instance, and without domain
378+
knowledge it cannot be safely retried.
379+
380+
Take for example this case:
381+
382+
```ruby
383+
session.with(safe: true)["users"].insert(name: "John")
384+
```
385+
386+
It's entirely possible that the insert command will be sent to Mongo, but the
387+
connection gets closed before we read the result for `getLastError`. In this
388+
case, there's no way to know whether the insert was actually successful!
389+
390+
If, however, you want to gracefully handle this in your own application, you
391+
could do something like:
392+
393+
```ruby
394+
document = { _id: Moped::BSON::ObjectId.new, name: "John" }
395+
396+
begin
397+
session["users"].insert(document)
398+
rescue Moped::Errors::SocketError
399+
session["users"].find(_id: document[:_id]).upsert(document)
400+
end
401+
```
402+
403+
# Replica Sets
404+
405+
Moped has full support for replica sets including automatic failover and node
406+
discovery.
407+
408+
## Automatic Failover
409+
410+
Moped will automatically retry lost connections and attempt to detect dead
411+
connections before sending an operation. Note, that it will *not* retry
412+
individual operations! For example, these cases will work and not raise any
413+
exceptions:
414+
415+
```ruby
416+
session[:users].insert(name: "John")
417+
# kill primary node and promote secondary
418+
session[:users].insert(name: "John")
419+
session[:users].find.count # => 2.0
420+
421+
# primary node drops our connection
422+
session[:users].insert(name: "John")
423+
```
424+
425+
However, you'll get an operation error in a case like:
426+
427+
```ruby
428+
# primary node goes down while reading the reply
429+
session.with(safe: true)[:users].insert(name: "John")
430+
```
431+
432+
And you'll get a connection error in a case like:
433+
434+
```ruby
435+
# primary node goes down, no new primary available yet
436+
session[:users].insert(name: "John")
437+
```
438+
439+
If your session is running with eventual consistency, read operations will
440+
never raise connection errors as long as any secondary or primary node is
441+
running. The only case where you'll see a connection failure is if a node goes
442+
down while attempting to retrieve more results from a cursor, because cursors
443+
are tied to individual nodes.
444+
445+
When two attempts to connect to a node fail, it will be marked as down. This
446+
removes it from the list of available nodes for `:down_interval` (default 30
447+
seconds). Note that the `:down_interval` only applies to normal operations;
448+
that is, if you ask for a primary node and none is available, all nodes will be
449+
retried. Likewise, if you ask for a secondary node, and no secondary or primary
450+
node is available, all nodes will be retreied.
451+
452+
## Node Discovery
453+
454+
The addresses you pass into your session are used as seeds for setting up
455+
replica set connections. After connection, each seed node will return a list of
456+
other known nodes which will be added to the set.
457+
458+
This information is cached according to the `:refresh_interval` option (default:
459+
5 minutes). That means, e.g., that if you add a new node to your replica set,
460+
it should be represented in Moped within 5 minutes.
461+
302462
# Thread-Safety
303463

304464
Moped is thread-safe -- depending on your definition of thread-safe. For Moped,

Diff for: lib/moped.rb

+4-2
Original file line numberDiff line numberDiff line change
@@ -6,14 +6,16 @@
66
require "moped/bson"
77
require "moped/cluster"
88
require "moped/collection"
9+
require "moped/connection"
910
require "moped/cursor"
1011
require "moped/database"
1112
require "moped/errors"
1213
require "moped/indexes"
1314
require "moped/logging"
15+
require "moped/node"
1416
require "moped/protocol"
1517
require "moped/query"
16-
require "moped/server"
1718
require "moped/session"
18-
require "moped/socket"
19+
require "moped/session/context"
20+
require "moped/threaded"
1921
require "moped/version"

Diff for: lib/moped/bson/object_id.rb

+31-51
Original file line numberDiff line numberDiff line change
@@ -8,31 +8,33 @@ class ObjectId
88
# Formatting string for outputting an ObjectId.
99
@@string_format = ("%02x" * 12).freeze
1010

11-
attr_reader :data
12-
1311
class << self
1412
def from_string(string)
1513
raise Errors::InvalidObjectId.new(string) unless legal?(string)
16-
data = []
14+
data = ""
1715
12.times { |i| data << string[i*2, 2].to_i(16) }
18-
new data
16+
from_data data
17+
end
18+
19+
def from_time(time)
20+
from_data @@generator.generate(time.to_i)
1921
end
2022

2123
def legal?(str)
22-
!!str.match(/^[0-9a-f]{24}$/i)
24+
!!str.match(/\A\h{24}\Z/i)
2325
end
24-
end
2526

26-
def initialize(data = nil, time = nil)
27-
if data
28-
@data = data
29-
elsif time
30-
@data = @@generator.generate(time.to_i)
31-
else
32-
@data = @@generator.next
27+
def from_data(data)
28+
id = allocate
29+
id.instance_variable_set :@data, data
30+
id
3331
end
3432
end
3533

34+
def data
35+
@data ||= @@generator.next
36+
end
37+
3638
def ==(other)
3739
BSON::ObjectId === other && data == other.data
3840
end
@@ -43,78 +45,56 @@ def hash
4345
end
4446

4547
def to_s
46-
@@string_format % data
48+
@@string_format % data.unpack("C12")
4749
end
4850

4951
# Return the UTC time at which this ObjectId was generated. This may
5052
# be used instread of a created_at timestamp since this information
5153
# is always encoded in the object id.
5254
def generation_time
53-
Time.at(@data.pack("C4").unpack("N")[0]).utc
55+
Time.at(data.unpack("N")[0]).utc
5456
end
5557

5658
class << self
5759
def __bson_load__(io)
58-
new io.read(12).unpack('C*')
60+
from_data(io.read(12))
5961
end
60-
6162
end
6263

6364
def __bson_dump__(io, key)
6465
io << Types::OBJECT_ID
6566
io << key
6667
io << NULL_BYTE
67-
io << data.pack('C12')
68+
io << data
6869
end
6970

7071
# @api private
7172
class Generator
7273
def initialize
7374
# Generate and cache 3 bytes of identifying information from the current
7475
# machine.
75-
@machine_id = Digest::MD5.digest(Socket.gethostname).unpack("C3")
76+
@machine_id = Digest::MD5.digest(Socket.gethostname).unpack("N")[0]
7677

7778
@mutex = Mutex.new
78-
@last_timestamp = nil
7979
@counter = 0
8080
end
8181

82-
# Return object id data based on the current time, incrementing a
83-
# counter for object ids generated in the same second.
82+
# Return object id data based on the current time, incrementing the
83+
# object id counter.
8484
def next
85-
now = Time.new.to_i
86-
87-
counter = @mutex.synchronize do
88-
last_timestamp, @last_timestamp = @last_timestamp, now
89-
90-
if last_timestamp == now
91-
@counter += 1
92-
else
93-
@counter = 0
94-
end
85+
@mutex.lock
86+
begin
87+
counter = @counter = (@counter + 1) % 0xFFFFFF
88+
ensure
89+
@mutex.unlock rescue nil
9590
end
9691

97-
generate(now, counter)
92+
generate(Time.new.to_i, counter)
9893
end
9994

100-
# Generate object id data for a given time using the provided +inc+.
101-
def generate(time, inc = 0)
102-
pid = Process.pid % 0xFFFF
103-
104-
[
105-
time >> 24 & 0xFF, # 4 bytes time (network order)
106-
time >> 16 & 0xFF,
107-
time >> 8 & 0xFF,
108-
time & 0xFF,
109-
@machine_id[0], # 3 bytes machine
110-
@machine_id[1],
111-
@machine_id[2],
112-
pid >> 8 & 0xFF, # 2 bytes process id
113-
pid & 0xFF,
114-
inc >> 16 & 0xFF, # 3 bytes increment
115-
inc >> 8 & 0xFF,
116-
inc & 0xFF,
117-
]
95+
# Generate object id data for a given time using the provided +counter+.
96+
def generate(time, counter = 0)
97+
[time, @machine_id, Process.pid, counter << 8].pack("N NX lXX NX")
11898
end
11999
end
120100

0 commit comments

Comments
 (0)