Permalink
Browse files

Implement read preferences for distributing reads among replica set m…

…embers PYTHON-367

Replace the 'mongo' dict with a Member object everywhere in ReplicaSetConnection.
A handful of commands obey read preferences; most are always sent to primary.
Track a 5-sample moving average of each replica set member's ping time.
Connection detects whether it's connected to primary, secondary, or mongos.
  • Loading branch information...
ajdavis committed Jul 25, 2012
1 parent 70bf190 commit f275b2291ab080c111ee99f962cff5f89986bbd5
@@ -22,6 +22,8 @@
.. autoattribute:: database
.. autoattribute:: slave_okay
.. autoattribute:: read_preference
+ .. autoattribute:: tag_sets
+ .. autoattribute:: secondary_acceptable_latency_ms
.. autoattribute:: safe
.. autoattribute:: uuid_subtype
.. automethod:: get_lasterror_options
@@ -17,11 +17,15 @@
.. autoattribute:: host
.. autoattribute:: port
+ .. autoattribute:: is_primary
+ .. autoattribute:: is_mongos
.. autoattribute:: nodes
.. autoattribute:: max_pool_size
.. autoattribute:: document_class
.. autoattribute:: tz_aware
.. autoattribute:: read_preference
+ .. autoattribute:: tag_sets
+ .. autoattribute:: secondary_acceptable_latency_ms
.. autoattribute:: slave_okay
.. autoattribute:: safe
.. autoattribute:: is_locked
@@ -20,6 +20,8 @@
.. autoattribute:: slave_okay
.. autoattribute:: read_preference
+ .. autoattribute:: tag_sets
+ .. autoattribute:: secondary_acceptable_latency_ms
.. autoattribute:: safe
.. automethod:: get_lasterror_options
.. automethod:: set_lasterror_options
@@ -13,7 +13,7 @@
Alias for :class:`pymongo.replica_set_connection.ReplicaSetConnection`.
- .. autoclass:: pymongo.ReadPreference
+ .. autoclass:: pymongo.read_preferences.ReadPreference
.. autofunction:: has_c
Sub-modules:
@@ -21,6 +21,8 @@
.. autoattribute:: primary
.. autoattribute:: secondaries
.. autoattribute:: read_preference
+ .. autoattribute:: tag_sets
+ .. autoattribute:: secondary_acceptable_latency_ms
.. autoattribute:: max_pool_size
.. autoattribute:: document_class
.. autoattribute:: tz_aware
View
@@ -100,7 +100,7 @@ Important New Features:
automatic failover handling and periodically checks the state of the
replica set to handle issues like primary stepdown or secondaries
being removed for backup operations. Read preferences are defined through
- :class:`~pymongo.ReadPreference`.
+ :class:`~pymongo.read_preferences.ReadPreference`.
- PyMongo supports the new BSON binary subtype 4 for UUIDs. The default
subtype to use can be set through
:attr:`~pymongo.collection.Collection.uuid_subtype`
View
@@ -53,8 +53,9 @@ Additionally, it will use a background greenlet instead of a background thread
to monitor the state of the replica set.
Using :meth:`~pymongo.replica_set_connection.ReplicaSetConnection.start_request()`
-with :class:`~pymongo.ReadPreference` PRIMARY ensures that the current greenlet
-uses the same socket for all operations until a call to :meth:`end_request()`.
+with :class:`~pymongo.read_preferences.ReadPreference` PRIMARY ensures that the
+current greenlet uses the same socket for all operations until a call to
+:meth:`end_request()`.
You must `install Gevent <http://gevent.org/>`_ to use
:class:`~pymongo.replica_set_connection.ReplicaSetConnection`
@@ -155,31 +155,113 @@ the operation will succeed::
ReplicaSetConnection
--------------------
-In Pymongo-2.1 a new ReplicaSetConnection class was added that provides
-some new features not supported in the original Connection class. The most
-important of these is the ability to distribute queries to the secondary
-members of a replica set. To connect using ReplicaSetConnection just
-provide a host:port pair and the name of the replica set::
+Using a :class:`~pymongo.replica_set_connection.ReplicaSetConnection` instead
+of a simple :class:`~pymongo.connection.Connection` offers two key features:
+secondary reads and replica set health monitoring. To connect using
+`ReplicaSetConnection` just provide a host:port pair and the name of the
+replica set::
>>> from pymongo import ReplicaSetConnection
>>> ReplicaSetConnection("morton.local:27017", replicaSet='foo')
ReplicaSetConnection([u'morton.local:27019', u'morton.local:27017', u'morton.local:27018'])
+Secondary Reads
+'''''''''''''''
+
By default an instance of ReplicaSetConnection will only send queries to
-the primary member of the replica set. To use secondary members for queries
-we have to change the read preference::
+the primary member of the replica set. To use secondaries for queries
+we have to change the :class:`~pymongo.read_preference.ReadPreference`::
>>> db = ReplicaSetConnection("morton.local:27017", replicaSet='foo').test
- >>> from pymongo import ReadPreference
- >>> db.read_preference = ReadPreference.SECONDARY
+ >>> from pymongo.read_preference import ReadPreference
+ >>> db.read_preference = ReadPreference.SECONDARY_PREFERRED
Now all queries will be sent to the secondary members of the set. If there are
no secondary members the primary will be used as a fallback. If you have
queries you would prefer to never send to the primary you can specify that
-using the SECONDARY_ONLY read preference::
+using the ``SECONDARY`` read preference::
- >>> db.read_preference = ReadPreference.SECONDARY_ONLY
+ >>> db.read_preference = ReadPreference.SECONDARY
Read preference can be set on a connection, database, collection, or on a
-per-query basis.
-
+per-query basis, e.g.::
+
+ >>> db.collection.find_one(read_preference=ReadPreference.PRIMARY)
+
+Reads are configured using three options: **read_preference**, **tag_sets**,
+and **secondary_acceptable_latency_ms**.
+
+**read_preference**:
+
+- ``PRIMARY``: Read from the primary. This is the default, and provides the
+ strongest consistency. If no primary is available, raise
+ :class:`~pymongo.errors.AutoReconnect`.
+
+- ``PRIMARY_PREFERRED``: Read from the primary if available, or if there is
+ none, read from a secondary matching your choice of ``tag_sets`` and
+ ``secondary_acceptable_latency_ms``.
+
+- ``SECONDARY``: Read from a secondary matching your choice of ``tag_sets`` and
+ ``secondary_acceptable_latency_ms``. If no matching secondary is available,
+ raise :class:`~pymongo.errors.AutoReconnect`.
+
+- ``SECONDARY_PREFERRED``: Read from a secondary matching your choice of
+ ``tag_sets`` and ``secondary_acceptable_latency_ms`` if available, otherwise
+ from primary (regardless of the primary's tags and latency).
+
+- ``NEAREST``: Read from any member matching your choice of ``tag_sets`` and
+ ``secondary_acceptable_latency_ms``.
+
+**tag_sets**:
+
+Replica-set members can be `tagged
+<http://www.mongodb.org/display/DOCS/Data+Center+Awareness>`_ according to any
+criteria you choose. By default, ReplicaSetConnection ignores tags when
+choosing a member to read from, but it can be configured with the ``tag_sets``
+parameter. ``tag_sets`` must be a list of dictionaries, each dict providing tag
+values that the replica set member must match. ReplicaSetConnection tries each
+set of tags in turn until it finds a set of tags with at least one matching
+member. For example, to prefer reads from the New York data center, but fall
+back to the San Francisco data center, tag your replica set members according
+to their location and create a ReplicaSetConnection like so:
+
+ >>> rsc = ReplicaSetConnection(
+ ... "morton.local:27017",
+ ... replicaSet='foo'
+ ... read_preference=ReadPreference.SECONDARY,
+ ... tag_sets=[{'dc': 'ny'}, {'dc': 'sf'}]
+ ... )
+
+ReplicaSetConnection tries to find secondaries in New York, then San Francisco,
+and raises :class:`~pymongo.errors.AutoReconnect` if none are available. As an
+additional fallback, specify a final, empty tag set, ``{}``, which means "read
+from any member that matches the mode, ignoring tags."
+
+**secondary_acceptable_latency_ms**:
+
+If multiple members match the mode and tag sets, ReplicaSetConnection reads
+from among the nearest members, chosen according to ping time. By default,
+only members whose ping times are within 15 milliseconds of the nearest
+are used for queries. You can choose to distribute reads among members with
+higher latencies by setting ``secondary_acceptable_latency_ms`` to a larger
+number. In that case, ReplicaSetConnection distributes reads among matching
+members within ``secondary_acceptable_latency_ms`` of the closest member's
+ping time.
+
+Health Monitoring
+'''''''''''''''''
+
+When ReplicaSetConnection is initialized it launches a background task to
+monitor the replica set for changes in:
+
+* Health: detect when a member goes down or comes up, or if a different member
+ becomes primary
+* Configuration: detect changes in tags
+* Latency: track a moving average of each member's ping time
+
+Replica-set monitoring ensures queries are continually routed to the proper
+members as the state of the replica set changes.
+
+It is critical to call
+:meth:`~pymongo.replica_set_connection.ReplicaSetConnection.close` to terminate
+the monitoring task before your process exits.
View
@@ -54,7 +54,8 @@ def __init__(self, database, collection="fs"):
self.__collection = database[collection]
self.__files = self.__collection.files
self.__chunks = self.__collection.chunks
- if not database.slave_okay and not database.read_preference:
+ connection = database.connection
+ if not hasattr(connection, 'is_primary') or connection.is_primary:
self.__chunks.ensure_index([("files_id", ASCENDING),
("n", ASCENDING)],
unique=True)
@@ -158,7 +159,7 @@ def get_version(self, filename=None, version=-1, **kwargs):
:Parameters:
- `filename`: ``"filename"`` of the file to get, or `None`
- - `version` (optional): version of the file to get (defualts
+ - `version` (optional): version of the file to get (defaults
to -1, the most recent version uploaded)
- `**kwargs` (optional): find files by custom metadata.
@@ -168,8 +169,8 @@ def get_version(self, filename=None, version=-1, **kwargs):
Accept keyword arguments to find files by custom metadata.
.. versionadded:: 1.9
"""
- database = self.__database
- if not database.slave_okay and not database.read_preference:
+ connection = self.__database.connection
+ if not hasattr(connection, 'is_primary') or connection.is_primary:
self.__files.ensure_index([("filename", ASCENDING),
("uploadDate", DESCENDING)])
View
@@ -48,49 +48,6 @@
ALL = 2
"""Profile all operations."""
-class ReadPreference:
- """An enum that defines the read preferences supported by PyMongo.
-
- +----------------------+--------------------------------------------------+
- | Connection type | Read Preference |
- +======================+================+================+================+
- | |`PRIMARY` |`SECONDARY` |`SECONDARY_ONLY`|
- +----------------------+----------------+----------------+----------------+
- |Connection to a single|Queries are |Queries are |Same as |
- |host. |allowed if the |allowed if the |`SECONDARY` |
- | |connection is to|connection is to| |
- | |the replica set |the replica set | |
- | |primary. |primary or a | |
- | | |secondary. | |
- +----------------------+----------------+----------------+----------------+
- |Connection to a |Queries are sent|Queries are |Same as |
- |mongos. |to the primary |distributed |`SECONDARY` |
- | |of a shard. |among shard | |
- | | |secondaries. | |
- | | |Queries are sent| |
- | | |to the primary | |
- | | |if no | |
- | | |secondaries are | |
- | | |available. | |
- | | | | |
- +----------------------+----------------+----------------+----------------+
- |ReplicaSetConnection |Queries are sent|Queries are |Queries are |
- | |to the primary |distributed |never sent to |
- | |of the replica |among replica |the replica set |
- | |set. |set secondaries.|primary. An |
- | | |Queries are sent|exception is |
- | | |to the primary |raised if no |
- | | |if no |secondary is |
- | | |secondaries are |available. |
- | | |available. | |
- | | | | |
- +----------------------+----------------+----------------+----------------+
- """
-
- PRIMARY = 0
- SECONDARY = 1
- SECONDARY_ONLY = 2
-
version_tuple = (2, 2, 1, '+')
def get_version_string():
@@ -103,6 +60,7 @@ def get_version_string():
from pymongo.connection import Connection
from pymongo.replica_set_connection import ReplicaSetConnection
+from pymongo.read_preferences import ReadPreference
def has_c():
"""Is the C extension installed?
Oops, something went wrong.

0 comments on commit f275b22

Please sign in to comment.