-
Notifications
You must be signed in to change notification settings - Fork 0
Replication
The replication is the process through which the consistent store’s content is duplicated (for higher availability) and kept up to date in real time, from a master replica to a slave replica. To achieve this, mutation transactions (tx) are replicated to slave replicas exactly as they were applied on the master. The replication uses a custom session based replication protocol. One of the task of this protocol is to to ensure the replicated transactions are ordered by timestamp, so they are applied in the exact same order.
[Image here]
The slave replica opens a replication session on the master using a NRV POST message on the path /replicate/:master/:token/service with those parameters:
- A token identifying the service member from which to replicate (a node may have multiple master replicas)
- A randomly generated cookie to identify the request. This will prevent races when multiple session initiations are negotiated simultaneously.
- A start timestamp used to specify at which point in time the master should start replicating.
- One of two possible modes:
- Live: The master replicates mutation transactions from its transaction log. This mode is unbounded i.e. new transactions are replicated as they are written in the master transaction log. An important implementation detail worth noting is that the very last transaction is never replicated. This is simply because the transaction log file needs to be kept opened for performance reasons, and reading the last record would cause the file to close.
- Bootstrap: The slave was recently created and only has a partial store. The transaction requested to the master no longer exist (have been cleaned). In this case, replication is “synthetic” and mutation transactions are reconstructed from the master’s consistent store. In this mode, the session is bounded i.e. ends at the service member consistent timestamp at the time of the session creation.
The master creates the sessions and returns :
- A unique session id used by the slave to identify the session when receiving transactions
- A start timestamp for the first transaction that needs to be replicated.
- An optional end timestamp used only when in bootstrap mode describing the point in time up to which the session will replicate.
- The same cookie that was received from the slave, to prevent races.
At this stage, the session is already opened and the master is constantly sending transactions to the slave using a NRV PUT message on the path /replicate/slave/ sessions/:sessionId with those parameters:
- A session id that identifies the session associated with the transaction.
- A sequence number that allows received transaction to be ordered so they can be applied on the store in the right order.
- A transaction timestamp to apply the transaction on the store as if it were applied at the moment specified by the master.
- Some optional transaction data describing the transaction to apply. In some cases, some messages do not require transaction data i.e. “keep alive” messages for the replication session.
Sometimes, e.g. when the slave node needs to be restarted, or after a successful bootstrap, the replication session needs to be closed. To close the session, the slave replica simply sends a NRV DELETE message on the path /replication/:master/:token/sessions/:session with those parameters:
- A token that identifies the master service member responsible for the session.
- A session id specifying the session to close.
When redirecting a message to another destination, a node uses a resolver to identify all replicas (master and slave) with the shard containing the token associated with the message. In a typical cluster, nodes are arranged in a pattern called “natural ring mapping”. A simple way to describe this cluster structure is to think of each node distributed in a circle, each with a set of one or more shards, each shard being replicated immediately on the next n nodes (where n is the number of times a shard is replicated). The following example of a natural ring mapping illustrates a partial cluster with a specific shard mapped to a series of nodes:
[image of sample cluster]
Three nodes from a typical natural cluster
In this example, the shard #1 is replicated on two distinct replicas, each being on a different node. Typically, the first occurrence of the shard is the master and the others are slaves (only true with a strict application of the natural ring mapping pattern). In such a case, resolving a token part of Shard#1 would give us Node#1 as a master and Node#2, Node#3 as slaves. At the store level, there is no distinction between masters and slaves replicas. In order to identify which one acts as the master, Zookeeper must be queried.
In the context of replication, the resolving is used to start a session from slaves to master and to query transactions by timestamp. Aside from replication, resolving is used to redirect messages. This is called routing. The routing logic brings two important behavior restrictions:
- Write messages (messages carrying a mutation transaction) are only sent to the master service member. This means that no automatic fallback to slave replicas is performed if the master is Down (for write messages). Failover must be performed manually in order to handle write messages. Otherwise, write message will simply be ignored.
- Read-only messages are sent to slave replicas when the master service member is unavailable. The fallback to slave replicas for these messages is automatically done when master is not Up (including Leaving, Joining and Down status). Known limitation: does not prevent read from stale slave (e.g. a slave in the process of bootstrapping).
In some case, it is not always possible to have a clean natural ring mapping pattern. In this case, an “explicit mapping” can be specified allowing any slave to replicate from any master. For this purpose, a special resolver called “explicit resolver” exists. It uses a predefined configuration to explicitly map each shard to a set of one or more nodes. Zookeeper should still be used to identify which one is the master.