-
Notifications
You must be signed in to change notification settings - Fork 0
Replication
To support high availability and reliability, NRV implements a form of session-based replication which will apply all transactions on a slave replica as they are applied on the master service member. When a node failure occurs, the service can automatically fallback to a slave replica. In order to copy these transaction properly and replicate, transaction logs are used. Consistency between the store and the transaction logs is enforced. In case of inconsistency, a recovery procedure will constantly attempt to restore consistency. The consistency is expressed using consistency states, which have a direct impact a the service member's status.
The replication is the process through which the consistent store’s content is duplicated (for higher availability) and kept up to date in real time, from a master replica to a slave replica. To achieve this, mutation transactions (tx) are replicated to slave replicas exactly as they were applied on the master. The replication uses a custom session based replication protocol. One of the task of this protocol is to to ensure the replicated transactions are ordered by timestamp, so they are applied in the exact same order.
[Image here]
The slave replica opens a replication session on the master using a NRV POST message on the path /replicate/:master/:token/service with those parameters:
- A token identifying the service member from which to replicate (a node may have multiple master replicas)
- A randomly generated cookie to identify the request. This will prevent races when multiple session initiations are negotiated simultaneously.
- A start timestamp used to specify at which point in time the master should start replicating.
- One of two possible modes:
- Live: The master replicates mutation transactions from its transaction log. This mode is unbounded i.e. new transactions are replicated as they are written in the master transaction log. An important implementation detail worth noting is that the very last transaction is never replicated. This is simply because the transaction log file needs to be kept opened for performance reasons, and reading the last record would cause the file to close.
- Bootstrap: The slave was recently created and only has a partial store. The transaction requested to the master no longer exist (have been cleaned). In this case, replication is “synthetic” and mutation transactions are reconstructed from the master’s consistent store. In this mode, the session is bounded i.e. ends at the service member consistent timestamp at the time of the session creation.
The master creates the sessions and returns :
- A unique session id used by the slave to identify the session when receiving transactions
- A start timestamp for the first transaction that needs to be replicated.
- An optional end timestamp used only when in bootstrap mode describing the point in time up to which the session will replicate.
- The same cookie that was received from the slave, to prevent races.
At this stage, the session is already opened and the master is constantly sending transactions to the slave using a NRV PUT message on the path /replicate/slave/ sessions/:sessionId with those parameters:
- A session id that identifies the session associated with the transaction.
- A sequence number that allows received transaction to be ordered so they can be applied on the store in the right order.
- A transaction timestamp to apply the transaction on the store as if it were applied at the moment specified by the master.
- Some optional transaction data describing the transaction to apply. In some cases, some messages do not require transaction data i.e. “keep alive” messages for the replication session.
Sometimes, e.g. when the slave node needs to be restarted, or after a successful bootstrap, the replication session needs to be closed. To close the session, the slave replica simply sends a NRV DELETE message on the path /replication/:master/:token/sessions/:session with those parameters:
- A token that identifies the master service member responsible for the session.
- A session id specifying the session to close.