Consistency Verification Procedure

The consistency verification procedure is executed before a master service member changes its status to up, and, in the case of slaves, before a replication session is initiated. The procedure does 2 things:

1. It compares the content of the transaction logs with the content of the store. The goal of this step is to verify that each transaction written in the logs has been successfully applied on the store.
2. It attempts to repair detected inconsistencies i.e. transactions that were sent to the store without ever receiving a success response. The repair is done by removing from the store the records associated with each incomplete transaction. Each incomplete transaction may or may not have been applied on the store, but after this procedure, only the completed transaction will remain applied on the store.

Fault-tolerance

The verification procedure was designed to be highly fault-tolerant. A server crash or hardware failure will not prevent the procedure from resuming normally on next server startup. To achieve such fault-tolerance, this procedure uses "recovery logs" which are copies of transaction logs used to keep track of the consistency verification procedure.

Also, the repair method used by the procedure is the safest and simplest recovery method. However, in some unusual cases (e.g. database failure), this simple method may not be able to recover consistency, leaving the replica inconsistent. Other recovery methods such as restoring store from logs or inserting new index logs are available with the consistency check tool (not publicly available yet).