Make checkpointing pluggable #161
Conversation
badrishc
added
enhancement
|
Fixes #148 |
| /// </summary> | ||
| /// <param name="logToken"></param> | ||
| /// <returns>Commit info, if valid checkpoint found, and null otherwise</returns> | ||
| byte[] GetLogCommitMetadata(Guid logToken); |
There was a problem hiding this comment.
It's unclear from the interface whether this needs to be implemented for fold over checkpoints
There was a problem hiding this comment.
Yes, it needs to be implemented for all log checkpoints (fold-over and snapshot). In case of foldover, this has information related to the status of the log as of the checkpoint, so we can resume from the correct log location (e.g., to handle the case if more uncheckpointed activity occurred on the log before the crash).
| /// <param name="indexToken"></param> | ||
| /// <param name="logToken"></param> | ||
| /// <returns>true if latest valid checkpoint found, false otherwise</returns> | ||
| bool GetLatestCheckpoint(out Guid indexToken, out Guid logToken); |
There was a problem hiding this comment.
It's unclear from the interface how this works for fold over checkpoints
There was a problem hiding this comment.
All full checkpoints (foldover or snapshot does not matter) consist of a Guid for the index and a Guid for the log. The implementer of GetLatestCheckpoint simply returns these two Guids. For a "full checkpoint" (index+log) these two Guids will usually be the same.
…ckpoint manager interface.
|
FASTER calls the checkpoint manager interface during checkpoint/recovery in this sequence: Checkpoint:
Recovery:
Provided devices will be closed directly by FASTER when done. |
The goal of this PR is to make FASTER C# checkpointing use a pluggable user-specified interface for providing devices (such as for index and snapshot) and for performing the atomic metadata commit. We will refactor the current implementation (currently hard coded to use LocalStorageDevice and C# Streams) as a reference implementation.
This will allow users to provide plugins that might, for example, (1) write checkpoints to remote devices such as Azure page blobs; (2) use a DBMS to commit and store the checkpoint metadata; and (3) write custom checkpoint adapters that for example, may encrypt or add checksums to the checkpoint data/metadata.