How to use raft to build a kv system？

[squarezx]

Thank you very much for providing a raft C # implementation. I now want to build a distributed kv system. Can I use your framework to achieve this?

[sakno]

You're welcome, @squarezx! In your case Raft in just a beginning of the way.
Part 1: Database backend

1. You need DotNext.AspNetCore.Cluster dependency from NuGet. It is required because it contains implementation of HTTP-based communication between Raft nodes on top of ASP.NET Core infrastructure with all its benefits: DI, logging etc. However, you're able to provide your own network transport (e.g. TCP) without this library. In this case you need DotNext.Net.Cluster library and RaftCluster class from it. This class provides transport-agnosic implementation of Raft and actively used by HTTP-specific implementation from DotNext.AspNetCore.Cluster library. But I think that HTTP is OK for you because this implementation supports HTTP 1 and 2 as well as SSL and other things.
2. Any reliable database engine constructed on top .NEXT framework should use persistent Write Ahead Log. You're lucky and it is shipped with library. Read this article carefully.
3. Now you have two key components for your K/V database. However, some level of customization is required. You need to derive from PersistentState and override CreateSnapshot method as described in the article above. This method is responsible for reducing size of Write Ahead Log through compaction of log entries into single one. This process knowns as Snapshotting. I hope you're aware about Write Ahead Log role in database engines. If not then this is different thread for discussion.
4. The next important method is ApplyAsync which is virtual by default and do nothing. In your derived class this method should be responsible for decoding command for K/V database from log entry. Decoded command can be applied to underlying K/V database engine to reflect all necessary changes (CRUD operations). Here you need to design your own binary format for each command. LogEntry value type contains all necessary methods for decoding binary data in stream-like format with low overhead. CreateSnapshot and ApplyAsync methods are called automatically in synchronized context so you don't need to care about concurrency.
5. Now you need to design K/V engine which can handle commands decoded from log entries and maintain K/V storage on disk. Let's assume that your engine can be described by interface IKeyValueDatabaseEngine and appropriate implementation registed using DI container as singleton.
6. Parameter of type IKeyValueDatabaseEngine should be added to constructor of your type derived from PersistentState. This approach allows to inject database engine using DI.

Now the backend for K/V database in finished. However, we need a fronted in the form of REST API which allows clients to operate with this database.
Part 2: Database frontend

1. Design your REST API. Read operations can be performed on every Raft node even if it is not a leader. Therefore, such API can call some methods from IKeyValueDatabaseEngine for reading data
2. All writes can be handled by leader node only. However, the request can be received by follower node due to load balancing and HTTP reverse proxy. Read this chapter carefully to solve this issue using automatic routing to leader node.
3. Every write produces a set of changes. The changeset can be represented in the form of log entries. These log entries should be submitted to Write Ahead Log and committed.
4. Use IReplicatedCluster interface that can be injected using DI for writing changes into Write Ahead Log. You need to call WriteAsync method with WriteConcern.LeaderOnly or WriteConcern.None depending on chosen consistency model.
5. WriteAsync will write uncommited entries into PersistentState and wait when all of them will be committed. When entries are marked as commited then ApplyAsync overridden method will be called automatically. Note that WriteAsync method can be called by leader node only. However, automatic redirection helps you with that.

Additionally, you can examine RaftNode example which demonstrates most of concepts described in this post.

DotNext.Threading library can help you to organize concurrent asynchronous access to the database and parallelize reads. Take a look at this for more info,

[squarezx]

Thank you very much

[sakno]

@squarezx adoption for 2.0 is improved and published here. I recommend you to use latest version of library because now you don't need to implement persistent state of your database. State reconstruction provided by persistent WAL implementation shipped with the library can do this for you.

[natilivni]

Hello, regarding the following:

"Use IReplicatedCluster interface that can be injected using DI for writing changes into Write Ahead Log. You need to call WriteAsync method"

I can not find this method on the interface nor can I find an extension method for this. Has this been refactored since this issue? If so, what is the current correct way to initiate a write to the audit log?

[sakno]

@natilivni , correct. 2.x branch doesn't contain WriteAsync method of IReplicationCluster interface anymore. Could you please open a new topic on Discussions tab?

[natilivni]

Done. Discussion #27