Does MicroRaft support Sharding?

[bootjp]

I use MicroRaft to develop distributed storage. (I've often been indebted to GitHub discussions)

So we want to do data sharding, for example by using multiple RaftGroups.

(such as CockroachDB's MultiRaft
eg. https://www.cockroachlabs.com/blog/scaling-raft )

Does current MicroRaft support sharding?
(Sorry if I missed it ...)

Are there any plans for future support?

[metanet]

Hi @bootjp. Welcome back! I am happy to support your use case. Please share your experience of using MicroRaft and I can take your input for planning future work!

With MicroRaft, you can run multiple Raft groups in the same system. Raft groups are identified by unique ids which is set in RaftNodeBuilder:

MicroRaft/microraft/src/main/java/io/microraft/RaftNode.java

Line 503 in d3696fe

RaftNodeBuilder setGroupId(@Nonnull Object groupId);

which is carried in RaftMessage objects:

MicroRaft/microraft/src/main/java/io/microraft/model/message/RaftMessage.java

Line 48 in d3696fe

Object getGroupId();

Basically you can map your logical shards into Raft groups where each Raft group runs a logical shard. In a single process, you can run multiple RaftNode objects belonging to different Raft groups. I think such a setup can work for building a sharded system. But there is definitely room for improvement. Some thoughts from me:

• RaftStore, StateMachine and Transport abstractions can take Raft group id as a parameter to their methods, so users can implement them to support multiple Raft groups more easily. For instance, you can write a State Machine implementation that can contain data for multiple shards.
• A single RaftNode object can take a list of group ids and natively be part of multiple Raft groups. This can be a way to eliminate each Raft group leader to send their heartbeats (similar to cockroachdb's impl). In my view, this is massive work which needs to be designed very carefully for the simplest and most effective solution.

I can hear your requirements.

Btw I just released v0.6 with a few new APIs that you may find interesting: https://github.com/MicroRaft/MicroRaft/releases/tag/v0.6

Cheers,

[bootjp]

Oh, thank you.

I will try to get Sharding implemented once.
If there are any APIs needed on top of that, I'll reopen and add them to this discussion.

By the way, I saw the release of v 0.6.
This is very useful for my use.
Thank you for your daily improvement.
I can't wait to contribute to MicroRaft in some way.

Thank you for always having a kind and polite discussion.