sm(shard manager)

Inspire by

• scaling-services-with-shard-manager
• Shard Manager: A Generic Shard Management Framework for Geo-distributed Applications
• Slicer: Auto-Sharding for Datacenter Applications
• MirrorMaker 2.0
• The Chubby lock service for loosely-coupled distributed systems

We build sm to handle the backend service which need partition data into several instance(k8s pod or actual machines, use container below), the data in each container may be seen as a job. sm developed base on etcd, to make sure that:

• When container changed, some shard moves will be made.
• When shard changed, some shard moves will be made.
• When shard or container load changed, shard moves(TODO).

Table of Contents

• Getting Started
  • Installing
• Concept explanation
  • Container
• Example
• Question
  • Question

Getting Started

Installing

Install etcd, test with etcd-v3.4.13

Install go >=go1.17.1, and run:

go get https://github.com/entertainment-venue/sm/server

go mod tidy

go run main.go --config-file sample.yml

Concept explanation

Container

Container is the resource for shard to run in, and Container establish Session with etcd, the Session is used for the management of keys in etcd, management meaning that:

• Container goes down, heartbeat which from container to etcd server will be ended.
• Only the unchanged properties of app or shard will stay in etcd permanently, make data to be managed as clean as the moonlight.

Container also define common protocol which need sharded application who want to integrate with sm:

type ShardPrimitives interface {
	Add(id string, spec *ShardSpec) error
	Drop(id string) error
}

You can implement the ShardPrimitives and inject the implementation into the Container with WithShardPrimitives, and also wrap common http api to interact with the sm server. The keep http path:

• /sm/admin/add-shard
• /sm/admin/drop-shard

For more information, you can read the source code for Container.

Example

You can see the test code as tip to understand how to construct you own sharded application:

https://github.com/entertainment-venue/sm/blob/main/client/client_test.go#L13

If you want to no better how the sm server running, see the code below as an entry point:

https://github.com/entertainment-venue/sm/blob/main/server/main.go

Question

Please feel free to report any issue or optimization suggestion in Issues