Master Detailed Design
Vearch is a distributed system for vector storage and similarity retrieval. It consists of three modules, master, router, and partitionServer. The Mainly responsible of the master is the storage and management of metadata, as well as the coordination and management of resources.
| key | value | remarks |
|---|---|---|
| /id/node | 1 | the current maximum id value, the next nodeId add 1 |
| /id/space | 1 | |
| /id/db | 1 | |
| /id/partition | 1 |
| key | value | remarks |
|---|---|---|
| /user/ | ||
| /lock/ | ||
| /lock/_cluster | ||
| /server/ | /server/$noidId →Server struct | server structure type |
| /space/ | /space/$spaceId →Space struct | |
| /partition/ | /partition/$spaceId →Partition struct | |
| /db/ | /db/$dbId →DB struct | |
| /db/body/ |
//server/id:\[body\] ttl 3m 3s
type Server struct {
ID NodeID \`json:\"name,omitempty\"\` //unique name for raft
RpcPort uint16 \`json:\"rpc\_port\"\`
RaftHeartbeatPort uint16 \`json:\"raft\_heartbeat\_port\"\`
RaftReplicatePort uint16 \`json:\"raft\_replicate\_port\"\`
Ip string \`json:\"ip,omitempty\"\`
PartitionIds \[\]PartitionID \`json:\"p\_ids,omitempty\"\`
Size uint64 \`json:\"size,omitempty\"\`
Private bool \`json:\"private\"\`
Version \*BuildVersion \`json:\"version\"\`
}//space/\[dbId\]/\[spaceId\]:\[spaceBody\]
type Space struct {
Id SpaceID \`json:\"id,omitempty\"\`
Name string \`json:\"name,omitempty\"\` //user setting
Version Version \`json:\"version,omitempty\"\`
DBId DBID \`json:\"db\_id,omitempty\"\`
Enabled \*bool \`json:\"enabled\"\` //Enabled flag whether the space can
work
Partitions \[\]\*Partition \`json:\"partitions\"\` // partitionids not
sorted
PartitionNum int \`json:\"partition\_num\"\`
ReplicaNum uint8 \`json:\"replica\_num\"\`
Properties json.RawMessage \`json:\"properties\"\`
Engine \*Engine \`json:\"engine\"\`
Models json.RawMessage \`json:\"models,omitempty\"\` //json model config
for python plugin
}//partition/\[id\]:\[body\]
type Partition struct {
Id PartitionID \`json:\"id,omitempty\"\`
SpaceId SpaceID \`json:\"space\_id,omitempty\"\`
DBId DBID \`json:\"db\_id,omitempty\"\`
//Slot stores the lower limit of the slot range
Slot SlotID \`json:\"partition\_slot\"\`
LeaderID NodeID \`json:\"leader\_name,omitempty\"\`
Replicas \[\]NodeID \`json:\"replicas,omitempty\"\` //leader in replicas
UpdateTime int64 \`json:\"update\_time,omitempty\"\`
Path string \`json:\"-\"\`
status PartitionStatus
lock sync.RWMutex
}//db/id/\[dbId\]:\[dbName\]
//db/name/\[dbName\]:\[dbId\]
//db/body/\[dbId\]:\[dbBody\]
type DB struct {
Id DBID \`json:\"id,omitempty\"\`
Name string \`json:\"name,omitempty\"\`
Ps \[\]string \`json:\"ps,omitempty\"\` //if set this , your db only use
you config ps
}| interface address | description | remarks |
|---|---|---|
| / | vearch versioninfo | |
| /clean_lock | ||
| /list/server | partition server info | |
| /list/db | list all dbname | |
| /list/space | list all spacename of the current dbname | |
| /list/partition | list all spartition | |
| /register | register the partitionServer into master | |
| /register_partition | register the partition of the partitionServer into master | |
| /db/_create | create db | |
| delete /db/$dbId | delete db | |
| get /db/$dbId | search db | |
| put /space/_create | create space | |
| delete /space/$spaceId | drop space | |
| get /space/$spaceId | get space | |
| post /space/$spaceId | update space | |
| /partition/change_member | add or delete the partition on the specified partitionServer | |
| /_cluster/health | cluster health status | |
| /_cluster/stats | partitionserver status |
The mainly responsible of the master is the management of metadata and the coordination of system resources.Master relies on etcd to manage metadata and rovides restful The interface manages database and table metadata.The master startup process is as follows:
summary： 1.Initialize the master configuration. 2. All masters start etcd service. 3. Initialize etcd Client. 4. Start http service and open restful interface. 5. Start a scheduled task to clean up invalid partitions and spaces in etcd metadata.