Skip to content

Commit

Permalink
update cluster_status
Browse files Browse the repository at this point in the history
  • Loading branch information
zcdb committed Jan 25, 2024
1 parent 94f226e commit 7abb3b2
Show file tree
Hide file tree
Showing 3 changed files with 80 additions and 2 deletions.
4 changes: 2 additions & 2 deletions docs/source/overview.rst
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,15 @@ Node: Vearch includes three types of nodes, master node (metadata management nod

db: Similar to a database, the db will be created first in the cluster.

Space: Similar to a database table, the library contains several tables.
Space: Similar to a database table, the db contains several spaces.

Partition: Usually each table contains several data fragments. Each data fragment only contains part of the data of the table, corresponding to partition_num.

Replica: Each data shard can have multiple replicas to ensure high availability and improve cluster performance (such as QPS, etc.), corresponding to replica_num.

Document: The basic data unit of the table. Each document usually contains several fields. Generally speaking, the document contains at least one vector field.

Field: The basic data unit of the document, including vector fields and scalar fields (numeric types integer, long, float, double and string type string).
Field: The basic data unit of the document, including vector fields and scalar fields (numeric types integer, long, float, double and string type).

Vector index: Use vector index to accelerate queries, so the index attribute must be set to true for vector fields.

Expand Down
61 changes: 61 additions & 0 deletions docs/source/use_op/cluster_status.rst
Original file line number Diff line number Diff line change
Expand Up @@ -25,3 +25,64 @@ Port Status
::

curl -XGET http://master_server/list/server

Clean lock
--------

::

curl -XGET http://master_server/clean_lock

The cluster will be locked when creating a table. If the service is abnormal during this process, the lock will not be released and will need to be manually cleared before a new table can be created.

Replica expansion and contraction
--------

::

curl -XPOST -H "content-type: application/json" -d'
{
"partition_id":1,
"node_id": 1,
"method": 0
}
' http://master_server/partition/change_member

method=0: Add a copy of shard id 1 on node id 1; method=1: Delete the copy of shard id 1 on node id 1.

Cluster data migration
--------
You can copy the data of a certain cluster to a new cluster through the following methods to realize cluster data migration.

1. Create a new target cluster

The number of nodes in the new cluster should be consistent with that of the cluster to be migrated, a complete Vearch system should be deployed, and the processes of all ps nodes in the new cluster should be killed.

2. Metadata synchronization

Use the mirror function of etcd to copy the metadata of the cluster to be migrated to the target cluster. etcdctl is the client tool for etcd.

The operation command is as follows:
::

export ETCDCTL_API=3
# The principle of etcd mirroring is to read the key row by row and write it to another cluster. Among them: sourceMasterIP is a node of the original cluster master, and targetMasterIP is a node of the target cluster master.
# ETCDCTL_API=3 ./etcdctl make-mirror target --endpoints=source
ETCDCTL_API=3 ./etcdctl make-mirror ${targetMasterIP}:2370 --endpoints=${sourceMasterIP}:2370


3. Delete the /$cluster_name/server meta-information of the target cluster
cluster_name can be found in the configuration file config.toml
::

export ETCDCTL_API=3
./etcdctl --endpoints=http://${targetMasterIP}:2370 del /$cluster_name/server --prefix


4. Copy vector data
::

scp -r root@sourcePsIP:/export/vdb/baud root@targetPsIP:/export/vdb
...

sourcePsIP is the IP of the PS node of the cluster to be migrated, and targetPsIP is the IP of the PS node of the target cluster. Here you only need to ensure that the ps node IPs of the cluster to be migrated and the target cluster are migrated one-to-one, and no special order is required.
17 changes: 17 additions & 0 deletions docs/source/use_op/op_space.rst
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,23 @@ IVFPQ:
"nsubvector": 64
}

set ivfpq with hnsw:

::

"index_size": 2600000,
"id_type": "string",
"retrieval_type": "IVFPQ",
"retrieval_param": {
"metric_type": "InnerProduct",
"ncentroids": 65536,
"nsubvector": 64,
"hnsw" : {
"nlinks": 32,
"efConstruction": 200,
"efSearch": 64
}
}

HNSW:

Expand Down

0 comments on commit 7abb3b2

Please sign in to comment.